Data processing method and device, equipment, storage medium and computer program product

Abstract

The embodiment of the invention provides a data processing method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: splicing a current frame of multi-view observation image into a target image; performing similarity matching on the target image and each historical image, and determining whether a similar frame image successfully matched with the target image exists in each historical image; wherein the historical image is obtained by splicing a plurality of historical observation images; and if the similar frame image exists, determining the pose of the current frame according to the pixel coordinates of the feature points of the current frame multi-view observation image and the three-dimensional coordinates of the corresponding feature points in the similar frame image. According to the embodiment of the invention, the pose can be optimized, so that the mapping accuracy is improved, and the effect of displaying virtual reality is better achieved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of mapping technology, and in particular to a data processing method, apparatus, device, storage medium, and computer program product. Background Art

[0002] With the development of visual mapping technology, various electronic products have emerged, and people have more and more requirements for electronic products in real life, such as virtual reality (VR) head-mounted devices and augmented reality (AR) head-mounted devices. Head-mounted devices can observe real scenes through their own cameras, identify the position or posture relative to the real scene to display virtual reality. Summary of the invention

[0003] The embodiments of the present disclosure provide a data processing method, apparatus, device, storage medium and computer program product, which can optimize posture, improve the accuracy of mapping, and better achieve the effect of displaying virtual reality.

[0004] In a first aspect, an embodiment of the present disclosure provides a data processing method, the method comprising:

[0005] Splice the current frame multi-view observation images into a target image;

[0006] Performing similarity matching between the target image and each historical image, and determining whether there is a similar frame image in each historical image that successfully matches the target image in similarity; wherein the historical image is obtained by splicing multiple historical observation images;

[0007] If a similar frame image exists, the posture of the current frame is determined according to the pixel coordinates of the feature points of the multi-eye observation image of the current frame and the three-dimensional coordinates of the corresponding feature points in the similar frame image.

[0008] In a second aspect, an embodiment of the present disclosure provides a data processing device, the device comprising:

[0009] The first processing module is used to stitch the multi-viewing images of the current frame into a target image;

[0010] The second processing module is used to perform similarity matching between the target image and each historical image, and determine whether there is a similar frame image in each historical image that successfully matches the target image; wherein the historical image is obtained by splicing multiple historical observation images;

[0011] The third processing module is used to determine the posture of the current frame according to the pixel coordinates of the feature points of the multi-eye observation image of the current frame and the three-dimensional coordinates of the corresponding feature points in the similar frame image when it is determined that there is a similar frame image.

[0012] In a third aspect, an embodiment of the present disclosure provides a head mounted device, wherein the head mounted device is used to execute the data processing method as described in any one of the first aspects above.

[0013] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0014] The memory stores computer-executable instructions;

[0015] The processor executes the computer-executable instructions stored in the memory, so that the processor performs the data processing method as described in any one of the first aspects above.

[0016] In a fifth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the data processing method described in the first aspect above is implemented.

[0017] In a sixth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing method as described in any one of the first aspects above.

[0018] The data processing method, apparatus, device, storage medium and computer program product provided by the embodiments of the present disclosure first stitch the multi-eye observation image of the current frame into a target image; perform similarity matching between the target image and each historical image to determine whether there is a similar frame image in each historical image that successfully matches the similarity with the target image; wherein the historical image is stitched from multi-eye historical observation images; if there is a similar frame image, determine the posture of the current frame according to the pixel coordinates of the feature points of the multi-eye observation image of the current frame and the three-dimensional coordinates of the corresponding feature points in the similar frame image. Therefore, by splicing the current frame multi-eye observation image into a target image, and then matching the target image with the historical image spliced ​​by multi-eye historical observation images, the matching calculation amount is reduced, thereby improving efficiency. Then, by searching whether there is a loop, when the loop is found, since the three-dimensional coordinates of the feature points in the previous image have been calculated, the position and posture of the current frame can be determined by the three-dimensional coordinates of the corresponding feature points in the similar frame image and the pixel coordinates of the feature points in the current frame multi-eye observation image, and then all the 3D points corresponding to the current frame are updated by updating the pose, thereby improving the mapping accuracy, achieving a better effect of displaying virtual reality, and realizing a better interactive effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 A schematic diagram of a scenario of a data processing method provided in an embodiment of the present disclosure;

[0021] Figure 2 A flowchart of a data processing method provided by an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of the process of online mapping provided by an embodiment of the present disclosure;

[0023] Figure 4 A flowchart of a data processing method provided by yet another embodiment of the present disclosure;

[0024] Figure 5 A structural block diagram of a data processing device provided in an embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0027] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0031] The names of messages or information exchanged between multiple devices or modules in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0032] At present, head-mounted devices can observe real scenes through their own cameras, identify the position or posture relative to the real scene to display virtual reality. This technology involves generating key frames as a map of the real scene during the observation of the real scene, then matching the current image with the stored key frames and identifying the position and posture of the current head-mounted device based on the matching results. The process involves a large amount of matching calculations, which is inefficient and has poor effects.

[0033] In order to solve the above problems, the technical concept of the present invention is to reduce the amount of matching calculations and improve efficiency. The current frame multi-eye observation image can be spliced ​​into a target image, and then the target image is matched with the historical image spliced ​​from the multi-eye historical observation images. At the same time, since the camera's posture itself has errors, as the user wears the head-mounted device and keeps walking, the posture error that may be obtained will become larger and larger. Therefore, in order to improve the accuracy, it is possible to update all 3D points corresponding to the current frame by updating the posture by searching whether there is a loop. Therefore, when the loop is found, since the three-dimensional coordinates of the feature points in the previous image have been calculated, the posture of the current frame can be determined by the three-dimensional coordinates of the corresponding feature points in the similar frame image and the pixel coordinates of the feature points of the current frame multi-eye observation image, and then all 3D points corresponding to the current frame are updated by updating the posture, thereby constructing a more accurate map, better achieving the effect of displaying virtual reality, and achieving better interactive effects.

[0034] In practical applications, the execution subject of the embodiments of the present disclosure may be an electronic device. The electronic device here may be a head-mounted device, such as a VR head-mounted device or an AR head-mounted device, etc., which is not specifically limited here. In the following, a VR head-mounted device is taken as an example, and the user interacts through the VR head-mounted device, that is, a map is built through the VR head-mounted device, thereby realizing the interaction between people and virtual objects. Specifically, by observing the real scene and using mapping technology, the information of the key frames is collected, such as the posture of the key frames, the 2D coordinates of the feature points in the key frames, and the corresponding 3D coordinates, and then the posture is continuously updated through the above key frame information, and then the 3D coordinates of the feature points are updated, so as to better achieve the effect of displaying virtual reality.

[0035] For example, see Figure 1 As shown, Figure 1 A scene diagram of a data processing method provided in an embodiment of the present disclosure. The VR head-mounted device may include a multi-eye camera, for example, four cameras (each camera can collect corresponding observation images), respectively located at the upper left, lower left, upper right and lower right. The user can interact with the VR head-mounted device through gestures or through a handle. The following takes the collection of observation images by four cameras in the head-mounted device as an example to describe the data processing method in detail.

[0036] Specifically, combined Figure 2 As shown, Figure 2 The present invention is a flowchart of a data processing method provided by an embodiment of the present invention. The present invention constructs an online + offline data processing method (i.e., online mapping + offline optimization) based on a multi-eye camera. When the user uses it, online mapping is performed very quickly and can run with low power consumption. When the user shuts down and exits the application or is in standby mode, the user can freely rotate whether to optimize offline. Offline optimization can make the map more accurate. Even if offline optimization is not selected, the map can still be used normally.

[0037] Specific, combined Figure 3 As shown, Figure 3 The schematic diagram of the process of online mapping is shown. The three-dimensional coordinates of the feature points observed in the current frame are calculated by online mapping (the user is in the online state when wearing the head-mounted device, and mapping is performed when the head-mounted device is in the online state), and then the map is constructed: first, the 4-eye camera of the head-mounted device is used for observation, and the observation image of the current frame is collected; the head-mounted device determines whether to process the current frame based on the multi-eye observation image of the current frame collected by the 4-eye camera and the posture of the camera of the current frame, according to the change of posture, if processing is required, first extract the feature points of the multi-eye image (or multiple images), and then perform feature matching with the previous multi-eye observation image to determine whether there is a target point that is not observed by the current frame. If so, perform multi-view triangulation to obtain the three-dimensional coordinates of the target point and the depth projected to the previous frames and store them.

[0038] In order to improve the accuracy of three-dimensional coordinates and thus improve the accuracy of mapping, in large space scenes, there may be loop scenes, that is, returning to scenes visited in the past: by matching the current frame with the stored images for similarity, it is detected whether the scene observed in the current frame is a scene visited in the past. If the scene is detected, the position of the camera corresponding to the current frame can be updated based on the three-dimensional coordinates corresponding to the feature points in the similar frame image, and then the three-dimensional coordinates corresponding to each feature point in the multi-eye observation image of the current frame can be updated. This data processing method for online mapping has less computing power, so it is very fast and can run with low power consumption, saving resources.

[0039] In addition, global optimization can be achieved through offline optimization (when the user takes off the headset or is in standby mode, it is offline, and the headset is optimized in offline mode), thereby updating the camera's posture and the position of 3D points in space, thereby improving the accuracy of mapping and achieving better interactive effects. Therefore, offline optimization can make the three-dimensional coordinates used for mapping more accurate, and the constructed map more accurate.

[0040] Therefore, in order to save computing power and based on the characteristics of head-mounted device observation (the camera posture may change very little for several consecutive frames), not every frame needs to be processed, so first determine whether the current frame needs to be processed. If it needs to be processed, the multi-eye observation image of the current frame is feature matched with the previous frame image. Due to the gradual association of image feature points, when a certain 3D point is no longer observed by the current data packet, the 3D point is determined to be the target point. The coordinates of the 3D point need to be calculated, and then the three-dimensional coordinates are updated through similarity matching combined with similar frame images, thereby constructing a more accurate map, thereby better achieving the effect of displaying virtual reality and achieving better interactive effects.

[0041] The technical solution of the present disclosure is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0042] In one embodiment, the data processing method can be implemented in the following ways:

[0043] refer to Figure 4 , Figure 4 A flowchart of a data processing method provided for another embodiment of the present disclosure. The data processing method may include:

[0044] S101, stitching the multi-viewing images of the current frame into a target image.

[0045] S102: Perform similarity matching between the target image and each historical image to determine whether there is a similar frame image in each historical image that successfully matches the target image.

[0046] In the disclosed embodiment, since the camera's posture itself has errors, for example, if the user keeps walking while wearing the head-mounted device, the posture error obtained may become larger and larger. Therefore, once a loop is detected, all 3D points corresponding to the current frame can be updated by updating the posture, thereby improving the mapping accuracy.

[0047] Specifically, each package of images is matched with the images stored in the map (here refers to the multi-eye historical observation images) based on the image algorithm for similarity matching (such as using netvlad or DBOW, etc.). Here, the information of the multi-eye camera is used to splice the four images into one image, and image retrieval is performed on the spliced ​​image. This is to quickly and cost-effectively find a robust loop, that is, to detect whether it has returned to the scene that has been visited in the past. If the similarity between the target image (here refers to the image after splicing the current frame) and any historical image (here refers to the image after splicing the previous frame) is greater than the similarity threshold, it is determined that there is a similar frame image in the historical observation image corresponding to the current frame multi-eye observation image that successfully matches the similarity with the current frame multi-eye observation image, that is, the scene corresponding to the current frame is detected as returning to the scene that has been visited in the past.

[0048] S103 . If a similar frame image exists, determine the posture of the current frame according to the pixel coordinates of the feature points of the multi-eye observation image of the current frame and the three-dimensional coordinates of the corresponding feature points in the similar frame image.

[0049] Among them, the pose of the current frame in the similar frame coordinate system is used to support the update of the three-dimensional coordinates corresponding to each feature point in the observation image of the current frame, and the three-dimensional coordinates are used for mapping.

[0050] In the disclosed embodiment, in order to improve the accuracy of three-dimensional coordinates and thus improve the accuracy of mapping, similar frame images corresponding to the current frame multi-viewing image can be determined through similarity matching, and then the three-dimensional coordinates corresponding to each feature point in the similar frame image can be combined to achieve the update of the three-dimensional coordinates, thereby constructing a more accurate VR map and achieving a better interactive effect.

[0051] The data processing method provided by the embodiment of the present disclosure first splices the multi-eye observation image of the current frame into a target image; performs similarity matching between the target image and each historical image, and determines whether there is a similar frame image in each historical image that successfully matches the similarity with the target image; wherein the historical image is obtained by splicing multi-eye historical observation images; if there is a similar frame image, the position and posture of the current frame are determined according to the pixel coordinates of the feature points of the multi-eye observation image of the current frame and the three-dimensional coordinates of the corresponding feature points in the similar frame image. Therefore, by splicing the multi-eye observation image of the current frame into a target image, and then matching the target image with the historical image spliced ​​by the multi-eye historical observation images, the amount of matching calculation is reduced, thereby improving the efficiency, and then by searching whether there is a loop, when the loop is found, since the feature points in the previous image have calculated the three-dimensional coordinates, the position and posture of the current frame can be determined here by the three-dimensional coordinates of the corresponding feature points in the similar frame image and the pixel coordinates of the feature points of the multi-eye observation image of the current frame, and then by updating the position and posture to update all 3D points corresponding to the current frame, thereby improving the accuracy of mapping, better achieving the effect of displaying virtual reality, and achieving better interactive effects.

[0052] In one embodiment of the present disclosure, the method further includes:

[0053] Step a1: When determining that the current frame data packet is a data packet to be processed, determine whether there is a target point through feature matching.

[0054] Among them, the data packet includes a multi-eye observation image captured by a multi-eye camera of the head-mounted device and the posture of the multi-eye camera, and the target point is a 3D point corresponding to a feature point that appears in the multi-eye observation images of the previous N consecutive frames before the current frame and does not appear in the multi-eye observation image of the current frame.

[0055] The method of the embodiment of the present disclosure can be applied in an electronic device, where the electronic device can be a head mounted device, such as a VR head mounted device. Multiple eyes (taking four eyes as an example) in the VR head mounted device can collect observation images.

[0056] Specifically, taking VR head-mounted devices as an example, such as VR helmets, since VR helmets have a visual-inertial odometry (VIO) module (sometimes also called a visual-inertial system (VINS), which is an algorithm that integrates camera and IMU data to achieve SLAM), it is able to obtain the pose of each frame of the camera. Assuming that the helmet is a multi-camera (for example, a 4-camera camera), it can simultaneously obtain 4 images at this moment and the pose of the inertial measurement unit (IMU) (i.e., the pose of the camera) every 30ms. Among them, the 4-camera image (or 4 images) and the pose of the IMU are referred to as a frame of data packets below.

[0057] In one embodiment of the present disclosure, the method further includes:

[0058] Step b1, determining whether the posture change exceeds the posture difference threshold according to the posture of the multi-eye camera in the current frame data packet and the posture of the multi-eye camera in the previous frame data packet;

[0059] Step b2: If the posture change exceeds the posture difference threshold, the current frame data packet is determined to be the data packet to be processed.

[0060] In the disclosed embodiment, in order to save computing power and based on the characteristics of the head mounted device observation (the camera pose may change very little for several consecutive frames), not every frame needs to be processed, so it is first determined whether the current frame needs to be processed.

[0061] Specifically, according to the change in posture, determine whether to process the current frame data packet: compare the posture of the multi-eye camera in the current frame data packet with the posture of the multi-eye camera in the previous frame data packet. If the posture change exceeds the posture difference threshold, for example, if the posture difference between the current frame and the last stored frame (here refers to the previous frame) is too large: translation 0.5 meters, angle difference 15 degrees, then determine that the current frame data packet needs to be processed. If processing is required, feature matching is performed between the current frame observation image and the previous frame image (here refers to the continuous first N frames of observation images before the current frame).

[0062] Step a2: If it is determined that the target point exists, determine the three-dimensional coordinates of the target point according to the posture corresponding to the first N frames of multi-eye observation images and the pixel coordinates corresponding to the target point in the first N frames of multi-eye observation images.

[0063] In the disclosed embodiment, due to the gradual association of image feature points, when a 3D point is no longer observed by the current frame data packet, multi-perspective triangulation is performed, and the three-dimensional coordinates of the 3D point are calculated by constructing a least squares problem based on the posture corresponding to the observation images of the previous N frames and the pixel coordinates corresponding to the target point in the observation images of the previous N frames.

[0064] Correspondingly, after determining the pose of the current frame in the similar frame coordinate system, the method also includes: updating the three-dimensional coordinates corresponding to each feature point in the multi-eye observation image of the current frame according to the three-dimensional coordinates of the target point, the pose of the current frame in the similar frame coordinate system and the first N frames of multi-eye observation images.

[0065] In an embodiment of the present disclosure, if the similarity between the target image and any historical image is greater than a similarity threshold, it is determined that there is a similar frame image in each historical image that successfully matches the similarity with the current frame multi-eye observation image, and the three-dimensional coordinates corresponding to each feature point in the current frame multi-eye observation image are updated according to the pixel coordinates corresponding to the feature points of the current frame multi-eye observation image, the three-dimensional coordinates of the target point, and the three-dimensional coordinates corresponding to each feature point in the similar frame image; the three-dimensional coordinates are used for mapping.

[0066] In order to save computing power and based on the characteristics of head-mounted device observation (the camera pose may change very little for several consecutive frames), not every frame needs to be processed. Therefore, it is first determined whether the current frame needs to be processed. If it needs to be processed, the multi-eye observation image of the current frame is matched with the observation image of the previous frame. Due to the gradual association of image feature points, when a 3D point is no longer observed by the current frame data packet, the 3D point is determined to be the target point and the coordinates of the 3D point are calculated.

[0067] Therefore, in order to improve the accuracy of three-dimensional coordinates and thus improve the accuracy of mapping, similarity matching can be used to determine the similar frame image corresponding to the current frame observation image, and then the three-dimensional coordinates corresponding to each feature point in the similar frame image can be combined to achieve the update of the three-dimensional coordinates, thereby constructing a more accurate VR map and achieving a better interactive effect.

[0068] Therefore, the present invention discloses a visual three-dimensional map generated by online mapping + offline optimization of map three-dimensional reconstruction using a multi-camera in a VR helmet. The map is mainly used to reposition the VR helmet device in three-dimensional space with six degrees of freedom (i.e., 6DOF), and calculate the 6-DOF information of the VR helmet in the map, which is convenient for VR interaction and multiplayer games.

[0069] In the data processing method of the disclosed embodiment, first, when it is determined that the current frame data packet is the data packet to be processed, it is determined whether there is a target point through feature matching; wherein the data packet includes the observation image collected by the multi-eye camera of the head-mounted device and the posture of the multi-eye camera, and the target point is a 3D point corresponding to a feature point that appears in the consecutive first N frames of observation images before the current frame and does not appear in the current frame observation image; if it is determined that the target point exists, the three-dimensional coordinates of the target point are determined according to the posture corresponding to the first N frames of observation images and the pixel coordinates corresponding to the target point in the first N frames of observation images; if it is determined that there is a similar frame image in the historical observation image corresponding to the current frame observation image that successfully matches the similarity with the current frame observation image, the three-dimensional coordinates corresponding to each feature point in the current frame observation image are updated according to the pixel coordinates corresponding to each feature point in the current frame observation image, the three-dimensional coordinates of the target point, and the three-dimensional coordinates corresponding to each feature point in the similar frame image; the three-dimensional coordinates are used for mapping. Therefore, by determining whether the current frame needs to be processed, if it needs to be processed, the multi-eye observation image of the current frame is feature matched with the multi-eye observation image of the previous frame. When a 3D point is no longer observed by the current frame data packet, the 3D point is determined to be the target point, and the coordinates of the 3D point are calculated. Then, similarity matching is combined with similar frame images to realize the update of the three-dimensional coordinates corresponding to each pixel point in the current frame observation image, thereby constructing a more accurate map, better achieving the effect of displaying virtual reality, and achieving better interactive effects.

[0070] In one embodiment of the present disclosure, a frame of multi-eye observation image includes multiple images captured by a multi-eye camera; determining whether a target point exists through feature matching includes:

[0071] Step c1, extracting feature points of multiple images corresponding to the current frame;

[0072] Step c2: for each image, feature matching is performed with the corresponding image in the previous frame;

[0073] Step c3: If the matching result corresponding to at least one image is that the target point exists, it is determined that the target point exists.

[0074] In the disclosed embodiment, according to the change in posture, it is determined that the current frame data packet needs to be processed, and the feature points of the multi-view image are first extracted. Here, the superpoint feature point method based on deep learning can be used to extract feature points and calculate descriptors for the four images of the current data packet, and then perform feature matching with the four images of the previous packet (here refers to the previous frame observation image or the four images corresponding to the previous frame).

[0075] Since the current frame of each round will be matched with the previous frame, based on the correlation between the gradual image feature points, it can be determined whether there is a 3D point that was observed by the previous data packet but is no longer observed by the current data packet.

[0076] Among them, when matching here, only multiple images are matched with their corresponding images (such as the upper left image is only matched with the upper left image). The reason is that the overlapping area between different cameras in the VR helmet is not large, the number of matching points is small, and the number of matching times should be reduced considering the impact of computing power (it is not necessary to match two by two, for example, the upper left and the upper right or lower left and lower right of the previous frame are matched; it is also possible to match two by two, which is not specifically limited here). Therefore, if it is detected that the matching result corresponding to at least one image is that the target point exists, it is determined that the target point exists.

[0077] In one embodiment of the present disclosure, determining the three-dimensional coordinates of the target point according to the pose corresponding to the first N frames of multi-eye observation images and the pixel coordinates corresponding to the target point in the first N frames of multi-eye observation images includes:

[0078] According to the posture corresponding to the first N frames of multi-eye observation images and the pixel coordinates corresponding to the target point in the first N frames of multi-eye observation images, a least squares model is constructed to obtain the three-dimensional coordinates of the target point.

[0079] In the embodiment of the present disclosure, it is assumed that a certain 3D point is p w (homogeneous coordinates), p w The transformation matrix to the camera plane is T (3*4 matrix), and the pixel coordinate is p uv (homogeneous coordinates), x is the depth value, and the formula is as follows:

[0080] T*p w =x*p uv

[0081] [T1,T2,T3] T *p w =x*[u,v,1] T

[0082] After finishing, we get:

[0083] T1*p w =x*u

[0084] T2*p w =x*v

[0085] T3*p w =x

[0086] Therefore, we can get the following two formulas:

[0087] (T1-u*T3)*p w =0

[0088] (T2-v*T3)*p w =0

[0089] Therefore, the 2D observation of a 3D point can construct two equations. The 2D observations of this 3D point are superimposed. Assuming there are N observations (N frames of observation images), the least squares problem is constructed:

[0090]

[0091] Perform singular value decomposition on H to solve p w , and project this 3D point into each frame observation, and store the depth of this 3D point in each frame image. That is:

[0092]

[0093] Among them, (R*p w +t) is λ, the depth.

[0094] The formula can be used to calculate the depth of a 3D point in space (including the target point) in each frame image, and store the depth of the 3D point in space in each frame image.

[0095] In one embodiment of the present disclosure, updating the three-dimensional coordinates corresponding to each feature point in the multi-eye observation image of the current frame according to the three-dimensional coordinates of the target point, the pose of the current frame, and the multi-eye observation images of the previous N frames includes:

[0096] Step d1, according to the three-dimensional coordinates of the target point, project the target point into each of the first N frames of multi-eye observation images, calculate the depth of the target point in each frame of the multi-eye observation image and the depth of the 3D points other than the target point projected in each frame of the multi-eye observation image in the corresponding observation image, and store the depth of the target point and the depth of the 3D points other than the target point;

[0097] Step d2: update the three-dimensional coordinates corresponding to each feature point in the multi-eye observation image of the current frame according to the posture of the current frame and the depth corresponding to the similar frame image.

[0098] In the disclosed embodiment, after the loop is found, since the 3D point coordinates (i.e., three-dimensional coordinates) of the feature points in the previous image have been calculated, the PNP algorithm (the input is the three-dimensional coordinates corresponding to each feature point in the similar frame image and the pixel coordinates corresponding to each feature in the multi-eye observation image of the current frame, and the output is the pose of the current frame) can be used to calculate the pose difference between the current frame and the past similar frame (i.e., similar frame) (here refers to the pose of the current frame in the similar frame coordinate system), and perform a one-step pose graph optimization. After the pose optimization is completed, the coordinates of all 3D points in the current frame can be updated according to the stored depth.

[0099] In one embodiment of the present disclosure, the method further includes:

[0100] When the head mounted device is in an offline state, for each 3D point in the space, the posture of the multi-eye camera and the three-dimensional coordinates of each 3D point in the space are updated by optimizing the error of the pixel coordinates corresponding to each 3D point in the space;

[0101] Among them, each 3D point in the space includes a 3D point corresponding to the multi-eye observation image of the current frame and a 3D point corresponding to the multi-eye historical observation image.

[0102] In the disclosed embodiment, offline optimization:

[0103] Global optimization: The optimization variable is the camera pose (R wc t wc ) and the position of the 3D point in space (i.e., three-dimensional coordinates) (P w ), the optimization constraints are:

[0104]

[0105] e=p uv -p measure

[0106] λ is the depth of the 3D point, K is the intrinsic parameter matrix of the camera, R, t is the position of the camera in the world coordinate system, p w is the coordinate of the 3D point in the world system, p uv is the pixel coordinate of this 3D point, and e is the optimization error.

[0107] Therefore, the present invention can perform image retrieval based on multi-eye images, detect whether there is a loop, that is, detect whether it has returned to the scene visited in the past, and update the posture corresponding to the current frame, and then update the three-dimensional coordinates; the optimization steps can also be adjusted according to the computing power. If offline optimization is determined, global optimization can be achieved based on the optimized constraint model, that is, updating the camera's posture and the three-dimensional coordinates of all 3D points in the space, thereby improving the accuracy of mapping and achieving better interactive effects.

[0108] Corresponding to the data processing method of the above disclosed embodiment, Figure 5 A structural block diagram of a data processing device provided in an embodiment of the present disclosure. The data processing device may be configured in an electronic device, such as a VR head mounted device or a VR head mounted device. For ease of description, only the parts related to the embodiment of the present disclosure are shown. Figure 5 , the data processing device 50 may include:

[0109] The first processing module 501 is used to stitch the multi-viewing images of the current frame into a target image;

[0110] The second processing module 502 is used to perform similarity matching between the target image and each historical image, and determine whether there is a similar frame image in each historical image that successfully matches the target image; wherein the historical image is obtained by splicing multiple historical observation images;

[0111] The third processing module 503 is used to determine the posture of the current frame according to the pixel coordinates of the feature points of the multi-eye observation image of the current frame and the three-dimensional coordinates of the corresponding feature points in the similar frame image when it is determined that there is a similar frame image.

[0112] The first processing module 501, the second processing module 502 and the third processing module 503 provided by the embodiment of the present disclosure, by splicing the multi-eye observation image of the current frame into a target image, and then matching the target image with the historical image spliced ​​by the multi-eye historical observation images, reduce the amount of matching calculation, thereby improving efficiency, and then retrieve whether there is a loop. After the loop is found, since the three-dimensional coordinates of the feature points in the previous image have been calculated, the pose of the current frame can be determined here by the three-dimensional coordinates of the corresponding feature points in the similar frame image and the pixel coordinates of the feature points of the multi-eye observation image of the current frame, and then all the 3D points corresponding to the current frame are updated by updating the pose, thereby improving the mapping accuracy, achieving a better effect of displaying virtual reality, and realizing a better interactive effect.

[0113] The device provided by the embodiment of the present disclosure can be used to execute the technical solution of the data processing method embodiment described in any one of the first aspects above. Its implementation principle and technical effects are similar, and the embodiments of the present disclosure will not be repeated here.

[0114] In one embodiment of the present disclosure, the apparatus further includes: a fourth processing module and a fifth processing module;

[0115] A fourth processing module is used to determine whether there is a target point by feature matching when determining that the current frame data packet is a data packet to be processed; wherein the data packet includes a multi-eye observation image acquired by a multi-eye camera of the head-mounted device and the position and posture of the multi-eye camera, and the target point is a 3D point corresponding to a feature point that appears in the multi-eye observation images of the previous N consecutive frames before the current frame and does not appear in the multi-eye observation image of the current frame;

[0116] The fifth processing module is used to determine the three-dimensional coordinates of the target point when it is determined that the target point exists, according to the posture corresponding to the first N frames of multi-eye observation images and the pixel coordinates corresponding to the target point in the first N frames of multi-eye observation images.

[0117] In one embodiment of the present disclosure, the device further includes: a sixth processing module; the sixth processing module is used to:

[0118] After determining the pose of the current frame, the three-dimensional coordinates corresponding to each feature point in the multi-eye observation image of the current frame are updated according to the three-dimensional coordinates of the target point, the pose of the current frame and the multi-eye observation images of the previous N frames.

[0119] In one embodiment of the present disclosure, the fourth processing module is described in detail based on the above disclosed embodiment. A frame of multi-eye observation image includes multiple images captured by the multi-eye camera; the fourth processing module is specifically used to:

[0120] Extract feature points of multiple images corresponding to the current frame;

[0121] For each image, feature matching is performed with the corresponding image in the previous frame;

[0122] If the matching result corresponding to at least one image indicates that the target point exists, it is determined that the target point exists.

[0123] In one embodiment of the present disclosure, the fifth processing module is specifically configured to:

[0124] According to the posture corresponding to the first N frames of multi-eye observation images and the pixel coordinates corresponding to the target point in the first N frames of multi-eye observation images, a least squares model is constructed to obtain the three-dimensional coordinates of the target point.

[0125] In one embodiment of the present disclosure, the sixth processing module is specifically configured to:

[0126] According to the three-dimensional coordinates of the target point, the target point is projected into each of the first N frames of multi-eye observation images, and the depth of the target point in each frame of the multi-eye observation image and the depth of the 3D points other than the target point projected in each frame of the multi-eye observation image in the corresponding observation image are calculated and the depth of the target point and the depth of the 3D points other than the target point are stored;

[0127] According to the pose of the current frame in the similar frame coordinate system and the depth corresponding to the similar frame image, the three-dimensional coordinates corresponding to each feature point in the multi-eye observation image of the current frame are updated.

[0128] In one embodiment of the present disclosure, the data processing device further includes: a seventh processing module; the seventh processing module is used to:

[0129] When the head mounted device is in an offline state, for each 3D point in the space, the posture of the multi-eye camera and the three-dimensional coordinates of each 3D point in the space are updated by optimizing the error of the pixel coordinates corresponding to each 3D point in the space;

[0130] Among them, each 3D point in the space includes a 3D point corresponding to the multi-eye observation image of the current frame and a 3D point corresponding to the multi-eye historical observation image.

[0131] In one embodiment of the present disclosure, the data processing device further includes: an eighth processing module; the eighth processing module is used to:

[0132] According to the posture of the multi-eye camera in the current frame data packet and the posture of the multi-eye camera in the previous frame data packet, determine whether the posture change exceeds the posture difference threshold;

[0133] If the posture change exceeds the posture difference threshold, the current frame data packet is determined to be the data packet to be processed.

[0134] The above modules may be implemented as software components executed on one or more general-purpose processors, or may be implemented as hardware such as programmable logic devices and / or application-specific integrated circuits that perform certain functions or combinations thereof. In some embodiments, these modules may be embodied in the form of a software product that may be stored in a non-volatile storage medium that includes a computer device (e.g., a personal computer, a server, a network device, a mobile terminal, etc.) that implements the method described in the embodiments of the present invention. In one embodiment, the above modules may also be implemented on a single device or may be distributed on multiple devices. The functions of these modules may be combined with each other or further split into multiple submodules.

[0135] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the modules of the above-described device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0136] Based on the same inventive concept as the method, an embodiment of the present application also provides a head-mounted device, which is used to execute the method as described in any one of the first aspects.

[0137] Based on the same inventive concept as the method, an embodiment of the present application further provides an electronic device, including a processor and a memory;

[0138] The memory stores computer-executable instructions;

[0139] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the methods described in the above-mentioned various method embodiments.

[0140] refer to Figure 6 , which shows a structural schematic diagram of an electronic device suitable for implementing an embodiment of the present disclosure, and the electronic device may be a terminal device. The electronic device includes a processor and a memory; the memory is used to store a program for executing the method described in the above-mentioned various method embodiments; the processor is configured to execute the program stored in the memory. The processor may include a central processing unit (CPU) or other forms of processing units with mapping capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. The memory may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may run the program instructions to implement the functions of the embodiments of the present disclosure described above and / or other desired functions.

[0141] The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDA), tablet computers (Portable Android Device, PAD), portable multimedia players (Portable Media Player, PMP), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 6The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0142] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0143] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0144] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0145] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0146] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0147] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above disclosed embodiment.

[0148] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0151] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0152] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] An embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing method as described in the first aspect above.

[0154] In a first aspect, an embodiment of the present disclosure provides a data processing method, including:

[0155] Splice the current frame multi-view observation images into a target image;

[0156] Performing similarity matching between the target image and each historical image, and determining whether there is a similar frame image in each historical image that successfully matches the target image in similarity; wherein the historical image is obtained by splicing multiple historical observation images;

[0157] If a similar frame image exists, the posture of the current frame is determined according to the pixel coordinates of the feature points of the multi-eye observation image of the current frame and the three-dimensional coordinates of the corresponding feature points in the similar frame image.

[0158] According to one or more embodiments of the present disclosure, the method further includes:

[0159] When determining that the current frame data packet is a data packet to be processed, determining whether there is a target point by feature matching; wherein the data packet includes a multi-eye observation image acquired by a multi-eye camera of the head-mounted device and the position and posture of the multi-eye camera, and the target point is a 3D point corresponding to a feature point that appears in the multi-eye observation images of the previous N consecutive frames before the current frame and does not appear in the multi-eye observation image of the current frame;

[0160] If it is determined that the target point exists, the three-dimensional coordinates of the target point are determined according to the posture corresponding to the first N frames of multi-eye observation images and the pixel coordinates corresponding to the target point in the first N frames of multi-eye observation images.

[0161] According to one or more embodiments of the present disclosure, after determining the posture of the current frame, the method further includes: updating the three-dimensional coordinates corresponding to each feature point in the multi-eye observation image of the current frame according to the three-dimensional coordinates of the target point, the posture of the current frame and the multi-eye observation images of the previous N frames.

[0162] According to one or more embodiments of the present disclosure, a frame of multi-eye observation image includes multiple images captured by a multi-eye camera; determining whether a target point exists through feature matching includes:

[0163] Extract feature points of multiple images corresponding to the current frame;

[0164] For each image, feature matching is performed with the corresponding image in the previous frame;

[0165] If the matching result corresponding to at least one image indicates that the target point exists, it is determined that the target point exists.

[0166] According to one or more embodiments of the present disclosure, determining the three-dimensional coordinates of the target point according to the pose corresponding to the first N frames of multi-eye observation images and the pixel coordinates corresponding to the target point in the first N frames of multi-eye observation images includes:

[0167] According to the posture corresponding to the first N frames of multi-eye observation images and the pixel coordinates corresponding to the target point in the first N frames of multi-eye observation images, a least squares model is constructed to obtain the three-dimensional coordinates of the target point.

[0168] According to one or more embodiments of the present disclosure, updating the three-dimensional coordinates corresponding to each feature point in the multi-eye observation image of the current frame according to the three-dimensional coordinates of the target point, the pose of the current frame, and the first N frames of multi-eye observation images includes:

[0169] According to the three-dimensional coordinates of the target point, the target point is projected into each of the first N frames of multi-eye observation images, and the depth of the target point in each frame of the multi-eye observation image and the depth of the 3D points other than the target point projected in each frame of the multi-eye observation image in the corresponding observation image are calculated and the depth of the target point and the depth of the 3D points other than the target point are stored;

[0170] According to the posture of the current frame and the depth corresponding to the similar frame image, the three-dimensional coordinates corresponding to each feature point in the current frame multi-eye observation image are updated.

[0171] According to one or more embodiments of the present disclosure, the method further includes:

[0172] When the head mounted device is in an offline state, for each 3D point in the space, the posture of the multi-eye camera and the three-dimensional coordinates of each 3D point in the space are updated by optimizing the error of the pixel coordinates corresponding to each 3D point in the space;

[0173] Among them, each 3D point in the space includes a 3D point corresponding to the multi-eye observation image of the current frame and a 3D point corresponding to the multi-eye historical observation image.

[0174] According to one or more embodiments of the present disclosure, the method further includes:

[0175] According to the posture of the multi-eye camera in the current frame data packet and the posture of the multi-eye camera in the previous frame data packet, determine whether the posture change exceeds the posture difference threshold;

[0176] If the posture change exceeds the posture difference threshold, the current frame data packet is determined to be the data packet to be processed.

[0177] In a second aspect, an embodiment of the present disclosure provides a data processing device, the device comprising:

[0178] The first processing module is used to stitch the multi-viewing images of the current frame into a target image;

[0179] The second processing module is used to perform similarity matching between the target image and each historical image, and determine whether there is a similar frame image in each historical image that successfully matches the target image; wherein the historical image is obtained by splicing multiple historical observation images;

[0180] The third processing module is used to determine the posture of the current frame according to the pixel coordinates of the feature points of the multi-eye observation image of the current frame and the three-dimensional coordinates of the corresponding feature points in the similar frame image when it is determined that there is a similar frame image.

[0181] According to one or more embodiments of the present disclosure, the apparatus further includes: a fourth processing module and a fifth processing module;

[0182] A fourth processing module is used to determine whether there is a target point by feature matching when determining that the current frame data packet is a data packet to be processed; wherein the data packet includes a multi-eye observation image acquired by a multi-eye camera of the head-mounted device and the position and posture of the multi-eye camera, and the target point is a 3D point corresponding to a feature point that appears in the multi-eye observation images of the previous N consecutive frames before the current frame and does not appear in the multi-eye observation image of the current frame;

[0183] The fifth processing module is used to determine the three-dimensional coordinates of the target point when it is determined that the target point exists, according to the posture corresponding to the first N frames of multi-eye observation images and the pixel coordinates corresponding to the target point in the first N frames of multi-eye observation images.

[0184] According to one or more embodiments of the present disclosure, the device further includes: a sixth processing module; the sixth processing module is configured to:

[0185] After determining the pose of the current frame, the three-dimensional coordinates corresponding to each feature point in the multi-eye observation image of the current frame are updated according to the three-dimensional coordinates of the target point, the pose of the current frame and the multi-eye observation images of the previous N frames.

[0186] According to one or more embodiments of the present disclosure, a frame of multi-eye observation image includes multiple images captured by a multi-eye camera; the fourth processing module is specifically used to:

[0187] Extract feature points of multiple images corresponding to the current frame;

[0188] For each image, feature matching is performed with the corresponding image in the previous frame;

[0189] If the matching result corresponding to at least one image indicates that the target point exists, it is determined that the target point exists.

[0190] According to one or more embodiments of the present disclosure, the fifth processing module is specifically configured to:

[0191] According to the posture corresponding to the first N frames of multi-eye observation images and the pixel coordinates corresponding to the target point in the first N frames of multi-eye observation images, a least squares model is constructed to obtain the three-dimensional coordinates of the target point.

[0192] According to one or more embodiments of the present disclosure, the sixth processing module of the data processing device is specifically configured to:

[0193] According to the three-dimensional coordinates of the target point, the target point is projected into each of the first N frames of multi-eye observation images, and the depth of the target point in each frame of the multi-eye observation image and the depth of the 3D points other than the target point projected in each frame of the multi-eye observation image in the corresponding observation image are calculated and the depth of the target point and the depth of the 3D points other than the target point are stored;

[0194] According to the pose of the current frame in the similar frame coordinate system and the depth corresponding to the similar frame image, the three-dimensional coordinates corresponding to each feature point in the multi-eye observation image of the current frame are updated.

[0195] According to one or more embodiments of the present disclosure, the data processing device further includes: a seventh processing module; the seventh processing module is configured to:

[0196] The data processing device updates the posture of the multi-camera and the three-dimensional coordinates of each 3D point in the space by optimizing the error of the pixel coordinates corresponding to each 3D point in the space when the head mounted device is in an offline state;

[0197] Among them, each 3D point in the space includes a 3D point corresponding to the multi-eye observation image of the current frame and a 3D point corresponding to the multi-eye historical observation image.

[0198] According to one or more embodiments of the present disclosure, the data processing device further includes: an eighth processing module; the eighth processing module is configured to:

[0199] According to the posture of the multi-eye camera in the current frame data packet and the posture of the multi-eye camera in the previous frame data packet, determine whether the posture change exceeds the posture difference threshold;

[0200] If the posture change exceeds the posture difference threshold, the current frame data packet is determined to be the data packet to be processed.

[0201] In a third aspect, an embodiment of the present disclosure provides a head mounted device, wherein the head mounted device is used to execute the data processing method as described in any one of the first aspects above.

[0202] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0203] The memory stores computer-executable instructions;

[0204] The processor executes the computer-executable instructions stored in the memory, so that the processor performs the data processing method as described in any one of the first aspects above.

[0205] In a fifth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the data processing method as described in any one of the first aspects above is implemented.

[0206] In a sixth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing method as described in the first aspect above.

[0207] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0208] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0209] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Splice the current frame multi-view observation images into a target image; Performing similarity matching between the target image and each historical image, and determining whether there is a similar frame image in each historical image that successfully matches the target image in similarity; wherein the historical image is obtained by splicing multiple historical observation images; If a similar frame image exists, the posture of the current frame is determined according to the pixel coordinates of the feature points of the multi-eye observation image of the current frame and the three-dimensional coordinates of the corresponding feature points in the similar frame image.

2. The method according to claim 1, characterized in that The method further comprises: When determining that the current frame data packet is a data packet to be processed, determining whether there is a target point by feature matching; wherein the data packet includes a multi-eye observation image acquired by a multi-eye camera of the head-mounted device and the position and posture of the multi-eye camera, and the target point is a 3D point corresponding to a feature point that appears in the multi-eye observation images of the previous N consecutive frames before the current frame and does not appear in the multi-eye observation image of the current frame; If it is determined that the target point exists, determining the three-dimensional coordinates of the target point according to the pose corresponding to the first N frames of multi-eye observation images and the pixel coordinates corresponding to the target point in the first N frames of multi-eye observation images; Correspondingly, after determining the posture of the current frame, the method further includes: updating the three-dimensional coordinates corresponding to each feature point in the multi-eye observation image of the current frame according to the three-dimensional coordinates of the target point, the posture of the current frame and the multi-eye observation images of the previous N frames.

3. The method according to claim 2, characterized in that A frame of multi-eye observation image includes multiple images collected by multi-eye cameras; determining whether there is a target point through feature matching includes: Extract feature points of multiple images corresponding to the current frame; For each image, feature matching is performed with the corresponding image in the previous frame; If the matching result corresponding to at least one image indicates that the target point exists, it is determined that the target point exists.

4. The method according to claim 2 or 3, characterized in that: The determining the three-dimensional coordinates of the target point according to the posture corresponding to the first N frames of multi-eye observation images and the pixel coordinates corresponding to the target point in the first N frames of multi-eye observation images includes: According to the posture corresponding to the first N frames of multi-eye observation images and the pixel coordinates corresponding to the target point in the first N frames of multi-eye observation images, a least squares model is constructed to obtain the three-dimensional coordinates of the target point.

5. The method according to claim 4, characterized in that The updating of the three-dimensional coordinates corresponding to each feature point in the multi-eye observation image of the current frame according to the three-dimensional coordinates of the target point, the posture of the current frame and the multi-eye observation images of the previous N frames includes: According to the three-dimensional coordinates of the target point, the target point is projected into each of the first N frames of multi-eye observation images, and the depth of the target point in each frame of the multi-eye observation image and the depth of the 3D points other than the target point projected in each frame of the multi-eye observation image in the corresponding observation image are calculated and the depth of the target point and the depth of the 3D points other than the target point are stored; According to the posture of the current frame and the depth corresponding to the similar frame image, the three-dimensional coordinates corresponding to each feature point in the current frame multi-eye observation image are updated.

6. The method according to claim 5, characterized in that The method further comprises: When the head mounted device is in an offline state, for each 3D point in the space, the posture of the multi-eye camera and the three-dimensional coordinates of each 3D point in the space are updated by optimizing the error of the pixel coordinates corresponding to each 3D point in the space; Among them, each 3D point in the space includes a 3D point corresponding to the multi-eye observation image of the current frame and a 3D point corresponding to the multi-eye historical observation image.

7. The method according to claim 2 or 3, characterized in that: The method further comprises: According to the posture of the multi-eye camera in the current frame data packet and the posture of the multi-eye camera in the previous frame data packet, determine whether the posture change exceeds the posture difference threshold; If the posture change exceeds the posture difference threshold, the current frame data packet is determined to be the data packet to be processed.

8. A data processing device, characterized in that: The device comprises: The first processing module is used to stitch the multi-viewing images of the current frame into a target image; The second processing module is used to perform similarity matching between the target image and each historical image, and determine whether there is a similar frame image in each historical image that successfully matches the target image; wherein the historical image is obtained by splicing multiple historical observation images; The third processing module is used to determine the posture of the current frame according to the pixel coordinates of the feature points of the multi-eye observation image of the current frame and the three-dimensional coordinates of the corresponding feature points in the similar frame image when it is determined that there is a similar frame image.

9. A head mounted device, characterized in that: The head mounted device is used to execute the data processing method as described in any one of claims 1-7.

10. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the data processing method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the data processing method according to any one of claims 1 to 7 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images