Image processing method and electronic equipment

Abstract

The invention provides an image processing method and electronic equipment. The image processing method comprises the following steps: acquiring a time sequence phase diagram sequence adopted when a laser projector projects structured light to a target object; acquiring a multi-view image obtained by acquiring the target object by the multi-view camera; determining a first disparity map of the multi-view image; the time sequence phase diagram sequence, the multi-view image and the first disparity map are input into the pre-trained disparity prediction model for disparity prediction, the second disparity map of the multi-view image is obtained, the time sequence phase diagrams are fused, and therefore the precision of the obtained second disparity map can be improved. Besides, through the prediction of the parallax prediction model, the influence of the material and environment of the object on the second parallax image can be avoided, and the accuracy of the second parallax image is further improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method and electronic device. Background Art

[0002] At present, passive binocular vision is usually used to achieve three-dimensional reconstruction. Passive binocular vision uses two cameras to simultaneously capture images of the same object from different angles, and determines the three-dimensional point cloud of the object by calculating the parallax information between the two images.

[0003] Among them, the use of passive binocular vision cannot accurately determine the disparity information, which leads to the problem of low accuracy of three-dimensional reconstruction of objects. Summary of the invention

[0004] Various aspects of the present disclosure provide an image processing method and an electronic device to solve the technical problem that disparity information cannot be accurately determined.

[0005] The first aspect of the embodiment of the present disclosure provides an image processing method, which is applied to a three-dimensional reconstruction system. The three-dimensional reconstruction system includes: a laser projector and a multi-eye camera. The image processing method includes:

[0006] Acquire a temporal phase image sequence used by a laser projector when projecting structured light onto a target object;

[0007] Acquire a multi-eye image of a target object captured by a multi-eye camera;

[0008] Determine a first disparity map of the multi-view image;

[0009] A temporal phase image sequence, a multi-view image and a first disparity map are input into a pre-trained disparity prediction model to perform disparity prediction, thereby obtaining a second disparity map of the multi-view image.

[0010] A third aspect of the embodiments of the present disclosure provides an image processing device, which is applied to a three-dimensional reconstruction system. The three-dimensional reconstruction system includes: a laser projector and a multi-eye camera. The image processing device includes:

[0011] A first acquisition module is used to acquire a timing phase image sequence used by a laser projector when projecting structured light onto a target object;

[0012] The second acquisition module is used to acquire a multi-eye image of the target object acquired by the multi-eye camera;

[0013] A determination unit, configured to determine a first disparity map of a multi-view image;

[0014] The prediction unit is used to input the temporal phase image sequence, the multi-view image and the first disparity map into a pre-trained disparity prediction model to perform disparity prediction and obtain a second disparity map of the multi-view image.

[0015] A third aspect of an embodiment of the present disclosure provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the image processing method of the first aspect is implemented when the processor executes the computer program.

[0016] A fourth aspect of an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the image processing method of the first aspect.

[0017] A fifth aspect of an embodiment of the present disclosure provides a computer program product, which includes: a computer program, which is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the image processing method of the first aspect.

[0018] The disclosed embodiment is applied in a scene, by obtaining a sequence of time-series phase images used when a laser projector projects structured light onto a target object; obtaining a multi-eye image of the target object acquired by a multi-eye camera; determining a first disparity map of the multi-eye image; inputting the sequence of time-series phase images, the multi-eye image and the first disparity map into a pre-trained disparity prediction model for disparity prediction, and obtaining a second disparity map of the multi-eye image. The disclosed embodiment fuses the time-series phase map, thereby improving the accuracy of the obtained second disparity map. In addition, through the prediction of the disparity prediction model, the influence of the material and environment of the object on the second disparity map can be avoided, further improving the accuracy of the second disparity map. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0020] Figure 1 An application scenario diagram of an image processing method provided by an exemplary embodiment of the present disclosure;

[0021] Figure 2 A flowchart of an image processing method provided for an exemplary embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of a timing phase diagram provided for an exemplary embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of a disparity prediction model provided for an exemplary embodiment of the present disclosure;

[0024] Figure 5A flowchart of another disparity prediction model training method provided for an exemplary embodiment of the present disclosure;

[0025] Figure 6 A flowchart of three-dimensional reconstruction steps provided for an exemplary embodiment of the present disclosure;

[0026] Figure 7 A structural block diagram of an image processing device provided for an exemplary embodiment of the present disclosure;

[0027] Figure 8 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure will be clearly and completely described below in combination with the specific embodiments of the present disclosure and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0029] In the related technology, the three-dimensional reconstruction of objects is mainly achieved through two methods: passive binocular vision or active imaging. Among them, the active imaging method is a technology that uses a laser projector to emit structured light spots (such as stripes or grid-shaped light) to the object, and collects the light spots reflected or deformed from the object surface to determine the three-dimensional point cloud of the object. This three-dimensional reconstruction method is easily affected by the surface material of the object, resulting in low three-dimensional reconstruction accuracy. In addition, the use of passive binocular vision is also easily affected by the surface material and / or environment (such as highlights), and it is impossible to accurately determine the parallax information, resulting in low three-dimensional reconstruction accuracy.

[0030] Based on the above problems, firstly, the present invention is based on passive binocular vision and integrates the temporal phase map of active imaging, so as to improve the accuracy of the obtained second disparity map. In addition, through the prediction of the disparity prediction model, the influence of the material and environment of the object on the second disparity map can be avoided, further improving the accuracy of the second disparity map.

[0031] In addition, an application scenario of the embodiment of the present disclosure is as follows Figure 1 ,exist Figure 1 , a 3D reconstruction system 10 is shown, the 3D reconstruction system 10 comprises: a laser projector 11 and a multi-eye camera 12, wherein the multi-eye camera 12 comprises a plurality of cameras, such as Figure 1The first camera 121 and the second camera 122 in the multi-lens camera 12. The laser projector 11 emits structured light to the target object, and the structured light is emitted by the laser projector 11 according to a preset timing phase diagram sequence, and the timing phase diagram sequence includes: multiple fringe diagrams or multiple grid diagrams. The structured light is deformed on the target object, and then each camera of the multi-lens camera 12 collects images for the target object, and the images contain light spots projected by the structured light on the target object.

[0032] in, Figure 1 This is just an exemplary application scenario, and the embodiment of the present disclosure can be applied to any 3D reconstruction scenario of an object. The embodiment of the present disclosure does not limit the specific application scenario.

[0033] Figure 2 A flowchart of an image processing method provided by an exemplary embodiment of the present disclosure, the image processing method is applied to Figure 1 The three-dimensional reconstruction system shown specifically includes the following steps:

[0034] S201, obtaining a timing phase diagram sequence used by a laser projector to project structured light onto a target object.

[0035] The time-series phase image sequence includes: multiple time-series phase images, and the multiple time-series phase images are projected onto the target object in time sequence. The time-series phase image is in stripe or grid shape. Figure 3 , showing a striped timing phase diagram.

[0036] In the disclosed embodiment, the timing phase diagram is pre-encoded and can be directly obtained.

[0037] S202, obtaining a multi-eye image of the target object captured by the multi-eye camera.

[0038] The multi-eye camera includes at least two cameras. The multi-eye images include: images collected by different cameras.

[0039] For example, refer to Figure 1 The multi-eye camera is a binocular camera, including two cameras, namely a first camera and a second camera. The multi-eye image includes: an image captured by the first camera and an image captured by the second camera.

[0040] In the disclosed embodiment, the laser projector projects structured light onto the target object according to a projection timing phase diagram sequence. The structured light is deformed on the target object due to the structure of the target object. The binocular camera collects the deformed structured light on the target object to obtain a multi-eye image.

[0041] S203: Determine a first disparity map of the multi-view image.

[0042] In an embodiment of the present disclosure, if the multi-eye camera includes two cameras, namely the first camera and the second camera, the multi-eye image includes: an image captured by the first camera and an image captured by the second camera, and the first disparity map is a disparity map of the image captured by the first camera and the image captured by the second camera. If the number of cameras included in the multi-eye camera is greater than 2, the multi-eye image includes: images captured by each camera, and the first disparity map is a disparity map of images captured by any two cameras. For example, the multi-eye camera includes: three cameras, namely the first camera, the second camera and the third camera, then the first disparity map can be a disparity map of the image captured by the first camera and the image captured by the second camera, or the first disparity map is a disparity map of the image captured by the second camera and the image captured by the third camera, or the first disparity map is a disparity map of the image captured by the first camera and the image captured by the third camera.

[0043] In the present disclosure, the first disparity map is determined based on structured light, wherein the first disparity map may be determined by using any technology, and the present disclosure does not limit the method for determining the first disparity map.

[0044] S204, inputting the temporal phase image sequence, the multi-view image and the first disparity map into a pre-trained disparity prediction model to perform disparity prediction, and obtaining a second disparity map of the multi-view image.

[0045] The multi-view image includes: images captured by any two cameras respectively; the first disparity map is a disparity map of the multi-view image determined by any technology; and the second disparity map is a disparity image of the multi-view image output by a disparity prediction model.

[0046] Specifically, refer to Figure 4 The disparity prediction model 40 includes: a disparity prediction unit 41 and a regression unit 42.

[0047] The first disparity map includes: the first disparity of each pixel in the plurality of pixels, and the second disparity map includes: the second disparity of each pixel in the plurality of pixels. It can be understood that the first disparity map includes a plurality of first disparities, and the pixels and the first disparities correspond one to one. The second disparity map includes: a plurality of second disparities, and the pixels and the second disparities correspond one to one. For example, the first disparity map is represented by P1, and the first disparity of the jth pixel can be represented by g j The second disparity map is represented by P2, and the second disparity of the j-th pixel can be represented by T j Represents, where j ranges from 1 to M, M is a positive integer, representing the number of multiple pixels.

[0048] Furthermore, a temporal phase image sequence, a multi-view image and a first disparity map are input into a pre-trained disparity prediction model for disparity prediction to obtain a second disparity map of the multi-view image, including: performing disparity prediction on the temporal phase image sequence and the multi-view image through a disparity prediction unit to obtain multiple predicted disparities for each of a plurality of pixels; for each pixel, performing regression processing on the multiple predicted disparities based on the first disparity through a regression unit to obtain a second disparity for each pixel.

[0049] Among them, the disparity prediction unit outputs multiple predicted disparities for each pixel. It can be understood that the i-th predicted disparity corresponding to the j-th pixel can be d ji Represents, where i ranges from 1 to N, and N is the number of multiple predicted disparities.

[0050] Furthermore, for the jth pixel, the regression unit is based on the first disparity (g j ) for multiple predicted disparities (d ji ) is regressed to obtain the second disparity (T j ). Understandably, T j is one of multiple predicted disparities.

[0051] Among them, the multi-eye camera includes a binocular camera, and the multi-eye image includes a first image and a second image respectively captured by each camera of the binocular camera. For example, the binocular camera includes a first camera and a second camera, then the multi-eye image includes: a first image captured by the first camera and a second image captured by the second camera.

[0052] Reference Figure 4 The disparity prediction unit 41 includes: a feature extraction layer 411 and a disparity prediction layer 412. The disparity prediction unit performs disparity prediction on the temporal phase map sequence and the multi-view image to obtain multiple predicted disparities of each pixel in the multiple pixels, including: performing feature extraction on the first image through the feature extraction layer to obtain first image features; performing feature extraction on the second image through the feature extraction layer to obtain second image features; performing feature extraction on the temporal phase map sequence through the feature extraction layer to obtain third image features; processing the first image features, the second image features and the third image features through the disparity prediction layer to obtain multiple predicted disparities for each pixel in the multiple pixels.

[0053] Furthermore, the disparity prediction layer includes: a merging sublayer and a cost feature sublayer. The first image feature, the second image feature and the third image feature are processed by the disparity prediction layer to obtain multiple predicted disparities for each of the multiple pixels, including: merging the first image feature and the third image feature by the merging sublayer to obtain the fourth image feature; merging the second image feature and the third image feature by the merging sublayer to obtain the fifth image feature; predicting the disparity of the fourth image feature and the fifth image feature by the cost feature sublayer to obtain multiple predicted disparities for each of the multiple pixels.

[0054] In the embodiment of the present disclosure, after projecting structured light onto the target object according to the temporal phase map sequence, the collected first image and second image can well reflect the shape change of the target object under the projection pattern, and imply the geometric features of the target object. The temporal phase map sequence is input into the disparity prediction unit for processing, so that the temporal phase map sequence can be merged with the first image to obtain the fourth image feature, and the temporal phase map sequence can be merged with the second image to obtain the fifth image feature, which can effectively enhance the prediction ability of the target object and thereby improve the accuracy of the determined second disparity.

[0055] Among them, a regression unit is used to perform regression processing on multiple predicted disparities based on the first disparity to obtain a second disparity for each pixel, including: for each predicted disparity in the multiple predicted disparities, a cost value of the predicted disparity relative to the first disparity is determined by the regression unit; and the predicted disparity with the largest cost value is determined as the second disparity of the corresponding pixel.

[0056] Further, determining, by the regression unit, a cost value of the predicted disparity relative to the first disparity includes determining the cost value according to the following expression:

[0057]

[0058] Among them, f represents the cost value, λ and σ are preset coefficients, and d ji represents the i-th predicted disparity of the j-th pixel, j is an integer greater than 1, where i ranges from 1 to N, N represents the number of multiple predicted disparities of the j-th pixel, g j represents the first disparity of the j-th pixel.

[0059] In the open, λ is a positive number, 2σ 2 Greater than 1. Among them, the measured parallax d ji With the first parallax g j The larger the deviation, the smaller the cost value f, then it can be understood that the second parallax is equal to the first parallax g j The predicted disparity with the smallest deviation.

[0060] The above expression is used to determine the second disparity of each pixel, and then obtain the second disparity map.

[0061] In the disclosed embodiment, the disparity prediction model is a pre-trained network model, wherein the disparity prediction model can effectively extract implicit features of the temporal phase image sequence, the multi-view image and the first disparity map, thereby effectively improving the second disparity map of the multi-view image.

[0062] Furthermore, the disparity prediction model can integrate the imaging accuracy of active imaging and the imaging completeness of passive binocular vision to improve the accuracy of the second disparity map, so that the determination of the second disparity map is not affected by the environment and material. For example, in a high-light environment and / or when the target object is made of transparent material, the determined second disparity map is accurate.

[0063] Figure 5 A flowchart of a disparity prediction model training method provided for an exemplary embodiment of the present disclosure specifically includes the following steps:

[0064] S501, obtaining training samples.

[0065] The training samples include: sample multi-view images, sample time-series phase image sequences, and sample disparity images of sample multi-view images.

[0066] In the present disclosure, the sample multi-eye image, the sample time-series phase image sequence and the sample disparity image are obtained by simulation using a simulator. Figure 1 In the scenario of the 3D reconstruction system shown in FIG. , sample multi-eye images, sample time-series phase image sequences and sample disparity images are simulated. It can be understood that in Figure 2 The internal and external parameters of the three-dimensional reconstruction system used in the embodiment are consistent with the internal and external parameters used in the simulation training samples.

[0067] Furthermore, the sample multi-view image includes: a first sample image and a second sample image. The first sample image is obtained by simulating the first camera to capture the sample object, and the second sample image is obtained by simulating the simulator.

[0068] Specifically, for sample multi-view images, you need to use the material editor to build the material information of the sample object, and then use the ray tracing renderer for realistic rendering to generate realistic sample multi-view images. For example, for sample objects with transparent materials, you need to set BSDF (Bidirectional Scattering Distribution Function) information similar to that of glass materials to render sample multi-view images similar to the actual ones.

[0069] For the sample time-series phase image sequence, it is necessary to simulate the projection of the array structured light and integrate it into the ray tracing renderer, so as to generate the correct sample time-series phase image sequence. In practice, the required time-series coded fringe image sequence is added to the renderer light source in the form of projected texture using the projection texture technology, and the required sample time-series phase image sequence can be generated.

[0070] For the sample disparity map corresponding to the sample multi-view image, it is necessary to convert the depth map generated by the renderer into the corresponding sample disparity map. Because the renderer can automatically obtain the depth map corresponding to the virtual image, and since we already know the internal and external parameters of the binocular camera, we can directly calculate the sample disparity corresponding to each pixel based on the depth map and the internal and external parameters to obtain the sample disparity map, where the sample disparity map includes multiple sample disparities.

[0071] In reality, there is a problem of low efficiency in obtaining training samples of the disparity prediction model by actual acquisition. The present disclosure adopts a simulation method to simulate and obtain training samples, thereby improving the efficiency of obtaining training samples.

[0072] S502, training a disparity prediction model according to the sample multi-view images, the sample temporal phase map sequence and the sample disparity map, and obtaining a trained disparity prediction model when the disparity prediction model meets a preset training condition.

[0073] Among them, the first disparity map includes: the first sample disparity of each pixel in multiple pixels, the disparity prediction model includes: a disparity prediction unit and a regression unit, and the disparity prediction model is trained according to the sample multi-view image, the sample time-series phase map sequence and the sample disparity map, including: performing disparity prediction on the sample time-series phase map sequence and the sample multi-view image through the disparity prediction unit to obtain multiple predicted sample disparities for each pixel; for each predicted sample disparity in the multiple predicted sample disparities, determining the predicted cost value of the predicted sample disparity relative to the first sample disparity through the regression unit; and updating the network parameters of the disparity prediction unit according to the predicted cost value and the current network parameters of the disparity prediction unit.

[0074] In an embodiment of the present disclosure, a sample temporal phase map sequence and a sample multi-view image are input into a disparity prediction model for training, so that the disparity prediction model can learn the fusion features of the sample temporal phase map sequence and the sample multi-view image, which can effectively enhance the disparity prediction model's perception of the object and more accurately determine the second disparity.

[0075] Wherein, updating the network parameters of the disparity prediction unit according to the predicted cost value and the current network parameters of the disparity prediction unit includes: determining the product of the maximum predicted cost value and the current network parameters of the disparity prediction unit as the updated network parameters.

[0076] In the present disclosure, the sample disparity map is integrated into the regression unit to determine the prediction cost value, thereby guiding the disparity prediction model to be trained in the right direction, better optimizing the disparity prediction model, and improving the accuracy of the disparity prediction model in predicting the second disparity.

[0077] Specifically, the predicted cost value also uses the above expression Determine, the specific determination method refers to the determination of the cost value mentioned above, which will not be repeated here.

[0078] It can be understood that each pixel corresponds to multiple prediction cost values. For each pixel, the maximum prediction cost value f is determined. m The product of the current network parameters of the disparity prediction unit is the updated network parameters.

[0079] Furthermore, the current network parameter for the j-th pixel is represented by C(j), and the updated network parameter for the j-th pixel is represented by C′(j), where C′(j)=f m C(j). The updated network parameter C′(j) is used to replace the current network parameter C(j) for subsequent training of the disparity prediction unit.

[0080] In addition, the preset training conditions include the number of training times or the convergence requirement of the disparity prediction model. In the present disclosure, the preset training conditions are not limited, and the trained visual prediction model can accurately predict the second disparity.

[0081] In the present disclosure, by fusing the sample temporal phase map sequence and sample disparity map of the planar array structured light, the disparity prediction model training can be guided to obtain a more accurate second disparity, while the imaging integrity under special materials can be ensured, thereby improving the robustness of the trained disparity prediction model.

[0082] In summary, the sample time-series phase map sequence used for structured light projection is fused into the image channel of the multi-eye sample image as the basic image feature. At the same time, the sample disparity map is fused into the regression unit to enhance the prediction of the second disparity, and then the disparity prediction model that can accurately predict the second disparity can be trained.

[0083] Further, refer to Figure 6 , after determining the second disparity map, the following steps are also included:

[0084] S601, obtaining the baseline length and camera intrinsic parameters of the multi-eye camera.

[0085] The multi-eye camera includes: multiple cameras, the camera internal parameter is represented by F, and the baseline length is represented by B to represent the distance between any two cameras.

[0086] S602, determining a depth image of the target object according to the baseline length, the camera intrinsic parameters and the second disparity map.

[0087] The depth image includes: the depth value of each pixel in the plurality of pixels. The specific expression for determining the depth value is:

[0088] D j =F×B / T j

[0089] In the above expression, T j represents the second disparity of the j-th pixel in the second disparity map. j Represents the depth value of the jth pixel in the depth map.

[0090] S603, determining a three-dimensional point cloud of the target object according to the depth image and the camera intrinsic parameters.

[0091] In the embodiment of the present disclosure, the depth image can be converted into a point cloud using the camera intrinsic parameter, thereby obtaining a three-dimensional point cloud of the target object. The present disclosure does not limit the method of converting the depth image into a point cloud.

[0092] In summary, the present application uses simulated training samples to train a highly robust disparity prediction model. Then, the trained disparity prediction model is used to perform disparity prediction on the target object's temporal phase image sequence, multi-eye images, and the first disparity map to obtain an accurate second disparity map. The second disparity map is used for three-dimensional reconstruction of the target object, thereby improving the accuracy of the three-dimensional reconstruction.

[0093] Reference Figure 7 , is a structural block diagram of an image processing device 70 provided by the present disclosure, and the image processing device 70 includes:

[0094] A first acquisition unit 71 is used to acquire a timing phase image sequence used by a laser projector when projecting structured light onto a target object;

[0095] The second acquisition unit 72 is used to acquire a multi-eye image of the target object acquired by the multi-eye camera;

[0096] A determining unit 73, configured to determine a first disparity map of the multi-view image;

[0097] The prediction unit 74 is used to input the temporal phase image sequence, the multi-view image and the first disparity map into the pre-trained disparity prediction model to perform disparity prediction to obtain a second disparity map of the multi-view image.

[0098] In an optional embodiment, the disparity prediction model includes: a disparity prediction unit and a regression unit; the first disparity map includes: a first disparity of each pixel in a plurality of pixels, the second disparity map includes: a second disparity of each pixel in a plurality of pixels, and the prediction unit 74 is specifically used to: perform disparity prediction on a temporal phase map sequence and a multi-view image through the disparity prediction unit to obtain a plurality of predicted disparities for each pixel in a plurality of pixels; for each pixel, perform regression processing on the plurality of predicted disparities based on the first disparity through the regression unit to obtain a second disparity for each pixel.

[0099] In an optional embodiment, the multi-eye camera includes a binocular camera, the multi-eye image includes: a second image respectively captured by each camera of the binocular camera, the disparity prediction unit includes: a feature extraction layer and a disparity prediction layer, and the prediction unit 74 performs disparity prediction on the time series phase image sequence and the multi-eye image through the disparity prediction unit to obtain multiple predicted disparities for each of the multiple pixels. The prediction unit 74 is specifically used to: extract features from the first image through the feature extraction layer to obtain first image features; extract features from the second image through the feature extraction layer to obtain second image features; extract features from the time series phase image sequence through the feature extraction layer to obtain third image features; and process the first image features, the second image features, and the third image features through the disparity prediction layer to obtain multiple predicted disparities for each of the multiple pixels.

[0100] In an optional embodiment, the disparity prediction layer includes: a merging sublayer and a cost feature sublayer, and the first image feature, the second image feature and the third image feature are processed by the disparity prediction layer to obtain multiple predicted disparities for each of the multiple pixels, including: merging the first image feature and the third image feature by the merging sublayer to obtain the fourth image feature; merging the second image feature and the third image feature by the merging sublayer to obtain the fifth image feature; predicting the disparity of the fourth image feature and the fifth image feature by the cost feature sublayer to obtain multiple predicted disparities for each of the multiple pixels.

[0101] In an optional embodiment, when the prediction unit 74 performs regression processing on multiple predicted disparities based on the first disparity through the regression unit to obtain the second disparity of each pixel, it is specifically configured to: for each predicted disparity in the multiple predicted disparities, determine the cost value of the predicted disparity relative to the first disparity through the regression unit; and determine the predicted disparity with the largest cost value as the second disparity of the corresponding pixel. In an optional embodiment, the prediction unit 74 is specifically configured to determine the cost value according to the following expression:

[0102]

[0103] Among them, f represents the cost value, λ and σ are preset coefficients, and d jirepresents the i-th predicted disparity of the j-th pixel, j is an integer greater than 1, where i ranges from 1 to N, N represents the number of multiple predicted disparities of the j-th pixel, g j represents the first disparity of the j-th pixel.

[0104] In an optional embodiment, it also includes: a training unit (not shown), which is specifically used to train the disparity prediction model by the following method: obtaining training samples, the training samples including: sample multi-view images, sample time-series phase map sequences, and sample disparity maps of sample multi-view images; training the disparity prediction model according to the sample multi-view images, the sample time-series phase map sequences, and the sample disparity maps, and obtaining a trained disparity prediction model when the disparity prediction model meets preset training conditions.

[0105] In an optional embodiment, the sample multi-eye image, the sample time-series phase image sequence and the sample disparity image are obtained by simulation using a simulator.

[0106] In an optional embodiment, the first disparity map includes: a first sample disparity of each pixel among multiple pixels, the disparity prediction model includes: a disparity prediction unit and a regression unit, and when the training unit trains the disparity prediction model according to the sample multi-view image, the sample time-series phase map sequence and the sample disparity map, the training unit is specifically used to: perform disparity prediction on the sample time-series phase map sequence and the sample multi-view image through the disparity prediction unit to obtain multiple predicted sample disparities for each pixel; for each predicted sample disparity among the multiple predicted sample disparities, determine the predicted cost value of the predicted sample disparity relative to the first sample disparity through the regression unit; and update the network parameters of the disparity prediction unit according to the predicted cost value and the current network parameters of the disparity prediction unit.

[0107] In an optional embodiment, when the training unit updates the network parameters of the disparity prediction unit according to the predicted cost value and the current network parameters of the disparity prediction unit, it is specifically used to: determine the product of the maximum predicted cost value and the current network parameters of the disparity prediction unit as the updated network parameters.

[0108] In an optional embodiment, it also includes: a reconstruction unit, used to obtain the baseline length and camera intrinsic parameters of the multi-eye camera; determine the depth image of the target object based on the baseline length, the camera intrinsic parameters and the second disparity map; determine the three-dimensional point cloud of the target object based on the depth image and the camera intrinsic parameters.

[0109] The image processing device provided by the present disclosure can implement the above-mentioned image processing method. Please refer to the above for details and will not be repeated here.

[0110] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and are only used to distinguish between different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "second", "first", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit "second" and "first" to be different types.

[0111] Figure 8 A schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present disclosure. Figure 8 As shown, the electronic device 80 includes: a processor 81, and a memory 82 communicatively connected to the processor 81, and the memory 82 stores computer-executable instructions.

[0112] Among them, the processor executes the computer execution instructions stored in the memory to implement the image processing method and / or defect detection method provided by any of the above-mentioned method embodiments, and the specific functions and technical effects that can be achieved are not repeated here.

[0113] The embodiment of the present disclosure further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement any of the above methods.

[0114] The embodiments of the present disclosure also provide a computer program product, which includes: a computer program, which is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes any of the above methods.

[0115] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0116] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0117] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0118] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform some steps of the methods of various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0119] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0120] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0121] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: Applied to a three-dimensional reconstruction system, the three-dimensional reconstruction system includes: a laser projector and a multi-eye camera, and the image processing method includes: Acquire a timing phase image sequence used by the laser projector when projecting structured light onto a target object; Acquire a multi-eye image of a target object acquired by the multi-eye camera; Determining a first disparity map of the multi-view image; The temporal phase image sequence, the multi-view image and the first disparity map are input into a pre-trained disparity prediction model to perform disparity prediction, so as to obtain a second disparity map of the multi-view image.

2. The image processing method according to claim 1, characterized in that: The disparity prediction model includes: a disparity prediction unit and a regression unit; the first disparity map includes: a first disparity of each pixel in a plurality of pixels, the second disparity map includes: a second disparity of each pixel in the plurality of pixels, and the temporal phase map sequence, the multi-view image and the first disparity map are input into the pre-trained disparity prediction model for disparity prediction to obtain a second disparity map of the multi-view image, including: Performing disparity prediction on the temporal phase image sequence and the multi-view image by the disparity prediction unit to obtain a plurality of predicted disparities of each pixel in the plurality of pixels; For each pixel, the regression unit performs regression processing on the multiple predicted disparities based on the first disparity to obtain a second disparity for each pixel.

3. The image processing method according to claim 2, characterized in that: The multi-eye camera includes a binocular camera, the multi-eye image includes: a first image and a second image respectively collected by each camera of the binocular camera, the disparity prediction unit includes: a feature extraction layer and a disparity prediction layer, and the disparity prediction unit performs disparity prediction on the temporal phase image sequence and the multi-eye image to obtain multiple predicted disparities of each pixel in the multiple pixels, including: Performing feature extraction on the first image through the feature extraction layer to obtain first image features; Extract features from the second image using the feature extraction layer to obtain second image features; Performing feature extraction on the time series phase image sequence through the feature extraction layer to obtain a third image feature; The first image feature, the second image feature, and the third image feature are processed by the disparity prediction layer to obtain a plurality of predicted disparities for each of the plurality of pixels.

4. The image processing method according to claim 3, characterized in that: The disparity prediction layer includes: a merging sublayer and a cost feature sublayer, and the first image feature, the second image feature and the third image feature are processed by the disparity prediction layer to obtain a plurality of predicted disparities of each pixel in the plurality of pixels, including: Merging the first image feature and the third image feature through the merging sublayer to obtain a fourth image feature; Merging the second image feature and the third image feature through the merging sublayer to obtain a fifth image feature; The disparity between the fourth image feature and the fifth image feature is predicted by the cost feature sublayer to obtain a plurality of predicted disparities of each pixel in the plurality of pixels.

5. The image processing method according to any one of claims 2 to 4, characterized in that: The step of performing regression processing on the plurality of predicted disparities based on the first disparity by the regression unit to obtain a second disparity for each pixel includes: For each predicted disparity among a plurality of predicted disparities, determining, by the regression unit, a cost value of the predicted disparity relative to the first disparity; The predicted disparity with the maximum cost value is determined as the second disparity of the corresponding pixel.

6. The image processing method according to claim 5, characterized in that: The determining, by the regression unit, a cost value of the predicted disparity relative to the first disparity includes determining the cost value according to the following expression: Wherein, f represents the cost value, λ and σ are preset coefficients, and d ji represents the i-th predicted disparity of the j-th pixel, j is an integer greater than 1, where i ranges from 1 to N, N represents the number of multiple predicted disparities of the j-th pixel, g j represents the first disparity of the j-th pixel.

7. The image processing method according to claim 1, characterized in that: The disparity prediction model is trained using the following method: Acquire training samples, the training samples comprising: sample multi-view images, sample time-series phase image sequences, and sample disparity maps of the sample multi-view images; The disparity prediction model is trained according to the sample multi-view images, the sample temporal phase map sequence and the sample disparity map, and a trained disparity prediction model is obtained when the disparity prediction model meets a preset training condition.

8. The image processing method according to claim 7, characterized in that: The sample multi-eye image, the sample time-series phase image sequence and the sample disparity image are obtained by simulation using a simulator.

9. The image processing method according to claim 7, characterized in that: The first disparity map includes: a first sample disparity of each pixel in a plurality of pixels, the disparity prediction model includes: a disparity prediction unit and a regression unit, and the disparity prediction model is trained according to the sample multi-view image, the sample temporal phase map sequence and the sample disparity map, including: Performing disparity prediction on the sample temporal phase image sequence and the sample multi-view image by the disparity prediction unit to obtain a plurality of predicted sample disparities for each pixel; For each predicted sample disparity among a plurality of predicted sample disparities, determining, by the regression unit, a prediction cost value of the predicted sample disparity relative to the first sample disparity; The network parameters of the disparity prediction unit are updated according to the prediction cost value and the current network parameters of the disparity prediction unit.

10. The image processing method according to claim 9, characterized in that: The updating of the network parameters of the disparity prediction unit according to the predicted cost value and the current network parameters of the disparity prediction unit includes: The product of the maximum prediction cost value and the current network parameter of the disparity prediction unit is determined as the updated network parameter.

11. The image processing method according to any one of claims 1 to 4, characterized in that: Also includes: Obtaining the baseline length and camera intrinsic parameters of the multi-eye camera; Determining a depth image of the target object according to the baseline length, the camera intrinsic parameter and the second disparity map; A three-dimensional point cloud of the target object is determined according to the depth image and the camera intrinsic parameters.

12. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the image processing method according to any one of claims 1 to 11 is implemented when the processor executes the computer program.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the image processing method according to any one of claims 1 to 11.

Images