Data processing method and device, computer equipment and storage medium

Abstract

The invention discloses a data processing method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining image data, collected by a plurality of image collection devices, of a target object at different visual angle positions, and enabling the target object to be irradiated by a color light source and an infrared light source at different visual angle positions; the wavelength range and the irradiation range of light emitted by the color light source and the infrared light source are matched with the spatial range formed by the visual angle positions, the plurality of image acquisition devices comprise image acquisition devices at different visual angle positions in the space where the target object is located, and the image data comprise color image data and infrared image data; determining depth image data of the target object based on the infrared image data of the different view angle positions; determining three-dimensional point cloud data corresponding to the target object based on the depth image data; and generating a three-dimensional model corresponding to the target object based on the color image data and the three-dimensional point cloud data. The method can improve the reconstruction effect of the three-dimensional model.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Patent Application No. 63 / 650,303, filed May 21, 2024, entitled “MULTI-VIEW ILLUMINATION CAPTURING AND RECONSTRUCTION SYSTEM”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of image processing technology, and more specifically, to a data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0004] 3D reconstruction is an important research area in computer graphics and computer vision, with wide applications in many fields. In particular, 3D reconstruction of objects such as the human body is widely used in motion analysis, visual effects, virtual reality, e-commerce, and other fields, possessing high research and commercial value. However, the quality of 3D reconstruction of objects such as the human body currently needs improvement. Summary of the Invention

[0005] This application proposes a data processing method, apparatus, computer equipment, and storage medium that can improve the effect of 3D reconstruction.

[0006] In a first aspect, embodiments of this application provide a data processing method, the method comprising: acquiring image data from multiple image acquisition devices at different viewpoint positions of a target object, wherein the target object is illuminated by a color light source and an infrared light source at the different viewpoint positions, the wavelength range and illumination range of the light emitted by the color light source and the infrared light source match the spatial range formed by the viewpoint positions, the multiple image acquisition devices including image acquisition devices at different viewpoint positions in the space where the target object is located, and the image data including color image data and infrared image data; determining depth image data of the target object based on the infrared image data at the different viewpoint positions; determining three-dimensional point cloud data corresponding to the target object based on the depth image data; and generating a three-dimensional model corresponding to the target object based on the color image data and the three-dimensional point cloud data.

[0007] In a second aspect, an embodiment of the present application provides a data processing apparatus, the apparatus comprising: an image acquisition module, a depth acquisition module, a point cloud acquisition module, and a model generation module, wherein the image acquisition module is configured to acquire image data of different view positions collected by a plurality of image acquisition devices for a target object, the target object being irradiated by color light sources and infrared light sources at the different view positions, the wavelength range and irradiation range of the light emitted by the color light sources and the infrared light sources matching a spatial range formed by the view positions, the plurality of image acquisition devices comprising image acquisition devices at different view positions in a space where the target object is located, and the image data comprising color image data and infrared image data; the depth acquisition module is configured to determine depth image data of the target object based on the infrared image data at different view positions; the point cloud acquisition module is configured to determine three-dimensional point cloud data corresponding to the target object based on the depth image data; and the model generation module is configured to generate a three-dimensional model corresponding to the target object based on the color image data and the three-dimensional point cloud data.

[0008] In a third aspect, an embodiment of the present application provides a computer device, comprising: one or more processors; a memory; and one or more application programs stored in the memory and configured to be executed by the one or more processors, the one or more application programs being configured to execute the data processing method provided in the first aspect.

[0009] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing program code, the program code being executable by a processor to execute the data processing method provided in the first aspect.

[0010] The scheme provided in the application obtains image data of different view positions collected by a plurality of image acquisition devices for a target object, the target object is irradiated by color light sources and infrared light sources at different view positions, the wavelength range and irradiation range of light emitted by the color light sources and the infrared light sources match the spatial range formed by the view positions, the plurality of image acquisition devices include image acquisition devices at different view positions in a space where the target object is located, the image data includes color image data and infrared image data; depth image data of the target object is determined based on the infrared image data at different view positions; three-dimensional point cloud data corresponding to the target object is determined based on the depth image data; and a three-dimensional model corresponding to the target object is generated based on the color image data and the three-dimensional point cloud data. Thus, the target object for which a three-dimensional model needs to be reconstructed can be irradiated by color light sources and infrared light sources at a plurality of different view positions, and the wavelength range of light emitted by the light sources matches the spatial range co-promoted by the plurality of view positions, so that the lighting effect on the target object can be ensured, and then after three-dimensional reconstruction is performed according to information of a plurality of view positions of the target object, the effect of three-dimensional reconstruction can be improved, and the quality of the three-dimensional model can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 A schematic diagram of a fixed frame in a three-dimensional reconstruction system provided by an embodiment of the present application is shown.

[0013] Figure 2 A schematic diagram of an application scenario of a three-dimensional reconstruction system provided by an embodiment of the present application is shown.

[0014] Figure 3 An installation schematic diagram of an image acquisition device and a light source module provided by an embodiment of the present application is shown.

[0015] Figure 4 A flowchart of a data processing method according to an embodiment of the present application is shown.

[0016] Figure 5 A flowchart of a data processing method according to another embodiment of the present application is shown.

[0017] Figure 6 A flowchart of a data processing method according to still another embodiment of the present application is shown.

[0018] Figure 7A flow chart of a data processing method according to another embodiment of the present application is shown.

[0019] Figure 8 A flow chart of a data processing method according to yet another embodiment of the present application is shown.

[0020] Figure 9 An effect diagram provided by an embodiment of the present application is shown.

[0021] Figure 10 An effect diagram provided by an embodiment of the present application is shown.

[0022] Figure 11 An effect diagram provided by an embodiment of the present application is shown.

[0023] Figure 12 An effect diagram provided by an embodiment of the present application is shown.

[0024] Figure 13 An effect diagram provided by an embodiment of the present application is shown.

[0025] Figure 14 A block diagram of a data processing apparatus according to an embodiment of the present application is shown.

[0026] Figure 15 is a block diagram of a computer device for performing a data processing method according to an embodiment of the present application.

[0027] Figure 16 is a storage unit for storing or carrying program code for implementing a data processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] To enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application.

[0029] Three-dimensional reconstruction refers to establishing a mathematical model suitable for computer identification and processing for a three-dimensional object, which is usually constructed by two-dimensional images under corresponding lighting conditions. By using three-dimensional reconstruction, a real entity object in reality can be "moved to" a virtual space, so that a user can see a realistic entity object.

[0030] When reconstructing a model of a target object such as a human body, although the three-dimensional model can be reconstructed by image acquisition of the object for which the three-dimensional model is to be constructed, the three-dimensional model obtained by reconstruction has poor accuracy and cannot restore the texture information of the target object well.

[0031] To solve the above problems, the data processing method, device, computer device and storage medium provided in the embodiments of the present application can realize irradiation of a target object requiring three-dimensional model reconstruction by a plurality of color light sources and infrared light sources at different viewing angle positions, and the wavelength range of the light emitted by the light sources matches the spatial range co-promoted by the plurality of viewing angle positions, so as to ensure the lighting effect of the target object, and then after three-dimensional reconstruction according to the information of the plurality of viewing angle positions of the target object, the effect of three-dimensional reconstruction can be improved, and the quality of the three-dimensional model can be improved. The specific data processing method is described in detail in the subsequent embodiments.

[0032] First, the scene related to the embodiments of the present application is introduced.

[0033] The scene related to the embodiments of the present application includes a three-dimensional reconstruction system, which includes a fixed frame, a light source module, an image acquisition device and a computer device. As shown in Figure 1 The fixed frame 10 can include a plurality of connecting strips 11, and every three connecting strips 11 form a triangular structure to ensure stability. The plurality of connecting strips 11 can constitute a spherical structure, and one side of the fixed frame is used to be fixed to the ground. The light source module and the image acquisition device can be arranged on at least part of the plurality of connecting strips 11, so that for any position inside the fixed frame, there are light source modules and image acquisition devices at a plurality of viewing angle positions. The computer device can be connected with the above image acquisition device to process the images collected by the image acquisition device, and then realize three-dimensional reconstruction.

[0034] In some embodiments, the light source module and the image acquisition device on the connecting strip 11 above can be used as a group of data acquisition devices. In each group of data acquisition devices, the image acquisition device can include a color camera and an infrared camera. The color camera can be used to collect color images with color information under visible light, and the infrared camera can be used to collect infrared light images containing more texture information. The light source module can include a color light source and an infrared light source. The color light source is used to provide visible light, and the infrared light source is used to provide infrared light. The infrared light source can be an infrared projector, such as a dot matrix infrared projector. The infrared light source can be used in cooperation with the infrared camera to obtain relevant depth information when three-dimensional reconstruction of the physical object is performed. The wavelength range and irradiation range of the light emitted by the color light source and the infrared light source in the light source module can match the spatial range of the internal space formed by the fixed frame, that is, the wavelength range and irradiation range of the light emitted by the color light source and the infrared light source are determined according to the size of the internal space formed by the fixed frame, that is, the wavelength range and irradiation range of the light emitted by the color light source and the infrared light source match the spatial range formed by the above each viewing angle position.

[0035] In a possible implementation, the above color camera and infrared camera can both be industrial cameras. In the case that the computer device is equipped with a Peripheral Component Interconnect Express (PCIE) interface and a Solid State Disk (SSD), the above color camera and infrared camera can capture 4K images at a high frame rate (e.g., 60 Hz), so as to make the final constructed three-dimensional model have more structural and textural details in cooperation with the visible color light and infrared light.

[0036] In a possible implementation, one computer device can be allocated to each set of the above data acquisition devices, and the computer device can be used to store the images captured by the image acquisition devices. Since the amount of data is large when performing three-dimensional reconstruction, a high-speed storage card can be used for storage. Meanwhile, the computer device can be used to control the synchronization operation between the image acquisition devices and the light source module. The image acquisition devices and the light source module can be synchronously set, so that the image acquisition devices can capture images under the light conditions required by the user.

[0037] In the above implementation, the computer device can also be configured to process the captured images in parallel. Moreover, the large amount of data sets are distributed to different computer devices, and it is ensured that the amount of data of each part is small enough to be processed by a single computer device. The processing of the data sets can include image correction, image segmentation processing, generation of depth images, generation of point clouds, fusion of point clouds, mesh construction, calculation of mesh normals, calculation of albedo, and the like in the three-dimensional reconstruction process.

[0038] Referring to Figure 2 In the above three-dimensional reconstruction system, the target object to be three-dimensionally reconstructed can be located at the center position of the internal space formed by the fixed frame 10, so that each set of the above data acquisition devices (including the image acquisition device 20 and the light source module 30) arranged on the fixed frame 10 can acquire data of the target object.

[0039] In a possible implementation, referring to Figure 3 In the above triangle formed by the connecting strips 11, the mounting rod of the image acquisition device 20 can be arranged on one of the connecting strips 11, the image acquisition device 20 can be mounted on the mounting rod 40, and the light source module 30 can be arranged on one of the other two connecting strips 11.

[0040] In a possible implementation, a control circuit can be arranged for each set of the above data acquisition devices, which can provide power supply for the light source module and the image acquisition device, and transmit control signals to the light source module and the image acquisition device to ensure that the light source module and the image acquisition device can be turned on and off synchronously.

[0041] Through the above three-dimensional reconstruction system, each set of the above data acquisition devices at different view positions can be controlled to work synchronously, that is, each light source module irradiates the target object, and the image acquisition device acquires images of the target object; according to the acquired color image data and infrared image data at different view positions, three-dimensional reconstruction can be performed, and a three-dimensional model corresponding to the target object can be obtained.

[0042] The data processing method provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0043] Please refer to Figure 4 , Figure 4 The flowchart of the data processing method provided by an embodiment of the present application is shown. The data processing method will be described in detail below with reference to the flowchart shown in Figure 4 The data processing method can specifically include the following steps:

[0044] Step S110: acquiring image data at different view positions acquired by a plurality of image acquisition devices for a target object, the target object being irradiated by color light sources and infrared light sources at the different view positions, the wavelength range and the irradiation range of the light emitted by the color light sources and the infrared light sources matching the spatial range formed by the view positions, the plurality of image acquisition devices including image acquisition devices at different view positions in the space where the target object is located, and the image data including color image data and infrared image data.

[0045] In the embodiments of the present application, when performing three-dimensional reconstruction on the target object, the target object can be irradiated by color light sources and infrared light sources at different view positions in the space where the target object is located, and the target object can be imaged by image acquisition devices at each view position. The image acquisition devices can include color cameras and infrared cameras, so that color image data and infrared image data acquired at different view positions can be obtained. Moreover, the wavelength range and the irradiation range of the light emitted by the color light sources and the infrared light sources in the embodiments of the present application match the spatial range formed by the above view positions, so that the lighting effect on the target object can be ensured to ensure the accuracy and quality of the obtained color image data and infrared image data.

[0046] In some embodiments, the above infrared light source can be an infrared light projector, such as a dot-matrix infrared light projector, which can project infrared patterns to the target object, the infrared patterns can be captured by the infrared camera, and then the infrared image data collected for the infrared patterns projected to the target object can be acquired. According to the acquired infrared image data, the depth information can be acquired by using the structured light.

[0047] In the embodiments of the present application, each light source module and each image acquisition device can be controlled to work synchronously, each light source module irradiates the target object at the same time, and the color image data and the infrared image data obtained by the image acquisition device of the target object at each view angle can be used as a frame of image data of the target object at different view angles.

[0048] Step S120: determining the depth image data of the target object based on the infrared image data at different view angles.

[0049] In the embodiments of the present application, after the color image data and the infrared image data collected at different view angles are acquired, the three-dimensional model can be constructed based on the color image data and the infrared image data at different view angles. When the three-dimensional model corresponding to the target object is acquired, the depth image data of the target object can be determined based on the infrared image data at different view angles. The depth image is a two-dimensional image, each pixel value represents the distance (i.e. depth value) from the position point of the target object corresponding to the position to the image acquisition device, and the depth image is usually stored in the form of a gray scale image, and the smaller the gray scale value, the closer the distance, and vice versa.

[0050] In some embodiments, when the depth image data of the target object is determined based on the infrared image data at different view angles, the depth can be calculated by using binocular disparity, structured light projection, or geometric principle of light time of flight (ToF).

[0051] Step S130: determining the three-dimensional point cloud data corresponding to the target object based on the depth image data.

[0052] In the embodiments of the present application, after the above depth image data is acquired, the depth image data can be converted into three-dimensional point cloud data, so as to generate the three-dimensional model corresponding to the target object according to the three-dimensional point cloud data subsequently.

[0053] In some embodiments, when determining the three-dimensional point cloud data of the target object based on the depth image data of the target object, the pixel coordinates of each pixel point in the depth image data and the depth value corresponding to each pixel point can be converted into three-dimensional point cloud coordinates according to the camera intrinsic parameters of the image acquisition device, and the obtained three-dimensional point cloud coordinates corresponding to each pixel point can be used as the three-dimensional point cloud data corresponding to the target object.

[0054] The camera intrinsic parameters are a set of parameters describing the internal geometry and optical characteristics of the camera, which are used to establish the mapping relationship between the three-dimensional world coordinates and the two-dimensional image pixels. The camera intrinsic parameters can be used for coordinate mapping, such as projecting a three-dimensional world point to a two-dimensional image plane. The camera intrinsic parameters can also be used for distortion correction to eliminate lens distortion (such as barrel distortion, pincushion distortion, etc.). The camera intrinsic parameters can also be used as a reference for camera extrinsic parameter calculation and three-dimensional reconstruction. The camera intrinsic parameters can include focal length, principal point, pixel size, etc. The focal length is the distance from the optical center to the imaging plane. The principal point is the intersection coordinate of the optical axis and the image plane, and the ideal value of the principal point is the image center. The pixel size is the physical size of a single pixel. Of course, the specific parameters of the camera intrinsic parameters are not limited, for example, they can also include distortion coefficients (such as radial distortion and tangential distortion, etc.), and can also include coordinate axis non-orthogonality correction (usually 0).

[0055] For example, given the pixel coordinates (u, v) and the corresponding depth value Z of a pixel point in the depth image, the calculation formula of the three-dimensional point cloud coordinates (X, Y, Z) is as follows:

[0056]

[0057] Z = Z,

[0058] where (f x , f y ) is the focal length of the camera, (c x , c y ) is the principal point coordinate (optical center position), and Z is the depth value.

[0059] Step S140: generating a three-dimensional model corresponding to the target object based on the color image data and the three-dimensional point cloud data.

[0060] In the embodiments of the present application, after obtaining the three-dimensional point cloud data, a three-dimensional model corresponding to the target object can be generated based on the color image data and the three-dimensional point cloud data at different viewing angles. The three-dimensional model can be a point cloud model or a mesh model. The point cloud model is composed of a set of discrete three-dimensional coordinate points, each point containing position information (X / Y / Z) and extensible color, normal vector, etc. The mesh model is composed of a topological network of vertices, edges and faces, and is usually dominated by triangular meshes.

[0061] In some embodiments, since the three-dimensional point cloud data does not have color information, the color information in the color image data can be mapped to the corresponding point cloud positions in the three-dimensional point cloud data, and then a three-dimensional model corresponding to the target object is generated according to the point cloud data with color information.

[0062] The data processing method provided by the embodiments of the present application comprises the following steps: acquiring image data of different visual angle positions of a target object collected by a plurality of image acquisition devices, the target object being irradiated by color light sources and infrared light sources at different visual angle positions, the wavelength range and the irradiation range of the light emitted by the color light sources and the infrared light sources matching the spatial range formed by the visual angle positions, the plurality of image acquisition devices comprising image acquisition devices at different visual angle positions in a space where the target object is located, the image data comprising color image data and infrared image data, determining depth image data of the target object based on the infrared image data at different visual angle positions, determining three-dimensional point cloud data corresponding to the target object based on the depth image data, and generating a three-dimensional model corresponding to the target object based on the color image data and the three-dimensional point cloud data. Thus, the target object for which a three-dimensional model needs to be reconstructed can be irradiated by color light sources and infrared light sources at a plurality of different visual angle positions, and the wavelength range of the light emitted by the light sources matches the spatial range formed by the plurality of visual angle positions, so that the lighting effect on the target object can be ensured, and the effect of three-dimensional reconstruction and the quality of the three-dimensional model can be improved after three-dimensional reconstruction is performed according to the information of the plurality of visual angle positions of the target object.

[0063] Please refer to Figure 5 , Figure 5 The flowchart of the data processing method provided by another embodiment of the present application is shown. The data processing method will be described in detail below with respect to the flowchart shown in Figure 5 The data processing method can specifically comprise the following steps:

[0064] Step S210: acquiring image data of different visual angle positions of a target object collected by a plurality of image acquisition devices, the target object being irradiated by color light sources and infrared light sources at different visual angle positions, the wavelength range and the irradiation range of the light emitted by the color light sources and the infrared light sources matching the spatial range formed by the visual angle positions, the plurality of image acquisition devices comprising image acquisition devices at different visual angle positions in a space where the target object is located, the image data comprising color image data and infrared image data.

[0065] In the embodiments of the present application, step S210 can refer to the content of other embodiments, which will not be described here again.

[0066] Step S220: matching texture information of different infrared image data to obtain a texture matching result.

[0067] In the embodiments of the present application, when the depth image data is determined based on the infrared image data of different perspective positions, the stereo vision technology is considered to match the images with rich textures as features. With rich textures, different images can be easily matched, and thus the depth information can be determined according to the disparity information between the textures by matching the disparity information between the textures. The texture information of different infrared image data can be matched to obtain texture matching results, so as to determine the disparity data according to the texture matching results.

[0068] In some embodiments, in order to ensure the accuracy of the obtained depth image data, the infrared image data of different perspectives can be preprocessed and feature enhanced before texture matching. The preprocessing and feature enhancement can include at least one of image registration and thermal radiation enhancement. The image registration can include aligning the infrared images of different perspectives and eliminating the baseline deviation. The thermal radiation enhancement can include extracting temperature gradient features (such as calculating thermal edges by Sobel operator).

[0069] Step S230: determining the disparity data corresponding to the plurality of feature points of the target object based on the texture matching results.

[0070] In the embodiments of the present application, after the texture matching results are obtained, the disparity data corresponding to the plurality of feature points of the target object can be determined according to the texture matching results, so as to determine the depth information according to the disparity data.

[0071] In some embodiments, the texture matching results can represent the similarity of corresponding pixels in different infrared image data. The texture matching results obtained by texture matching can obtain a matching cost. The matching cost is aggregated, for example, the matching path is optimized along the epipolar line. The disparity data is obtained by optimizing the disparity according to the matching cost. The disparity optimization is used to eliminate the false matching, which can be achieved by consistency check and sub-pixel difference.

[0072] Step S240: determining the depth image data of the target object based on the disparity data.

[0073] In the embodiments of the present application, after the disparity data corresponding to the plurality of feature points of the target object is obtained, the depth information corresponding to different feature points can be calculated by using the disparity formula, so as to obtain the depth image data of the target object.

[0074] Exemplarily, the disparity formula can be: wherein Z is the depth, f is the focal length of the camera, B is the baseline, and d is the disparity.

[0075] Step S250: determining three-dimensional point cloud data corresponding to the target object based on the depth image data.

[0076] In the embodiments of the present application, step S250 can refer to the content of other embodiments, which will not be described here.

[0077] Step S260: generating a three-dimensional model corresponding to the target object based on the texture information in the depth image data, the color image data and the three-dimensional point cloud data.

[0078] In the embodiments of the present application, when generating a three-dimensional model corresponding to the target object based on the obtained three-dimensional point cloud data, it is considered that the texture of the target object itself may not be rich enough, and the light points projected by the infrared light source (infrared transilluminator) irradiate the target object, and the texture information is also carried in the collected infrared image data, and when the depth image data is obtained, the texture information is also matched, so the texture information in the above depth image data can be supplemented, that is, the three-dimensional model corresponding to the target object is generated based on the texture information in the depth image data, the color image data and the three-dimensional point cloud data, so that a three-dimensional model with higher quality can be generated using more detailed information.

[0079] The data processing method provided by the embodiments of the present application can realize irradiation of the target object requiring three-dimensional model reconstruction by the color light source and the infrared light source at multiple different viewing angle positions, and the wavelength range of the light emitted by the light source matches the spatial range promoted by the multiple viewing angle positions, so as to ensure the lighting effect of the target object, and then after three-dimensional reconstruction according to the information of multiple viewing angle positions of the target object, the effect of three-dimensional reconstruction can be improved, and the quality of the three-dimensional model can be improved. In addition, the texture information in the infrared image data when the texture matching between the infrared image data is performed in the process of obtaining the depth image data supplements the texture information of the target object, so that a three-dimensional model with higher quality can be generated using more detailed information, and the quality of the generated three-dimensional model is further improved.

[0080] Please refer to Figure 6 , Figure 6 A flowchart of a data processing method provided by another embodiment of the present application is shown. The following will be described in detail with respect to the flowchart shown in Figure 6 The data processing method can specifically include the following steps:

[0081] Step S310: Obtain image data of different view positions collected by a plurality of image collection devices for a target object, the target object being irradiated by color light sources and infrared light sources at the different view positions, the wavelength range and irradiation range of the light emitted by the color light sources and the infrared light sources matching a spatial range formed by the view positions, the plurality of image collection devices including image collection devices at different view positions in a space where the target object is located, and the image data including color image data and infrared image data.

[0082] In the embodiments of the present application, step S310 can refer to the content of other embodiments, which will not be repeated here.

[0083] Step S320: Segment image data of a region where the target object is located from the infrared image data as first image data.

[0084] In the embodiments of the present application, in order to ensure the accuracy of the three-dimensional model generated for the target object and avoid the generated three-dimensional model containing other content that does not need to be constructed into a three-dimensional model, after obtaining the color image data and the infrared image data, the image data of the region where the target object is located can be segmented from the infrared image data, and the segmented image data can be used as the first image data, thereby ensuring the accuracy of the infrared image data on which the three-dimensional model is generated.

[0085] In some embodiments, when segmenting the image data of the region where the target object is located from the infrared image data, the region segmentation can be implemented based on the depth clues of the infrared image. In this regard, the precise positioning of the region where the target object is located can be achieved through a multi-stage processing procedure by combining the infrared thermal radiation characteristics and geometric features.

[0086] In a possible implementation, the depth clues can include at least one of thermal radiation gradient clues, stereo disparity clues, and temperature distribution clues. In this regard, when obtaining the thermal radiation gradient clues, a thermal edge map can be constructed using the gradient direction of temperature change in the infrared image (such as a Sobel operator) as the thermal radiation gradient clues; when obtaining the stereo disparity clues, the depth distribution can be calculated through a disparity map (Disparity Map) of a binocular infrared camera, and a high-disparity region corresponds to a close-range target; and when obtaining the temperature distribution clues, the infrared image can be divided into different temperature regions through clustering analysis (such as K-means).

[0087] In a possible implementation, after the above depth clues are obtained, the infrared image can be preprocessed and enhanced. For example, infrared noise can be suppressed by non-local mean filtering (BM3D), and thermal edge visibility can be improved by adaptive histogram equalization (CLAHE). Then, the region where the target object is located is segmented based on the above depth clues. When the depth clues include the thermal radiation gradient clues, the gradient amplitude threshold segmentation can be used to connect the discontinuous edges by morphological dilation. When the depth clues include the stereo disparity clues, the regions with a disparity greater than a target threshold, for example, a disparity greater than 10 pixels, can be selected as the region where the target object is located based on the disparity clues. When the depth clues include the temperature distribution clues, the connected component analysis can be performed on the temperature clustering result, and the large-area high-temperature / low-temperature region is retained to determine the region where the target object is located.

[0088] In some embodiments, after the infrared image data in the region where the target object is located is segmented, morphological operations, bounding box correction, and the like can be performed on the region where the target object is located to further ensure the accuracy of the infrared image data in the region where the target object is located. The morphological operations can include: a closing operation to fill the internal holes of the segmented region where the target object is located, and an opening operation to remove isolated noise. The bounding box correction can include: fitting a minimum bounding rectangle or an ellipse to adapt to an irregular target shape.

[0089] Step S330: determining depth image data of the target object based on the first image data.

[0090] In the embodiments of the present application, after the above first image data in the region where the target object is located is obtained, the depth image data of the target object can be determined based on the first image data, so as to avoid the interference of information in other regions and ensure the accuracy of the depth image data.

[0091] Step S340: determining three-dimensional point cloud data corresponding to the target object based on the depth image data.

[0092] In the embodiments of the present application, step S340 can refer to the content of other embodiments, which will not be described here again.

[0093] Step S350: segmenting image data in the region where the target object is located from the color image data as second image data.

[0094] In the embodiments of the present application, in order to ensure the accuracy of the three-dimensional model generated for the target object and avoid that the generated three-dimensional model contains other contents that do not need to be constructed into a three-dimensional model, after the color image data and the infrared image data are acquired, the image data of the region where the target object is located can be segmented from the color image data before the three-dimensional model is generated based on the color image data.

[0095] In some embodiments, when the image data of the region where the target object is located is segmented from the color image data, the region with a specific semantic category in the color image data can be recognized and located by using a semantic segmentation model to obtain the image data of the region where the target object is located. The semantic segmentation model can be a neural network model, and the specific type of the model is not limited.

[0096] In a possible implementation, before the image data of the region where the target object is located is segmented from the color image data, the color image data can be preprocessed. The preprocessing can include image normalization, which can normalize the pixel values of the pixel points in the color image data to the range of [0, 1] or [-1, 1] to accelerate the model convergence.

[0097] In a possible implementation, when the image data of the region where the target object is located is segmented from the color image data by using the semantic segmentation model, the result output by the semantic segmentation model can be a probability map, which includes the probability that each pixel point is a pixel point in the region where the target object is located. The probability map can be converted into a binary mask, in which the pixel points with a probability greater than a threshold (for example, the threshold can be 0.5) can be 1, and the pixel points with a probability not greater than the threshold can be 0. For the above binary mask, a connected region can be determined, and morphological operations can be performed on the connected region, such as filling holes by using a closing operation, removing noise by using an opening operation, and the like. For the connected region, regions with an area and an aspect ratio meeting the requirements can be screened to obtain the image data of the region where the target object is located.

[0098] Step S360: generating a three-dimensional model corresponding to the target object based on the second image data and the three-dimensional point cloud data.

[0099] In the embodiments of the present application, after the second image data of the region where the target object is located is obtained from the color image data, a three-dimensional model corresponding to the target object can be generated based on the second image data and the three-dimensional point cloud data.

[0100] The data processing method provided in the embodiments of the present application can realize irradiation of a target object requiring three-dimensional model reconstruction by color light sources and infrared light sources at multiple different view angle positions, and the wavelength range of the light emitted by the light sources matches the spatial range co-promoted by the multiple view angle positions, so as to ensure the lighting effect on the target object, and then after three-dimensional reconstruction according to the information of the multiple view angle positions of the target object, the effect of three-dimensional reconstruction can be improved, and the quality of the three-dimensional model can be improved. In addition, before three-dimensional reconstruction according to the obtained color image data and infrared image data, segmentation of the region where the target object is located is performed on the color image data and the infrared image data, so as to exclude the interference of other contents, and the accuracy of the generated three-dimensional model is further improved.

[0101] Please refer to Figure 7 , Figure 7 The flowchart of the data processing method provided in another embodiment of the present application is shown. The following will be described in detail with respect to the flowchart shown in Figure 7 The data processing method can specifically include the following steps:

[0102] Step S410: Obtain image data at different view angle positions collected by multiple image acquisition devices for a target object, the target object is irradiated by color light sources and infrared light sources at the different view angle positions, the wavelength range and the irradiation range of the light emitted by the color light sources and the infrared light sources match the spatial range constituted by the view angle positions, the multiple image acquisition devices include image acquisition devices at different view angle positions in the space where the target object is located, and the image data includes color image data and infrared image data.

[0103] Step S420: Determine depth image data of the target object based on the infrared image data at different view angle positions.

[0104] Step S430: Determine three-dimensional point cloud data corresponding to the target object based on the depth image data.

[0105] In the embodiments of the present application, steps S410 to S430 can refer to the content of other embodiments, which will not be described here.

[0106] Step S440: Map color information in the color image data to corresponding point clouds in the three-dimensional point cloud data to obtain target point cloud data containing color information.

[0107] In the embodiments of the present application, when generating a three-dimensional model corresponding to the target object based on the obtained color image data and three-dimensional point cloud data, the color information in the color image data can be mapped to corresponding point clouds in the three-dimensional point cloud data, so that each point cloud carries color information, and target point cloud data containing color information is obtained.

[0108] In some embodiments, the intrinsic parameters of the color camera in the image acquisition device can be acquired, which are used to project the camera pixel coordinates into the three-dimensional space. Then, according to the intrinsic parameters, the color information in the color image data is mapped to the corresponding point cloud in the three-dimensional point cloud data, so as to realize the back projection of the pixel information of the color image data into the three-dimensional space.

[0109] Step S450: based on the extrinsic parameters of the image acquisition device, the target point cloud data is fused into the global point cloud data corresponding to the target object.

[0110] In the embodiments of the present application, after the above target point cloud data containing color information is acquired, the target point cloud data can be fused into the global point cloud data corresponding to the target object based on the extrinsic parameters of the image acquisition device. The camera extrinsic parameters describe the position and direction of the camera in the world coordinate system, and define the geometric transformation relationship between the three-dimensional space and the camera coordinate system. The camera extrinsic parameters include 6 parameters, which are divided into a rotation matrix (R) and a translation vector (T). The rotation matrix is a 3x3 matrix, which describes the rotation attitude of the camera coordinate system relative to the world coordinate system. The translation vector is a 3x1 vector, which represents the position offset of the camera coordinate system origin in the world coordinate system.

[0111] In some embodiments, when the target point cloud data is fused into the global point cloud data corresponding to the target object based on the extrinsic parameters of the image acquisition device, the point cloud can be transformed to the same coordinate system by using the camera extrinsic parameters. After transformation to the same coordinate system, the overlapping regions can be fused, and then the global point cloud data is obtained, that is, the complete geometric and color information of the target object is obtained.

[0112] Step S460: based on the global point cloud data, a three-dimensional mesh model corresponding to the target object is generated.

[0113] In the embodiments of the present application, the generated three-dimensional model of the target object can be a three-dimensional mesh model. After the above global point cloud data is acquired, the three-dimensional mesh model corresponding to the target object can be generated based on the global point cloud data.

[0114] In some embodiments, the Poisson reconstruction algorithm can be used to generate a closed three-dimensional mesh based on the global point cloud data, so as to obtain the three-dimensional mesh model. In addition, when the closed three-dimensional mesh is generated based on the global point cloud data, the color information can also be transmitted from the point cloud to the mesh vertex.

[0115] In a possible implementation, after the three-dimensional mesh model is generated, denoising processing can also be performed, for example, outliers can be counted and removed, so as to further ensure the quality of the generated three-dimensional mesh model.

[0116] Step S470: acquiring the three-dimensional mesh model generated according to the multi-frame color image data and the infrared image data.

[0117] In the embodiments of the present application, a dynamic three-dimensional model can also be generated, so that the generated three-dimensional model can present a dynamic motion effect. Wherein, multi-frame color image data and infrared image data collected at different view positions can be acquired, and a three-dimensional mesh model can be generated based on each frame of color image data and infrared image data, thereby obtaining multi-frame three-dimensional mesh models.

[0118] The way of generating the three-dimensional mesh model in the foregoing embodiments will be further described below. Wherein, after acquiring the color image data and the infrared image data at different view positions, the region where the target object is located can be segmented; then, the extrinsic parameters of the camera are used to carry out stereo matching, and in the stereo matching process, the image correction is calculated first, and then the depth data of the target object is calculated; according to the depth data, the color image data can be back-projected to the three-dimensional space, thereby obtaining the point cloud data. In addition, the depth data usually contains a large amount of noise, and before subsequent processing, the depth data can be denoised. Then, the camera extrinsic parameters obtained by calibration are used to fuse the point clouds obtained by different stereo image pairs into a complete three-dimensional model; then, the Poisson reconstruction algorithm is used to generate a three-dimensional mesh from the point cloud data. In addition, considering that some meshes in the generated three-dimensional mesh model may have defects, the generated mesh can also be cleaned up.

[0119] Step S480: performing mesh alignment processing on the multi-frame three-dimensional mesh models.

[0120] In the embodiments of the present application, since the three-dimensional mesh models of different frames differ in facets, vertices, and noise, etc., the models of each frame in the initial state are not consistent in appearance, in order to make the motion performance of the three-dimensional model consistent, generate a high-fidelity three-dimensional mesh, and avoid poor visual inconsistency results between different frames, a mesh consistency algorithm can be used to perform mesh alignment processing between the three-dimensional mesh models of different frames.

[0121] In some embodiments, since the time interval between two adjacent frames is very small, the topology structure of the two can be defaulted to be consistent, and the network between the two consecutive frames can be marked with the same ID, and according to the same ID, the meshes of the two consecutive frames can be aligned. Wherein, each mesh of the current frame can be aligned with the mesh of the previous frame; then, every N frames (N>10), a key frame can be selected, and all frames thereafter will be aligned with the key frame.

[0122] In some embodiments, the observability scores of each frame of the three-dimensional mesh model can be calculated first, and at least one frame of the three-dimensional mesh model can be selected as a reference frame of the three-dimensional mesh model, which can provide a good reference basis for several frames of the three-dimensional mesh model before and after the reference frame of the three-dimensional mesh model. Then, the non-rigid registration algorithm can be used to register the three-dimensional mesh model adjacent to the reference frame of the three-dimensional mesh model with the reference frame of the three-dimensional mesh model.

[0123] The data processing method provided by the embodiments of the present application can realize irradiation of a target object requiring three-dimensional model reconstruction by color light sources and infrared light sources at multiple different viewing angle positions, and the wavelength range of the light emitted by the light sources matches the spatial range co-promoted by the multiple viewing angle positions, so as to ensure the lighting effect of the target object, and then after three-dimensional reconstruction according to the information of the multiple viewing angle positions of the target object, the effect of three-dimensional reconstruction can be improved, and the quality of the three-dimensional model can be improved. In addition, the generated three-dimensional model is a three-dimensional mesh model, and multiple three-dimensional mesh models can be generated according to multiple image data, so that the generated three-dimensional mesh model can present a dynamic motion effect, and the grid alignment processing is performed on the multiple three-dimensional mesh models, further ensuring the accuracy of the generated multiple three-dimensional mesh models.

[0124] Please refer to Figure 8 , Figure 8 The flowchart of the data processing method provided by another embodiment of the present application is shown. The following will be described in detail with respect to the flowchart shown in Figure 8 The data processing method can specifically include the following steps:

[0125] Step S510: Obtain image data of different viewing angle positions collected by multiple image acquisition devices with respect to a target object, the target object is irradiated by color light sources and infrared light sources at the different viewing angle positions, the wavelength range and the irradiation range of the light emitted by the color light sources and the infrared light sources match the spatial range formed by the viewing angle positions, the multiple image acquisition devices include image acquisition devices at different viewing angle positions in a space where the target object is located, and the image data includes color image data and infrared image data.

[0126] Step S520: Determine depth image data of the target object based on the infrared image data at different viewing angle positions.

[0127] Step S530: Determine three-dimensional point cloud data corresponding to the target object based on the depth image data.

[0128] Step S540: Generate a three-dimensional model corresponding to the target object based on the color image data and the three-dimensional point cloud data.

[0129] In the embodiments of the present application, steps S510 to S540 can refer to the content of other embodiments, which will not be described here again.

[0130] Step S550: determining the relighting parameters of the three-dimensional model based on the illumination information of the color light source.

[0131] In the embodiments of the present application, different illumination conditions are provided by means of color light sources and infrared light sources. In addition to the above method of realizing the reconstruction of the three-dimensional model, the relighting parameters of the three-dimensional model can also be estimated according to the illumination information of the color light source. These relighting parameters can help the reconstructed three-dimensional model to be relit, so that the target object can look as if it is in different external lighting environments, thereby improving the realism of the three-dimensional model.

[0132] In some embodiments, a polarizer for separating diffuse reflection and specular reflection is arranged on the light-emitting surface of the color light source, so as to be able to determine at least one of the normal, reflectivity, surface roughness, diffuse reflection parameter and highlight parameter of the three-dimensional model as the relighting parameter according to the illumination information of the color light source.

[0133] In the above embodiments, in the related art, the photometric stereo method can calculate the normal of the surface of the object according to multiple color image data. Each color image data records the object reflection characteristics captured by the camera when only one color light source is turned on, and these color light sources are placed in different directions. For each pixel on the image, the normal can be calculated using the following formula:

[0134] I = pn * l

[0135] In the above formula, I represents the brightness of the pixel, p represents the reflectivity, n is the surface normal, and l represents the illumination direction. If the reflection is Lambertian reflection, the above equation holds, so the light intensity reflected by each direction of the surface of the target object is the same. If the specular reflection characteristics are considered, the reflection function can be further improved. Assuming that there are k different illumination directions, the following vector can be constructed:

[0136] I = [I0,..., Ik] k-1 , Ik] k

[0137] L = [l0,..., lk] k-1 , lk] k

[0138] Based on the above vector, the following result can be obtained:

[0139] I = pnL

[0140] ​​Based on the above k sets of measurement values, a least square problem can be constructed by the k sets of measurement values. However, such a way can be affected by the shadow, measurement noise and mirror reflection characteristics, and therefore, the shadow, noise and mirror reflection characteristics can be treated as an additional noise item E, to obtain:

[0141] I = pnL + E

[0142] Based on I = pnL + E, the optimal matrix E can be determined, and the optimal normal vector can be estimated. Please refer to Figure 9 and Figure 10 , Figure 9 is an effect diagram after re-lighting processing of the result calculated by the traditional method, Figure 10 is an effect diagram after re-lighting processing of the result calculated by the above way in the embodiment of the present application, and it can be seen that the shadow presented by the result calculated by the above way is less in the marked areas A1 and A2 of the figure.

[0143] In addition, after the normal vector is obtained, the following reflection function can be used to consider the mirror reflection characteristics (reflectivity) and surface roughness:

[0144] I = pnL + k s (n * h) a

[0145] wherein I d = pn* l can be obtained by the polarized image without mirror reflection. In addition, I can also be obtained by the non-polarized camera, and thus the following results can be obtained:

[0146] I s = I - I d = k s (n * l) α

[0147] ln(I s ) = ln*(k s ) + a ln(n * l)

[0148] In this embodiment, the least square method can be used to solve the highlight parameter k and the roughness parameter a. Exemplarily, as shown in Figure 11 , Figure 12 and Figure 13 , Figure 11 is an effect diagram after re-lighting processing of the reflectivity calculated by the above way provided in the embodiment of the present application, Figure 12 is an effect diagram after re-lighting processing of the highlight parameter calculated by the above way provided in the embodiment of the present application, Figure 13 is an effect diagram after re-lighting processing of the surface roughness calculated by the above way provided in the embodiment of the present application.

[0149] In some embodiments, the color light source and the infrared light source can be several programmable printed circuit boards (PCB) with LEDs. Each PCB is provided with a plurality of LEDs of multiple different colors, which are within the visible light wavelength range and can be captured by a color camera. In addition, an infrared LED light is additionally provided on the circuit board, which emits infrared light that can be identified by an infrared camera. The wavelength range and illumination range of each color channel match the spatial range formed by the multiple view angle positions, so that the image acquisition device can capture the most abundant information.

[0150] Exemplarily, the wavelength range of the blue channel can be 455-460 nm, the wavelength range of the red channel can be 620-625 nm, the wavelength range of the green channel can be 520-525 nm, the wavelength range of the infrared channel can be 830-835 nm, and the wavelength range of the gray channel can be 520-525 nm; the wavelength range of the cyan-blue channel can be 490-495 nm; the wavelength range of the yellow channel can be 570-575 nm; and the wavelength range of the deep red channel can be 670-675 nm. These lights have a specific illumination range, for example, an angle of 30 degrees, which can cover most of the sample objects without causing excessive energy loss.

[0151] In a possible embodiment, as shown in the fixed frame of Figure 1 The fixed frame has three coordinate axes X, Y, and Z, and the illumination intensity of the color light source gradually changes along each coordinate axis, and this change rule can be set in the control program. Based on the illumination information of the color light source, the above relighting parameters of the three-dimensional model can be calculated, and after the three-dimensional model is relit based on the determined relighting parameters, the reflection effect of the object can be presented.

[0152] In a possible embodiment, the above LED arrangement mode is programmed so that a large number of color light sources can simulate various lighting conditions in actual scenes. The color LEDs can be driven by gradient lighting to calculate the reflection characteristics of the object. In addition to color LEDs, the color light source also includes high-brightness white LEDs. Unlike the simultaneous lighting mode of color LEDs, the white LEDs use a sequential lighting mode, that is, only one LED is in the lighting state at the same time stamp, and the reflection ability of the object can be calculated based on the photometric stereo algorithm. Among them, the Arduino program development board and the function generator can be used to control the signal, so as to accurately control the RGB light and the white light.

[0153] In a possible implementation, when the target object is a human body, the collector can feel dizzy during the image data collection process due to strong light irradiation and light flicker, and therefore the frequency of the color light source can be synchronized with the working frequency of the image collection device. In order to make the eyes of the user more friendly, the working frequency of the color light source can be higher, for example, 180 Hz, 240 Hz, etc.

[0154] In a possible implementation, a pulse width modulation (PWM) wave can be generated by a signal generator and transmitted to a central control panel, which is configured to distribute the signal to intermediate control panels. The intermediate control panels receive the signal from the central control panel and further distribute it to individual LEDs. Through this implementation, each intermediate control panel is connected to a dozen PCB circuit boards, thereby achieving high refresh rate of the LEDs and enabling them to present a variety of color modes.

[0155] In the above implementation, the traditional photometric stereo method is based on the assumption of parallel light irradiation to calculate the object normal and perform three-dimensional reconstruction, which assumes that the light source is very far away from the object. The data processing method provided in the embodiments of the present application uses a point light source assumption, which simulates the light direction condition in the real world and assumes that the light irradiating the object comes from different directions. When the light source has a large field of illumination (FOI) and is close to the target object, this assumption can provide more detailed information.

[0156] In some implementations, since the polarizer with a 90-degree polarization angle difference can eliminate specular reflection light, based on this characteristic, diffuse reflectance and highlight reflection normal can also be calculated. The polarizer and its polarization angle can be manually adjusted; of course, in order to reduce the difficulty of fixed frame installation, an automatic / digital polarizer controlled by a program can be used to achieve automatic adjustment of the polarizer.

[0157] Step S560: performing re-lighting processing on the three-dimensional model based on the re-illumination parameter.

[0158] In the embodiments of the present application, after the re-illumination parameter is obtained, the three-dimensional model can be re-lit based on the re-illumination parameter, so that the target object can look as if it is in a different external lighting environment, improving the realism of the three-dimensional model.

[0159] In some embodiments, the key to re-lighting processing of the three-dimensional model based on the re-illumination parameters is to restore the parameters of the bidirectional reflectance distribution function (BRDF). The BRDF is a function for defining the way light is reflected on an opaque surface. In the case of knowing the BRDF function, the light direction and the light intensity, the object can be rendered in a rendering engine. Among them, based on the gradient illumination measurement method, or the way of measuring each time only using one lamp when collecting image data, the parameters of the BRDF function can be obtained.

[0160] In a possible implementation, when gradient illumination lighting under the control of different light intensities is adopted and the reflection characteristics are tried to be restored, the spherical area integral method can be used. In the spherical area integral method, it is assumed that the light intensity changes continuously along the axis, that is, it is gradually weakened or gradually enhanced. However, in actual operation, because only a limited number of light sources can be arranged along the axis, the light changes present discrete characteristics. Therefore, the spherical area integral method is difficult to be completely accurate, and the calculation accuracy of the spherical area integral can be improved by increasing the number of light sources and increasing the light change intensity of each light source.

[0161] The data processing method provided by the embodiments of the present application can realize irradiation of a target object requiring three-dimensional model reconstruction by a plurality of color light sources and infrared light sources at different viewing angle positions, and the wavelength range of the light emitted by the light sources matches the spatial range promoted by the plurality of viewing angle positions, so as to ensure the lighting effect of the target object, and then after three-dimensional reconstruction according to the information of the plurality of viewing angle positions of the target object, the effect of three-dimensional reconstruction can be improved, and the quality of the three-dimensional model can be improved. In addition, by using the known light information of the plurality of light sources, the re-illumination parameters are determined, and the generated three-dimensional model is re-lit by using the re-illumination parameters, so that the realism of the three-dimensional model can be improved.

[0162] Please refer to Figure 14Fig. 6 shows a structural block diagram of a data processing apparatus 600 provided by an embodiment of the present application. The data processing apparatus 600 comprises an image acquisition module 610, a depth acquisition module 620, a point cloud acquisition module 630, and a model generation module 640. The image acquisition module 610 is configured to acquire image data of different view positions collected by a plurality of image acquisition devices for a target object, the target object being irradiated by color light sources and infrared light sources at the different view positions, the wavelength ranges and irradiation ranges of the light emitted by the color light sources and the infrared light sources matching a spatial range formed by the view positions, the plurality of image acquisition devices comprising image acquisition devices at different view positions in a space where the target object is located, and the image data comprising color image data and infrared image data. The depth acquisition module 620 is configured to determine depth image data of the target object based on the infrared image data at different view positions. The point cloud acquisition module 630 is configured to determine three-dimensional point cloud data corresponding to the target object based on the depth image data. The model generation module 640 is configured to generate a three-dimensional model corresponding to the target object based on the color image data and the three-dimensional point cloud data.

[0163] In some embodiments, the depth acquisition module 620 can be specifically configured to: perform texture information matching on different infrared image data to obtain texture matching results; determine disparity data corresponding to a plurality of feature points of the target object based on the texture matching results; and determine depth image data of the target object based on the disparity data.

[0164] In a possible implementation, the model generation module 640 can be specifically configured to generate a three-dimensional model corresponding to the target object based on texture information in the depth image data, the color image data, and the three-dimensional point cloud data.

[0165] In some embodiments, the depth acquisition module 620 can be specifically configured to: segment image data of a region where the target object is located from the infrared image data as first image data; and determine depth image data of the target object based on the first image data.

[0166] In some embodiments, the model generation module 640 can be specifically configured to: segment image data of a region where the target object is located from the color image data as second image data; and generate a three-dimensional model corresponding to the target object based on the second image data and the three-dimensional point cloud data.

[0167] In some embodiments, the model generation module 640 can be specifically configured to: map color information in the color image data to a corresponding point cloud in the three-dimensional point cloud data to obtain target point cloud data containing color information; based on the extrinsic parameters of the image acquisition device, fuse the target point cloud data into global point cloud data corresponding to the target object; and based on the global point cloud data, generate a three-dimensional mesh model corresponding to the target object.

[0168] In a possible implementation, the model generation module 650 can also be configured to: obtain the three-dimensional mesh models generated according to multiple frames of the color image data and the infrared image data; and perform mesh alignment processing on the multiple frames of the three-dimensional mesh models.

[0169] In some embodiments, the data processing apparatus 600 can further include a parameter acquisition module and a relighting module. The parameter acquisition module can be configured to, after generating the three-dimensional model corresponding to the target object based on the color image data and the three-dimensional point cloud data, determine a relighting parameter of the three-dimensional model based on the illumination information of the color light source; and the relighting module can be configured to perform relighting processing on the three-dimensional model based on the relighting parameter.

[0170] In a possible implementation, a polarizer for separating diffuse reflection and specular reflection can be arranged on the light emitting surface of the color light source, and the parameter acquisition module can be specifically configured to: determine at least one of a normal, a reflectivity, a surface roughness, a diffuse reflection parameter, and a highlight parameter of the three-dimensional model as the relighting parameter based on the illumination information of the color light source.

[0171] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described apparatuses and modules can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein.

[0172] In the several embodiments provided in the present application, the coupling between the modules can be electrical, mechanical, or other forms of coupling.

[0173] In addition, each functional module in each embodiment of the present application can be integrated in one processing module, or each module can exist physically independently, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0174] In summary, the scheme provided in the present application acquires image data of different view angle positions collected by multiple image acquisition devices for a target object, the target object is irradiated by color light sources and infrared light sources at different view angle positions, the wavelength range and irradiation range of the light emitted by the color light sources and the infrared light sources match the spatial range formed by the view angle positions, the multiple image acquisition devices include image acquisition devices at different view angle positions in the space where the target object is located, the image data includes color image data and infrared image data; based on the infrared image data at different view angle positions, depth image data of the target object is determined; based on the depth image data, three-dimensional point cloud data corresponding to the target object is determined; based on the color image data and the three-dimensional point cloud data, a three-dimensional model corresponding to the target object is generated. Thus, the target object for which a three-dimensional model needs to be reconstructed can be irradiated by multiple color light sources and infrared light sources at different view angle positions, and the wavelength range of the light emitted by the light sources matches the spatial range of the multiple view angle positions, so that the lighting effect on the target object can be ensured, and then after three-dimensional reconstruction according to the information of multiple view angle positions of the target object, the effect of three-dimensional reconstruction can be improved, and the quality of the three-dimensional model can be improved.

[0175] Please refer to Figure 15 which shows a structural block diagram of a computer device provided in an embodiment of the present application. The computer device 100 in the present application can include one or more of the following components: a processor 110, a memory 120, and one or more application programs, wherein the one or more application programs can be stored in the memory 120 and configured to be executed by the one or more processors 110, and the one or more application programs are configured to perform the method as described in the foregoing method embodiment.

[0176] The processor 110 can include one or more processing cores. The processor 110 connects various parts within the entire computer device 100 by various interfaces and lines, performs various functions of the computer device 100 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 110 can be integrated with a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes operating systems, user interfaces, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 110, but be implemented separately through a communication chip.

[0177] The memory 120 can include a random access memory (RAM) and can also include a read-only memory (ROM). The memory 120 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing each of the method embodiments described below, etc. The data storage area can also store data created by the computer device 100 in use (such as a phone book, audio and video data, chat record data, etc.).

[0178] Please refer to Figure 16 which shows a structural block diagram of a computer readable storage medium provided by the embodiments of the present application. The computer readable medium 800 stores program codes therein, and the program codes can be called and executed by a processor to perform the methods described in the above method embodiments.

[0179] The computer-readable storage medium 800 can be an electronic storage memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk or a ROM. Optionally, the computer-readable storage medium 800 comprises a non-transitory computer-readable medium. The computer-readable storage medium 800 has a storage space for program codes 810 to execute any of the method steps described above. These program codes can be read from or written to one or more computer program products. The program codes 810 can be compressed, for example, in an appropriate form.

[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art will understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent replacements for some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that, The method includes: The method acquires image data from multiple image acquisition devices at different viewing angles of a target object. The target object is illuminated by colored light sources and infrared light sources at the different viewing angles. The wavelength range and illumination range of the light emitted by the colored light sources and infrared light sources match the spatial range formed by the viewing angles. The multiple image acquisition devices include image acquisition devices at different viewing angles in the space where the target object is located. The image data includes color image data and infrared image data. Based on the infrared image data from different viewpoints, the depth image data of the target object is determined; Based on the depth image data, determine the three-dimensional point cloud data corresponding to the target object; Based on the color image data and the three-dimensional point cloud data, a three-dimensional model corresponding to the target object is generated.

2. The method according to claim 1, characterized in that, The method for determining the depth image data of the target object based on the infrared image data from different viewpoints includes: The texture information of different infrared image data is matched to obtain texture matching results; Based on the texture matching results, the disparity data corresponding to multiple feature points of the target object is determined; Based on the parallax data, the depth image data of the target object is determined.

3. The method according to claim 2, characterized in that, The step of generating a 3D model corresponding to the target object based on the color image data and the 3D point cloud data includes: Based on the texture information in the depth image data, the color image data, and the 3D point cloud data, a 3D model corresponding to the target object is generated.

4. The method according to claim 1, characterized in that, The method for determining the depth image data of the target object based on the infrared image data from different viewpoints includes: Image data of the region where the target object is located is segmented from the infrared image data and used as the first image data; Based on the first image data, the depth image data of the target object is determined.

5. The method according to claim 1, characterized in that, The step of generating a 3D model corresponding to the target object based on the color image data and the 3D point cloud data includes: Image data of the region where the target object is located is segmented from the color image data and used as the second image data; Based on the second image data and the three-dimensional point cloud data, a three-dimensional model corresponding to the target object is generated.

6. The method according to any one of claims 1-5, characterized in that, The step of generating a 3D model corresponding to the target object based on the color image data and the 3D point cloud data includes: The color information in the color image data is mapped to the corresponding point cloud in the three-dimensional point cloud data to obtain target point cloud data containing color information. Based on the external parameters of the image acquisition device, the target point cloud data is fused into global point cloud data corresponding to the target object; Based on the global point cloud data, a 3D mesh model corresponding to the target object is generated.

7. The method according to claim 6, characterized in that, The method further includes: Obtain the three-dimensional mesh model generated based on the multiple frames of color image data and infrared image data; Perform mesh alignment processing on the three-dimensional mesh model in multiple frames.

8. The method according to any one of claims 1-5, characterized in that, After generating the 3D model corresponding to the target object based on the color image data and the 3D point cloud data, the method further includes: Based on the illumination information of the colored light source, the relighting parameters of the three-dimensional model are determined; Based on the aforementioned lighting parameters, the 3D model is relit.

9. The method according to claim 8, characterized in that, The light-emitting surface of the colored light source is provided with a polarizer for separating diffuse reflection and specular reflection. Determining the relighting parameters of the three-dimensional model based on the illumination information of the colored light source includes: Based on the illumination information of the colored light source, at least one of the normal, reflectivity, surface roughness, diffuse reflection parameters, and specular parameters of the three-dimensional model is determined as the relighting parameters.

10. A data processing apparatus, characterized in that, The device includes: an image acquisition module, a depth acquisition module, a point cloud acquisition module, and a model generation module, wherein, The image acquisition module is used to acquire image data from different viewing angles of a target object collected by multiple image acquisition devices. The target object is illuminated by colored light sources and infrared light sources at the different viewing angles. The wavelength range and illumination range of the light emitted by the colored light sources and infrared light sources match the spatial range formed by the viewing angles. The multiple image acquisition devices include image acquisition devices at different viewing angles in the space where the target object is located. The image data includes color image data and infrared image data. The depth acquisition module is used to determine the depth image data of the target object based on the infrared image data from different viewpoints. The point cloud acquisition module is used to determine the three-dimensional point cloud data corresponding to the target object based on the depth image data; The model generation module is used to generate a 3D model corresponding to the target object based on the color image data and the 3D point cloud data.

11. A computer device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1-9.

Images