Data acquisition system and method, three-dimensional reconstruction method, matting method and related equipment

Abstract

The embodiment of the invention provides a data acquisition system and method, a three-dimensional reconstruction method, an image matting method and related equipment. In the embodiment of the invention, the light source array provides consistent illumination information for the multi-camera array, and the control equipment triggers the multi-camera array to synchronously shoot through the hardware synchronization signal, so that microsecond-level camera synchronous shooting is realized, and better synchronization precision and imaging consistency are achieved. Under the condition that the data acquisition precision is ensured, the multi-camera array and the light source array are adopted instead of a module-form acquisition device, so that the data acquisition cost is reduced. Furthermore, three-dimensional reconstruction and / or matting processing are / is carried out based on the image sequences, collected by the multi-camera array, under the multiple visual angles, and a better three-dimensional reconstruction effect and / or matting quality can be obtained.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data acquisition system and method, a three-dimensional reconstruction method, a cutout method and related equipment. Background Art

[0002] 4D (four-dimensional) human body data can more accurately record various dynamic changes such as human activities and bone movements in a time series, and is widely used in various fields such as 3D (three-dimensional) digital human production, virtual reality (VR), and augmented reality (AR).

[0003] The dynamic 3D human body scanning solution is a 4D human body data collection solution. It uses a 4D synchronous scanning module to scan the 3D shape of a moving human body in real time, obtains 4D human body data in three-dimensional scanning and time dimensions, and then performs various analysis and processing on the 4D human body data through data analysis software.

[0004] Although the dynamic 3D human body scanning solution can collect relatively accurate 4D human body data, the high cost of the 4D synchronous scanning module limits the application scenarios of 4D human body data. Therefore, under the premise of ensuring the accuracy of 4D human body data collection, a 4D human body data collection solution with lower implementation cost and simpler is urgently needed. Summary of the invention

[0005] Various aspects of the present application provide a data acquisition system and method, a three-dimensional reconstruction method, a cutout method and related equipment for a lower-cost and simpler data acquisition solution.

[0006] An embodiment of the present application provides a data acquisition system, including: a light source array, a multi-camera array and a control device; the light source array and the multi-camera array are deployed around a target object; the light source array and the multi-camera array meet the lighting consistency requirement at the deployment position, and are used to provide consistent lighting information for the multi-camera array from multiple perspectives for the target object; the control device is used to send a hardware synchronization signal to the multi-camera array to control the multi-camera array to synchronously shoot the target object; the multi-camera array is used to synchronously shoot the target object from multiple perspectives according to the hardware synchronization signal sent by the control device under the condition of consistent lighting information provided by the light source array, so as to obtain image sequences under multiple perspectives; the control device is also used to receive the image sequences under multiple perspectives sent by the multi-camera array; according to the image sequences under multiple perspectives, the target object is subjected to cutout processing and / or three-dimensional reconstruction.

[0007] The embodiment of the present application also provides a data acquisition method, including: determining a target object, where a light source array and a multi-camera array are deployed around the target object, and the light source array and the multi-camera array meet lighting consistency requirements in the deployment position; controlling the light source array to provide consistent lighting information for the multi-camera array from multiple perspectives for the target object; sending a hardware synchronization signal to the multi-camera array to control the multi-camera array to synchronously shoot the target object from multiple perspectives according to the hardware synchronization signal and the consistent lighting information provided by the light source array to obtain image sequences under multiple perspectives; and performing cutout processing and / or three-dimensional reconstruction on the target object according to the image sequences under multiple perspectives.

[0008] An embodiment of the present application also provides a three-dimensional reconstruction method, including: acquiring an image sequence of a target object at multiple viewing angles, the image sequence at any viewing angle including multiple original images captured at multiple shooting moments at any viewing angle; performing three-dimensional reconstruction of the target object based on the original images corresponding to the same shooting moment in the image sequence at multiple viewing angles, and obtaining multiple initial mesh models corresponding to the target object at multiple shooting moments; extracting mesh data adapted to the surface attachment information from the multiple initial mesh models based on surface attachment information of the target object, to obtain multiple local mesh models; and generating multiple target parameterized object models corresponding to the target object at multiple shooting moments based on the multiple local mesh models and the basic parameterized object model.

[0009] An embodiment of the present application also provides a cutout method, comprising: acquiring an image sequence of a target object at multiple viewing angles, the image sequence at any viewing angle including multiple original images captured at multiple shooting times at any viewing angle; for the image sequence at any viewing angle, using a cutout model to perform cutout processing on multiple original images in the image sequence to obtain multiple initial cutout images; performing morphological processing on the multiple initial cutout images to obtain multiple intermediate cutout images; performing foreground detection on the multiple intermediate cutout images to obtain a foreground image sequence at any viewing angle, the foreground image sequence including multiple foreground images, and the foreground images including the target object at any viewing angle.

[0010] An embodiment of the present application also provides an electronic device, comprising: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory, and is used to execute the computer program to execute the steps in the data acquisition method, the three-dimensional reconstruction method or the cutout method.

[0011] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps in the data acquisition method, the three-dimensional reconstruction method or the cutout method.

[0012] An embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is enabled to implement the steps in the data acquisition method, the three-dimensional reconstruction method or the cutout method.

[0013] In this embodiment, the light source array provides consistent lighting information for the multi-camera array, and the multi-camera array is triggered to shoot synchronously by the control device with a hardware synchronization signal, so as to achieve microsecond-level camera synchronization shooting, with good synchronization accuracy and imaging consistency. While ensuring the accuracy of data acquisition, a multi-camera array and a light source array are used instead of a module-shaped acquisition device, which reduces the data acquisition cost. Furthermore, three-dimensional reconstruction and / or cutout processing can be performed based on the image sequence under multiple perspectives acquired by the multi-camera array, so as to obtain better three-dimensional reconstruction effect and / or cutout quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0015] Figure 1 A structural diagram of a data acquisition system provided in an embodiment of the present application;

[0016] Figure 2 is a diagram showing an exemplary arrangement of a multi-camera array;

[0017] Figure 3 is a schematic diagram of an exemplary structure of a photographing cavity;

[0018] Figure 4 A flow chart of a data collection method provided in an embodiment of the present application;

[0019] Figure 5 A schematic diagram for an exemplary generation of a target parameterized object model;

[0020] Figure 6 is an internal schematic diagram of an exemplary cutout model;

[0021] Figure 7 A flowchart of a three-dimensional reconstruction method provided in an embodiment of the present application;

[0022] Figure 8 A flowchart of a method for cutting out an image provided in an embodiment of the present application;

[0023] Fig. 9 is an exemplary application scenario diagram;

[0024] Fig.10A schematic diagram of the structure of a data acquisition device provided in an embodiment of the present application;

[0025] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0027] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the access relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. In the text description of the present application, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship. In addition, in the embodiments of the present application, "first", "second", "third", etc. are only used to distinguish the contents of different objects and have no other special meanings.

[0028] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The following is a detailed description of the technical solutions provided by each embodiment of the present application in conjunction with the accompanying drawings.

[0029] Figure 1 This is a schematic diagram of the structure of a data acquisition system provided in an embodiment of the present application. Figure 1, the data acquisition system may include: a light source array 10, a multi-camera array 20 and a control device 30. The light source array 10 and the multi-camera array 20 are deployed around the target object; the light source array 10 and the multi-camera array 20 meet the lighting consistency requirements at the deployment position, and are used to provide consistent lighting information for the multi-camera array 20 from multiple perspectives for the target object. The control device 30 is used to send a hardware synchronization signal to the multi-camera array 20 to control the multi-camera array 20 to synchronously shoot the target object. The multi-camera array 20 is used to synchronously shoot the target object from multiple perspectives according to the hardware synchronization signal sent by the control device 30 under the condition of consistent lighting information provided by the light source array 10, so as to obtain image sequences under multiple perspectives; the control device 30 is also used to: receive the image sequences under multiple perspectives sent by the multi-camera array; and perform cutout processing and / or three-dimensional reconstruction on the target object according to the image sequences under multiple perspectives.

[0030] In this embodiment, the target object includes, but is not limited to, real people, real animals, various objects, etc. If it is necessary to be able to move or change shape, the object can be a robot or manipulator, various children's toy cars, etc. The above various target objects can use the data acquisition system to collect 4D data when performing three-dimensional reconstruction.

[0031] In this embodiment, the light source array 10 is evenly distributed around the target object to ensure that light can be irradiated onto the target object from different directions, thereby reducing shadows and providing uniform lighting. The multi-camera array 20 is arranged around the target object to ensure that the target object is photographed from multiple perspectives to obtain high-quality image data of the target object at different perspectives, so that the target object can be photographed from multiple perspectives at the same time, which is convenient for later synthesis of a complete three-dimensional model or analysis of actions.

[0032] Specifically, multi-view image acquisition requires consistent lighting conditions at different camera viewing angles to avoid image inconsistency problems caused by lighting changes. Consistent lighting information means that the lighting conditions remain consistent in images taken at different viewing angles or at different time points. This does not necessarily mean that the lighting parameters (such as brightness, color temperature) are exactly the same, but that the impact of lighting on imaging results is predictable and controllable.

[0033] In this embodiment, the light source array 10 can be any system composed of multiple light source units, which are arranged together in a certain arrangement (such as linear, grid or other geometric shapes) to provide uniform or specific lighting effects. Through reasonable layout and control, the light source array 10 can achieve uniform illumination of the entire scene, which is crucial for high-quality image capture. Since the light source array 10 is modular, the number of light source units can be increased or decreased according to actual needs, or the position and direction of the light source units can be adjusted to meet flexible and changeable application requirements.

[0034] Optionally, the light source array 10 is a programmable light source array 10. Specifically, the programmable light source array 10 includes: a control unit and a plurality of programmable light source units. The control unit is configured to respond to the lighting configuration instruction and configure the lighting parameters of the plurality of light source units according to the deployment positions of the plurality of light source units, so that the plurality of light source units provide consistent lighting information for the multi-camera array 20 from multiple viewing angles.

[0035] In this embodiment, the programmable light source unit can adjust its lighting parameters, which enhances the flexibility and control accuracy of lighting, which means that the lighting conditions can be dynamically changed according to different application requirements. The lighting parameters include, but are not limited to: lighting mode, color temperature and illumination.

[0036] Further optionally, the color temperature of the light source unit is 5600K (Kelvin) ± 300K, that is, the color temperature fluctuates by 300K on the basis of 5600K. 5600K ± 300K is a standard daylight color temperature range, which can provide natural lighting similar to daytime, help to accurately reproduce the color of objects, and ensure clear capture of the movements and texture information of various objects such as real people, real animals, plants, buildings, etc. at different viewing angles.

[0037] Optionally, the illumination of the light source unit is 4500 Lux / m 2 (4500 lux per square meter). The illumination is set to 4500Lux / m 2 , which is a fairly high illumination level, can ensure sufficient light intensity to reduce shadows, improve image quality, and capture more details.

[0038] Optionally, when configuring lighting parameters, in addition to considering the deployment position of the light source unit, one or more of the following factors may also be considered:

[0039] 1. Application scenarios Different application scenarios have different requirements for lighting. For example, applications such as 3D scanning, facial recognition, industrial inspection, and biomedical imaging all have their own specific needs. 3D scanning: uniform and high-contrast lighting is required to accurately capture the geometric features of objects. Facial recognition: uniform and soft lighting is required to avoid shadows and highlight areas to ensure that facial features are clearly visible. Industrial inspection: high brightness and uniform lighting are required to clearly display surface defects or measure dimensions. Biomedical imaging: a light source of a specific wavelength (such as excitation light in a fluorescence microscope) is required, and the lighting must be uniform to avoid background noise.

[0040] 2. Environmental factors such as background light, temperature and humidity can affect the consistency of lighting. Background light: In indoor environments, minimize interference from external light sources. You can use a light shield or dark room to control background light. Temperature: The performance of the light source may be affected by temperature. Ensure that the working environment temperature is stable and use constant temperature equipment (such as air conditioning) to control the temperature. Humidity: High humidity may cause water mist on the optical components, affecting the quality of lighting. Use dehumidification equipment to control humidity.

[0041] 3. The reflectivity, color, shape and material characteristics of the target object will affect the lighting effect. Reflectivity: Objects with high reflectivity (such as metal) may require lower light intensity to avoid overexposure; objects with low reflectivity (such as black plastic) require higher light intensity. Color: For colored objects, white light or full-spectrum light sources are required to ensure the true reproduction of colors. Shape: Complex geometric shapes may require multi-directional light sources to avoid shadows and highlight areas. Material: Transparent or translucent materials may require backlighting or side lighting to highlight their internal structure.

[0042] 4. Image quality requirements According to the required image quality, adjust the lighting parameters to meet specific needs. Resolution: High-resolution imaging requires uniform and shadow-free lighting. Contrast: High-contrast imaging requires appropriate lighting angle and intensity to highlight the edges and details of objects. Signal-to-noise ratio: High signal-to-noise ratio imaging requires stable lighting and low background noise.

[0043] In this embodiment, in addition to controlling the light source array 10 to provide consistent lighting information, the control device 30 may also send a hardware synchronization signal to the multi-camera array 20 to control the multi-camera array 20 to synchronously shoot the target object.

[0044] In practical applications, the control device 30 may be composed of multiple (e.g., 8) computers for controlling the shooting, synchronization, and collection and preliminary processing of image data of the camera. The control device 30 may be connected to the camera via a high-speed data transmission interface for quickly transmitting image data from the camera to the control device 30. The control device 30 may also be configured with a high-speed and large-capacity storage system to store the original image data; the control device 30 may be built-in with graphic control software, which provides a dedicated graphical user interface for monitoring the shooting process, adjusting parameters, and starting / stopping shooting.

[0045] In this embodiment, the control device 30 controls the multi-camera array 20 to synchronously shoot the target object through a hardware synchronization signal, and the hardware synchronization signal can be understood as a hardware level signal belonging to the synchronization signal. In practical applications, the control device 30 can use a dedicated synchronization cable to connect the control device 30 to the synchronization input port of each camera. These cables usually contain one or more signal lines for transmitting synchronization signals, and the control device 30 triggers the synchronous acquisition of all cameras through hardware level signals. Specifically, the control device 30 outputs a synchronization signal (usually a TTL or RS-422 level signal), which is transmitted to the synchronization input port of each camera. The TTL level signal is usually defined as 5V representing a logic high level (1) and 0V representing a logic low level (0). The RS-422 level signal adopts a differential signal transmission method, which can provide better anti-interference ability and a longer transmission distance.

[0046] In multi-view acquisition, the synchronization accuracy between different cameras is crucial, and synchronization within microsecond accuracy is required to ensure the temporal consistency of the image sequence. In this embodiment, hardware level triggering is used to achieve synchronous acquisition of all cameras, providing a microsecond hardware synchronization mechanism to achieve microsecond camera synchronous shooting.

[0047] In this embodiment, the multi-camera array 20 is used to synchronously shoot the target object from multiple perspectives according to the hardware synchronization signal sent by the control device 30 under the condition of consistent lighting information provided by the light source array 10, so as to obtain image sequences under multiple perspectives.

[0048] In practical applications, the multi-camera array 20 synchronously shoots the target object to obtain image sequences at multiple viewing angles at each moment. The image sequences at multiple viewing angles at each moment can be regarded as 4D data of the target object. Figure 2 For example, Figure 2 The middle grey part represents the camera, and the multi-camera array 20 performs multi-angle synchronous shooting of real people.

[0049] Specifically, the multi-camera array 20 can capture images from multiple perspectives, and the multi-camera array 20 has the following characteristics:

[0050] 1. High resolution and high frame rate: In order to capture details and dynamic changes, the multi-camera array 20 has efficient image sensors and efficient image processing capabilities, thereby providing high resolution and high frame rate acquisition capabilities.

[0051] 2. Synchronization accuracy: In multi-view acquisition, the synchronization accuracy between different cameras is crucial. It must be synchronized within microsecond accuracy to ensure the temporal consistency of the image sequence.

[0052] 3. Imaging consistency: Multi-view acquisition requires imaging correction for different cameras, including consistency of exposure, white balance, aperture, and color space conversion to ensure that all camera images remain consistent.

[0053] 4. Lighting consistency: Multi-view acquisition requires maintaining consistent lighting conditions under different camera perspectives to avoid image inconsistency problems caused by lighting changes.

[0054] In this embodiment, the multi-camera array 20 includes a plurality of camera units, and the focal length of each camera unit is not limited. Further optionally, the multi-camera array 20 includes a first type of camera unit with a first focal length and a second type of camera unit with a second focal length, the first focal length is greater than the second focal length, and the first type of camera unit is used to capture a specific part of the target object. The specific part is, for example, a face, a hand, a leg, etc., but is not limited thereto.

[0055] For example, the multi-camera array 20 includes 58 high-resolution cameras, which can capture 12-megapixel high-definition video at a frame rate of 20 FPS (Frames Per Second) in real time, and each camera is responsible for capturing image information of the target object from different perspectives. Among the 58 high-resolution cameras, 8 are telephoto lens cameras for capturing high-definition close-ups of human faces; the other 50 high-resolution cameras are standard focal length cameras.

[0056] Optionally, in order to improve the reliability of shooting, a shooting cavity is also designed to install the multi-camera array 20, and the shooting cavity is flexibly designed as needed.

[0057] Optionally, the shooting cavity includes multiple installation layers of different heights, the multi-camera array 20 is dispersedly installed on the multiple installation layers, and the lenses of the camera units in the multi-camera array 20 are aimed at a target position in the shooting cavity, where the target position is the position where the target object is located. The target position is, for example, a center position.

[0058] For example, the shooting cavity is Figure 3The polyhedron shown (for example, a 16-hedron) may be 3 meters in diameter and 2.5 meters in height, and a plurality of cameras arranged on the polyhedron may form a multi-camera array 20. The number of cameras on each face is the same; each face may be designed to include several mounting layers according to actual conditions to accommodate cameras at different heights. The multiple mounting layers of each face may be located at different heights to cover various angles from the ground to the head. For example, three layers of mounting racks may be provided, namely, a bottom layer, a middle layer, and a top layer, with a certain number of cameras installed on each layer to ensure all-round coverage from low to high. The lenses of all cameras are aimed at the center of the polyhedron, where the target object is standing. This design ensures that all cameras can capture the target object from a better or even optimal angle, reducing image differences caused by different viewing angles. Using a camera with a standard lens or a telephoto lens ensures that the facial and body details of the target object can be clearly captured from different distances and angles.

[0059] Further optionally, before data collection, in order to ensure the accuracy and consistency of all camera units in the multi-camera array 20 working in the same coordinate system and to ensure that the imaging characteristics of all cameras are consistent, a calibration device can be used to calibrate the internal and external parameters of each camera unit in the multi-camera array 20 using a calibration plate that rotates at a set rotation angle.

[0060] In practical applications, the calibration device may be the control device 30 or other specialized devices with calibration functions, and there is no limitation to this.

[0061] Camera calibration refers to the calibration of the camera's internal and external parameters. During the internal parameter calibration, the internal parameters of each camera unit are determined, including focal length, sensor center point position, pixel size, etc. During the external parameter calibration, the position and posture (rotation and translation) of the camera unit relative to the world coordinate system are determined.

[0062] In practical applications, the internal and external parameters of each camera can be calibrated separately, the entire field can be divided into different field of view directions, and the calibration plate can be adjusted with a rotation angle of ±30 degrees to ensure the calibration accuracy of different camera units at different angles. Common feature points are found in the calibration plate images taken by all camera units and mapped to the same coordinate system. The errors of all cameras when projecting common feature points are minimized through optimization algorithms (such as nonlinear least squares method) to obtain accurate internal and external parameters, that is, the internal and external parameters of the camera unit are obtained by minimizing the projection error.

[0063] Further optionally, before data collection, in order to ensure color consistency among multiple camera units, the calibration device is also used to perform color correction on each camera unit in the multi-camera array 20 using a standard light source and a standard color card. Color correction using a standard light source and a standard color card can obtain a color correction matrix (Color Correction Matrix, CCM) of each camera unit.

[0064] In practical applications, when determining the color correction matrix of each camera unit, a standard color card with known colors can be placed at the center of the multi-camera array 20, and all camera units can shoot it. Before shooting, adjust the hardware parameters of all camera units, such as the aperture, exposure time, and white balance, so that the hardware parameters of all camera units remain consistent. Under standard light sources, make the standard color card face the camera unit to be calibrated directly, and obtain the calibration plate image taken by the camera unit. Determine the color error between the RGB values ​​of each color block of the standard color card in the calibration plate image and the RGB values ​​of each color block of the known standard color card; find a color correction matrix through the least squares method to minimize the color error between the calibration plate image taken by the camera unit and the true color of the standard color card.

[0065] In an embodiment of the present application, the control device is also used to: receive a sequence of images at multiple viewing angles sent by a multi-camera array; perform cutout processing and / or three-dimensional reconstruction on the target object according to the sequence of images at multiple viewing angles. For more information about cutout processing and three-dimensional reconstruction, please refer to the subsequent content. Through cutout processing, foreground images of the target object at different times can be collected; through three-dimensional reconstruction, a three-dimensional model of the target object at different times, that is, 4D data of the target object, can be collected, and better three-dimensional reconstruction effect and / or cutout quality can be obtained.

[0066] In the data acquisition system provided by the embodiment of the present application, the light source array provides consistent lighting information for the multi-camera array, and triggers the multi-camera array to shoot synchronously by the control device with a hardware synchronization signal, thereby achieving microsecond-level camera synchronization shooting, and having good synchronization accuracy and imaging consistency. While ensuring the accuracy of data acquisition, a multi-camera array and a light source array are used instead of a module-shaped acquisition device, thereby reducing the cost of data acquisition. Furthermore, three-dimensional reconstruction and / or cutout processing are performed based on the image sequences from multiple perspectives acquired by the multi-camera array, which can obtain better three-dimensional reconstruction effects and / or cutout quality.

[0067] In some optional embodiments, in order to improve the quality of cutouts and / or three-dimensional reconstruction, the control device is also used to perform color correction on the image sequences under multiple perspectives before performing cutout processing and / or three-dimensional reconstruction on the target object according to the image sequences under multiple perspectives.

[0068] Figure 4 A flow chart of a data collection method provided in an embodiment of the present application. Figure 4 , the method may include the following steps:

[0069] 401. Determine a target object, where a light source array and a multi-camera array are deployed around the target object, and the light source array and the multi-camera array meet illumination consistency requirements at deployment positions;

[0070] 402. Control the light source array to provide consistent lighting information for the multi-camera array from multiple perspectives with respect to the target object.

[0071] 403. Send a hardware synchronization signal to the multi-camera array to control the multi-camera array to synchronously shoot the target object from multiple perspectives according to the hardware synchronization signal and consistent lighting information provided by the light source array to obtain image sequences under multiple perspectives.

[0072] 404. Perform cutout processing and / or three-dimensional reconstruction on the target object according to the image sequences under multiple viewing angles.

[0073] Further optionally, the above method also includes: responding to the lighting configuration instruction, configuring the lighting parameters of the multiple light source units according to the deployment positions of the multiple programmable light source units in the light source array, so that the multiple light source units provide consistent lighting information for the multi-camera array from multiple perspectives.

[0074] about Figure 4 The implementation method of each step in the illustrated embodiment can refer to the relevant content in the aforementioned embodiment, which will not be repeated here.

[0075] The technical solution provided by the embodiment of the present application is that the light source array provides consistent lighting information for the multi-camera array, and the multi-camera array is triggered to shoot synchronously by the control device with a hardware synchronization signal, so as to achieve microsecond-level camera synchronization shooting, with good synchronization accuracy and imaging consistency. While ensuring the accuracy of data acquisition, the multi-camera array and light source array are used instead of module-shaped acquisition equipment, which reduces the cost of data acquisition.

[0076] The following is an introduction to the 3D reconstruction process:

[0077] In practical applications, there is no restriction on the 3D reconstruction method, which includes, but is not limited to, NeuS (Neural Surface Reconstruction) method, multi-view stereo matching method, and the like.

[0078] Further optionally, in order to improve the quality of three-dimensional reconstruction, the image sequence at any perspective includes multiple original images captured at multiple shooting moments at any perspective; the implementation method of three-dimensional reconstruction of the target object based on the image sequence at multiple perspectives includes: performing three-dimensional reconstruction of the target object based on the original images corresponding to the same shooting moment in the image sequence at multiple perspectives, and obtaining multiple initial mesh models corresponding to the target object at multiple shooting moments; according to surface attachment information of the target object, extracting mesh data adapted to the surface attachment information from the multiple initial mesh models to obtain multiple local mesh models; based on the multiple local mesh models and the basic parametric object model, generating multiple target parametric object models corresponding to the target object at multiple shooting moments.

[0079] Specifically, during 3D reconstruction, the original images of multiple perspectives at each shooting moment captured by the multi-camera array are obtained, and the original images of multiple perspectives at each shooting moment are 3D reconstructed to obtain the initial mesh model at each shooting moment. The initial mesh model is a 3D mesh model, which is composed of a series of connected polygons (usually triangles) that construct an approximation of the object surface by sharing vertices.

[0080] Further optionally, the original image is firstly subjected to a cutout process to obtain a foreground image to remove background interference, and then three-dimensional reconstruction is performed based on the foreground image, thereby improving the accuracy and reliability of the three-dimensional reconstruction.

[0081] Based on the above, according to the original images corresponding to the same shooting moment in the image sequence under multiple perspectives, the target object is three-dimensionally reconstructed to obtain multiple initial mesh models corresponding to the target object at multiple shooting moments. The implementation method is as follows: according to the image sequence under multiple perspectives, the target object is cut out to obtain a foreground image sequence under multiple perspectives, the foreground image sequence includes multiple foreground images corresponding to the multiple shooting moments, and the foreground image includes the target object under the corresponding perspectives; according to the foreground images corresponding to the same shooting moment in the foreground image sequence under multiple perspectives, the target object is three-dimensionally reconstructed to obtain multiple initial mesh models corresponding to the target object at multiple shooting moments.

[0082] Specifically, during 3D reconstruction, the original images of multiple perspectives at each shooting moment captured by the multi-camera array are obtained; the original images at each shooting moment are subjected to cutout processing to obtain the foreground image at each shooting moment; 3D reconstruction is performed based on the foreground images of multiple perspectives at each shooting moment to obtain the initial mesh model at each shooting moment. For more information about cutout processing, please refer to the subsequent content.

[0083] In this embodiment, for the initial mesh model at each shooting moment, mesh data adapted to the surface attachment information is extracted from the initial mesh model according to the surface attachment information of the target object to obtain multiple local mesh models. The surface attachment information refers to the information of the elements attached to the surface of the target object, and the surface attachment information includes, but is not limited to, clothing, accessories, hair, etc. Taking the target object as a real person as an example, the local mesh model can be understood as a 3D mesh model of the clothing part, including 3D mesh data of the clothing part of the real person, and the clothing part includes one or more of clothing, accessories, hair, etc.

[0084] In this embodiment, for each shooting moment, a target parameterized object model is generated according to the local grid model and the basic parameterized object model at the shooting moment. The target parameterized object model is a more realistic and personalized 3D model that can represent the details of the target object in different states.

[0085] The basic parametric object model refers to a model that can use a set of parameters to define and control the geometric shape parameters and posture parameters of an object. The basic parametric object model includes, but is not limited to: SMPL (Skinned Multi-Person Linear model, a human body parametric representation model based on skin vertices).

[0086] In practical applications, the local grid model and the basic parameterized object model at each shooting moment can be fused to generate a target parameterized object model at each shooting moment. Further optionally, in order to improve the accuracy of three-dimensional reconstruction, multiple target parameterized object models corresponding to the target object at multiple shooting moments are generated according to multiple local grid models and the basic parameterized object model. The implementation method is as follows: according to the foreground images corresponding to the same shooting moment in the foreground image sequence under multiple perspectives, multiple multi-dimensional parameterized representation information corresponding to the target object at multiple shooting moments is determined; the basic parameterized object model is adjusted according to the multiple multi-dimensional parameterized representation information to obtain multiple intermediate state parameterized object models; and multiple target parameterized object models are generated according to the multiple intermediate state parameterized object models and the multiple local grid models.

[0087] In this embodiment, multiple multi-dimensional parameterized representation information at each moment can be identified based on the foreground image at each shooting moment. The parameterized representation information corresponding to different target objects may be different, and the parameterized representation information includes, but is not limited to: face shape parameters, expression coefficients, head posture parameters, hand shape parameters, hand posture parameters, body shape parameters, body posture parameters, etc.

[0088] Based on the above, the target object is a human object, and the implementation method of determining multiple multi-dimensional parameterized representation information corresponding to the target object at multiple shooting moments according to the foreground images corresponding to the same shooting moment in the foreground image sequence under multiple perspectives is as follows: determining at least two parameters of the facial parameters, expression coefficients, head posture parameters, hand shape parameters, hand posture parameters, body shape parameters and body posture parameters corresponding to the human object at each shooting moment according to the foreground images corresponding to the same shooting moment in the foreground image sequence under multiple perspectives; and using at least two parameters of the facial parameters, expression coefficients, head posture parameters, hand shape parameters, hand posture parameters, body shape parameters and body posture parameters corresponding to the human object at each shooting moment as the multi-dimensional parameterized representation information corresponding to the human object at each shooting moment.

[0089] It can be understood that face shape recognition is performed on the foreground image to obtain face shape parameters; expression recognition is performed on the foreground image to obtain expression coefficients; head posture recognition is performed on the foreground image to obtain head posture parameters; hand shape recognition is performed on the foreground image to obtain hand shape parameters; hand posture recognition is performed on the foreground image to obtain hand posture parameters; body shape recognition is performed on the foreground image to obtain body shape parameters; body posture recognition is performed on the foreground image to obtain body posture parameters.

[0090] It can be understood that the multi-dimensional parameterized representation information refers to a set of parameters used to describe the target object in multiple aspects (such as form, posture, expression, etc.). Therefore, the basic parameterized object model is adjusted according to the multi-dimensional parameterized representation information at each shooting moment to obtain an intermediate parameterized object model at each shooting moment. The intermediate parameterized object model is a more realistic and personalized 3D model that can represent the details of the target object in different states.

[0091] In this embodiment, multiple target parameterized object models are generated based on multiple intermediate parameterized object models and multiple local grid models. That is, the intermediate parameterized object model and the local grid model at each shooting moment are fused to obtain the target parameterized object model at each shooting moment. The target parameterized object model is a more realistic and personalized 3D model that can represent the details of the target object in different states.

[0092] For ease of understanding, the following Figure 5 Description: An initial mesh model of a real person is established through three-dimensional reconstruction, and a local mesh model (that is, the mesh model of the clothing part) is extracted from the initial mesh model; an intermediate parametric object model is obtained by adjusting the basic parametric object model, and the local mesh model and the intermediate parametric object model are fused to obtain a target parametric object model, that is, a realistic and personalized 3D human body model.

[0093] The following is an introduction to the cutout process:

[0094] In practical applications, there is no restriction on the cutout processing method. Further optionally, in order to improve the cutout accuracy, the image sequence at any perspective includes multiple original images captured at multiple shooting moments at any perspective; according to the image sequence at multiple perspectives, the implementation method of the cutout processing of the target object is as follows: for the image sequence at any perspective, the multiple original images in the image sequence are cutout processed using the cutout model to obtain multiple initial cutout images; the multiple initial cutout images are morphologically processed to obtain multiple intermediate cutout images; the multiple intermediate cutout images are foreground detected to obtain a foreground image sequence at any perspective, the foreground image sequence includes multiple foreground images, and the foreground images include the target object at any perspective.

[0095] The morphological processing includes, but is not limited to, dilation operation, erosion operation, opening operation or closing operation, etc.

[0096] It can be understood that the initial cutout image is cut out from the original image using the cutout model, and the initial cutout image is also an image with the target object as the foreground target. The initial cutout image is subjected to morphological processing to obtain an intermediate cutout image, and then the intermediate cutout image is subjected to foreground detection to determine the foreground area in the intermediate cutout image, and the foreground area is segmented from the intermediate cutout image to obtain a more accurate foreground image. The foreground image is segmented from the original image, which has undergone cutout model cutting, morphological processing and foreground detection, and can be considered as a two-stage foreground detection method to obtain a more accurate foreground image.

[0097] Further optionally, in order to improve the accuracy of cutouts, before performing cutout processing on the target object according to the image sequences under multiple perspectives, for the image sequence at any perspective, color correction is performed on the multiple original images in the image sequence using the respective color correction matrices of the multiple camera units in the multi-camera array; wherein the color correction matrix between the multiple camera units represents the color consistency between the multiple camera units.

[0098] In practical applications, there is no restriction on the network structure of the cutout model. Further optionally, in order to improve the cutout accuracy, the cutout model includes a backbone network, a semantic segmentation network and an edge extraction network; the cutout model is used to perform cutout processing on multiple original images in the image sequence to obtain multiple initial cutout images corresponding to multiple shooting moments. The implementation method is as follows: for any original image, the backbone network is used to extract the image features of multiple channels of the original image, and the image features of the multiple channels are subjected to feature aggregation processing to obtain aggregated image features, and the image features of the multiple channels and the aggregated image features are fused to obtain fused image features; the fused image features are input into the semantic segmentation network for processing to obtain a semantic segmentation image; the image features of the multiple channels are input into the edge detection network for processing to obtain the edge information of the target object, and a mask image is generated according to the edge information of the target object; the semantic segmentation image and the mask image are fused to obtain the initial cutout image corresponding to the original image.

[0099] See also Figure 6 , the original image is input into the backbone network for feature extraction, and the image features of multiple channels of the original image can be obtained. The backbone network includes, but is not limited to: DenseNet (densely connected convolutional network), residual network, etc. The image features of multiple channels are, for example: image features of the red color channel, image features of the green color channel, image features of the blue color channel. Feature aggregation processing integrates the feature information of these different channels to obtain a richer and more expressive feature representation. Feature aggregation processing includes, but is not limited to: channel addition, channel averaging, channel splicing, etc. The image features of multiple channels are fused with the aggregated image features to obtain fused image features. The semantic segmentation network performs semantic segmentation based on the fused image features to obtain a semantically segmented image.

[0100] Semantic segmentation aims to assign a category label to each pixel in an image. Semantic segmentation networks based on fused image features usually improve segmentation accuracy and details by extracting and fusing features at different levels. The edge detection network performs edge detection based on image features of multiple channels, determines the edge information of the target object in the original image, and sets the pixel values ​​of the pixels surrounded by the edge of the target object in the original image to the specified pixel values ​​corresponding to the foreground, and the pixel values ​​of the pixels not surrounded by the edge of the target object to the specified pixel values ​​corresponding to the background, thereby obtaining a mask image. The specified pixel value corresponding to the foreground is, for example, 1, and the specified pixel value corresponding to the background is, for example, 0. The semantic segmentation image and the mask image are fused to obtain the initial cutout image.

[0101] In this embodiment, there is no restriction on the training method of the cutout model. Further optionally, full use is made of training data sets of different qualities to improve the generalization ability and accuracy of the cutout model. The training process of the cutout model includes: obtaining a first training data set and a second training data set, the quality of the first training data set is lower than that of the second training data set; using the first training data set to pre-train the initial cutout model to obtain an intermediate cutout model; using the second training data set to fine-tune the intermediate cutout model to obtain the cutout model.

[0102] In practical applications, the first training data set and the second training data set include multiple labeled sample images, that is, labeled sample images. There is no restriction on the types of labels that the labeled sample images have. Further, optionally, the training data set with multiple labels can be fully utilized to improve the accuracy and robustness of the cutout model. Based on this, the first training data set and / or the second training data set

[0103] A labeled sample image with multiple labels. The multiple labels corresponding to the labeled sample image include at least one of the following: a foreground label indicating that the pixel belongs to the foreground, a foreground label indicating that the pixel belongs to the background, a transparency label indicating the transparency of the pixel, and an uncertain label indicating that the pixel is uncertain whether it belongs to the foreground or the background.

[0104] During model training, the labeled sample image is input into the matting model to be trained to obtain the matting image output by the matting model; the loss function is calculated based on the prediction results of each pixel in the output matting image and the label corresponding to each pixel, and the model parameters of the matting model are adjusted with the goal of minimizing the loss function. The above steps are iterated until the model training is completed to obtain a trained matting model.

[0105] Figure 7 A flowchart of a three-dimensional reconstruction method provided in an embodiment of the present application. Figure 7 , the method may include the following steps:

[0106] 701. Acquire an image sequence of a target object at multiple viewing angles, wherein the image sequence at any viewing angle includes multiple original images captured at multiple capturing moments at any viewing angle;

[0107] 702. Perform three-dimensional reconstruction of the target object according to the original images corresponding to the same shooting time in the image sequences under multiple viewing angles, and obtain multiple initial mesh models corresponding to the target object at multiple shooting times;

[0108] 703. According to the surface attachment information of the target object, mesh data adapted to the surface attachment information is extracted from the multiple initial mesh models to obtain multiple local mesh models;

[0109] 704. Generate multiple target parameterized object models corresponding to the target object at multiple shooting moments according to the multiple local grid models and the basic parameterized object model.

[0110] For the implementation of each step of this embodiment, please refer to the relevant content of the aforementioned embodiment, which will not be repeated here.

[0111] The technical solution provided by the embodiment of the present application performs three-dimensional reconstruction based on the original image obtained by shooting the target object from multiple perspectives to obtain an initial mesh model; extracts a local mesh model adapted to the surface attachment information of the target object from the initial mesh model; and generates a target parameterized object model of the target object based on the local mesh model and the basic parameterized object model. Thus, the quality of three-dimensional reconstruction is improved, and a more realistic and personalized 3D model is obtained, which can represent the details of the target object in different states.

[0112] Figure 8 This is a flow chart of a method for cutting out a picture provided by an embodiment of the present application. Figure 8 , the method may include the following steps:

[0113] 801. Acquire an image sequence of a target object at multiple viewing angles, wherein the image sequence at any viewing angle includes a plurality of original images captured at multiple capturing moments at any viewing angle;

[0114] 802. For an image sequence at any viewing angle, use a cutout model to perform cutout processing on a plurality of original images in the image sequence to obtain a plurality of initial cutout images;

[0115] 803. Perform morphological processing on the multiple initial cutout images to obtain multiple intermediate cutout images;

[0116] 804. Perform foreground detection on the multiple intermediate state cutout images to obtain a foreground image sequence at any viewing angle, wherein the foreground image sequence includes multiple foreground images, and the foreground image includes a target object at any viewing angle.

[0117] For the implementation of each step of this embodiment, please refer to the relevant content of the aforementioned embodiment, which will not be repeated here.

[0118] The technical solution provided in the embodiment of the present application segments a foreground image from an original image, and after cutout model cutout, morphological processing and foreground detection, it can be considered as a two-stage foreground detection method, which can obtain a more accurate foreground image.

[0119] In order to better understand the technical solution of the embodiment of the present application, Fig. 9 A specific scenario implementation example is introduced.

[0120] In various fields such as 3D (three-dimensional) digital human production, virtual reality (VR), augmented reality (AR), etc., the data acquisition system mentioned in the above embodiment is used as a 4D human body data acquisition system, and the 4D human body data acquisition system is used to collect 4D human body data, and the 4D human body data is used to create a 3D digital human or virtual character, etc. In the application scenario that pursues high fidelity and high real-time performance, it has better 3D reconstruction effect and efficiency.

[0121] Specifically, the 4D human body data acquisition system includes: a light source array, a multi-camera array and an acquisition workstation (i.e., a control device); the light source array is used to provide consistent lighting information to each viewing angle of the multi-camera array under the control of the acquisition workstation; the multi-camera array is used to synchronously shoot a real person from multiple viewing angles under the control of the acquisition workstation to obtain multiple original images at different times; the acquisition workstation is used to perform color correction on multiple original images at multiple times to obtain multiple intermediate images at multiple times; the intermediate images are cut out to obtain a foreground image including the real person; and the multiple foreground images at each time are processed. Three-dimensional reconstruction, to obtain a complete 3D mesh model at each moment; extract the local 3D mesh model at each moment from the complete 3D mesh model at each moment, the local 3D mesh model includes the 3D mesh data of the real person's clothing; extract the real person's multi-dimensional parametric representation information according to multiple foreground images at each moment, and adjust the basic parametric human body model according to the parametric information representation of multiple dimensions to obtain the real person's intermediate parametric human body model; generate the target parametric human body model at each moment according to the intermediate parametric human body model and the local 3D mesh model, and use the parametric human body models at multiple moments as 4D real person data. Subsequently, 3D digital people or virtual characters in virtual scenes can be constructed based on 4D real person data.

[0122] Specifically, the 4D human body data acquisition system has the following features:

[0123] 1. High resolution and high frame rate: The multi-camera array includes multiple high-resolution cameras, including telephoto lens cameras. The high-resolution cameras can capture 12-megapixel high-definition video at a frame rate of 20FPS in real time, achieving real-time scanning at a high frame rate; each camera is responsible for capturing the RGB information of the human body from different perspectives. 5 telephoto lens cameras are used to capture high-definition close-ups of human faces. The multi-perspective multi-camera array captures images from different angles through multiple cameras, which can provide more complete three-dimensional dynamic imaging of the human body.

[0124] 2. Microsecond synchronization accuracy: Use hardware level triggering to achieve microsecond camera synchronization.

[0125] 3. Imaging consistency: Multi-view acquisition requires imaging correction for different cameras, including consistency of exposure, white balance, aperture or color space conversion to ensure that all camera imaging remains consistent.

[0126] 4. Lighting consistency: Multi-view acquisition requires maintaining consistent lighting conditions under different camera perspectives to avoid image inconsistency problems caused by lighting changes.

[0127] 5. Convenient data storage and management: The camera is connected through a high-speed data transmission interface to quickly transfer image data from the camera to the acquisition workstation; equipped with a high-speed and large-capacity storage system to save the original image data.

[0128] 6. Consider cost-effectiveness while ensuring performance: By building a low-cost, high-quality multi-view industrial camera array, the same high-quality 3D real-person digital human effect can be achieved at less than one-tenth of the cost.

[0129] 7. Environmental adaptability: The multi-camera array includes multiple RGB (color) cameras. The RGB camera system is more adaptable to ambient light and is not as susceptible to interference from sunlight or other light sources as infrared or laser systems.

[0130] 8. Data richness: RGB cameras are able to capture rich color information, providing more natural and realistic visual effects. In contrast, infrared and laser systems mainly provide geometric information.

[0131] Therefore, a low-cost, high-quality 4D real-person digital human data acquisition system is proposed, which covers a multi-camera array, a light source array, and an acquisition workstation to optimize the quality and efficiency of dynamic 3D human reconstruction.

[0132] Specifically Fig. 9For example, before collecting 4D real-person data, the multi-camera array is calibrated for multi-view cameras to determine the camera parameters such as the intrinsic and extrinsic parameters of each camera, and the hardware parameters such as the white balance, exposure parameters and aperture of each camera can also be corrected; color correction is performed on each camera to determine the color correction parameters (i.e., the color correction matrix); the multi-camera array is controlled to collect images of real people with a microsecond-level synchronous shooting mechanism under the condition of consistent lighting, and a multi-view synchronous video is obtained. Image enhancement and portrait cutout processing are performed on each original image in the multi-view synchronous video to obtain multiple human images under multiple perspectives. Three-dimensional reconstruction is performed based on multiple human images under multiple perspectives to obtain a complete 3D mesh model corresponding to the real person, and a local 3D mesh model corresponding to the clothing part is extracted from the complete 3D mesh model; the local 3D mesh model and the parametric human body model representing the multi-dimensional parametric representation information of the real person are synthesized to obtain the final parametric human body model of the real person. The final parametric human body model can be used to establish a 3D digital human or a virtual character in a virtual scene, etc., providing strong data support and technical support for the production of high-quality 3D digital humans or virtual characters.

[0133] Fig.10 This is a structural diagram of a data acquisition device provided in an embodiment of the present application. Fig.10 , the device comprises:

[0134] A determination module 10 is used to determine a target object, where a light source array and a multi-camera array are deployed around the target object, and the light source array and the multi-camera array meet the lighting consistency requirement at the deployment position;

[0135] An illumination control module 20, for controlling the light source array to provide consistent illumination information for the multi-camera array from multiple viewing angles with respect to a target object;

[0136] The shooting control module 30 is used to send a hardware synchronization signal to the multi-camera array to control the multi-camera array to synchronously shoot the target object from multiple perspectives according to the hardware synchronization signal and the consistent lighting information provided by the light source array to obtain image sequences under multiple perspectives;

[0137] The processing module 40 is used to perform cutout processing and / or three-dimensional reconstruction on the target object according to the image sequences under multiple viewing angles.

[0138] Further optionally, the lighting control module 20 is specifically used to: respond to lighting configuration instructions, configure lighting parameters of multiple light source units according to the deployment positions of multiple programmable light source units in the light source array, so that the multiple light source units provide consistent lighting information for the multi-camera array from multiple perspectives.

[0139] Further optionally, the image sequence at any viewing angle includes a plurality of original images captured at a plurality of capturing moments at any viewing angle; when the processing module performs three-dimensional reconstruction, it is specifically used to: perform three-dimensional reconstruction on the target object according to the original images corresponding to the same capturing moment in the image sequence at the plurality of viewing angles, and obtain a plurality of initial mesh models corresponding to the target object at the plurality of capturing moments;

[0140] According to the surface attachment information of the target object, mesh data adapted to the surface attachment information is extracted from the multiple initial mesh models to obtain multiple local mesh models; according to the multiple local mesh models and the basic parameterized object model, multiple target parameterized object models corresponding to the target object at multiple shooting moments are generated.

[0141] Further optionally, when the processing module generates the initial mesh model, it is specifically used to: perform cutout processing on the target object according to the image sequence under multiple perspectives to obtain a foreground image sequence under multiple perspectives, the foreground image sequence includes multiple foreground images corresponding to multiple shooting moments, and the foreground image includes the target object under the corresponding perspectives; perform three-dimensional reconstruction of the target object according to the foreground image corresponding to the same shooting moment in the foreground image sequence under multiple perspectives to obtain multiple initial mesh models corresponding to the target object at multiple shooting moments.

[0142] Further optionally, when the processing module generates a target parameterized object model, it is specifically used to: determine multiple multi-dimensional parametric representation information corresponding to the target object at multiple shooting moments based on the foreground images corresponding to the same shooting moment in the foreground image sequence under multiple perspectives; adjust the basic parameterized object model according to the multiple multi-dimensional parameterized representation information to obtain multiple intermediate parameterized object models; generate multiple target parameterized object models based on the multiple intermediate parameterized object models and the multiple local grid models.

[0143] Further optionally, when the target object is a human object, the processing module determines the parameterized representation information, specifically for: determining at least two parameters of the facial parameters, expression coefficients, head posture parameters, hand shape parameters, hand posture parameters, body shape parameters and body posture parameters corresponding to the human object at each shooting moment according to the foreground image corresponding to the same shooting moment in the foreground image sequence under multiple perspectives; and using at least two parameters of the facial parameters, expression coefficients, head posture parameters, hand shape parameters, hand posture parameters, body shape parameters and body posture parameters corresponding to the human object at each shooting moment as the multi-dimensional parameterized representation information corresponding to the human object at each shooting moment.

[0144] Further optionally, the image sequence at any perspective includes multiple original images captured at multiple shooting moments at any perspective; when the processing module performs cutout processing, it is specifically used to: for the image sequence at any perspective, use a cutout model to perform cutout processing on the multiple original images in the image sequence to obtain multiple initial cutout images; perform morphological processing on the multiple initial cutout images to obtain multiple intermediate cutout images; perform foreground detection on the multiple intermediate cutout images to obtain a foreground image sequence at any perspective, the foreground image sequence includes multiple foreground images, and the foreground images include target objects at any perspective.

[0145] Further optionally, the cutout model includes a backbone network, a semantic segmentation network and an edge extraction network; when the processing module uses the cutout model to perform cutout processing on multiple original images in the image sequence to obtain multiple initial cutout images corresponding to multiple shooting moments, it is specifically used for: for any original image, using the backbone network to extract image features of multiple channels of the original image, and performing feature aggregation processing on the image features of the multiple channels to obtain aggregated image features, and fusing the image features of the multiple channels and the aggregated image features to obtain fused image features; inputting the fused image features into the semantic segmentation network for processing to obtain a semantic segmentation image; inputting the image features of the multiple channels into the edge detection network for processing to obtain edge information of the target object, and generating a mask image based on the edge information of the target object; fusing the semantic segmentation image and the mask image to obtain the initial cutout image corresponding to the original image.

[0146] Further optionally, the training process of the cutout model includes: obtaining a first training data set and a second training data set, the quality of the first training data set is lower than that of the second training data set; using the first training data set to pre-train the initial cutout model to obtain an intermediate cutout model; using the second training data set to fine-tune the intermediate cutout model to obtain the cutout model.

[0147] Further optionally, the first training data set, and / or the second training data set includes: labeled sample images with multiple labels, the multiple labels including at least one of the following: a foreground label indicating that the pixel belongs to the foreground, a foreground label indicating that the pixel belongs to the background, a transparency label indicating the transparency of the pixel, and an uncertain label indicating that the pixel is uncertain whether it belongs to the foreground or the background.

[0148] Further optionally, the processing module is also used to: before performing cutout processing on the target object according to the image sequence under multiple perspectives, for the image sequence at any perspective, use the respective color correction matrices of multiple camera units in the multi-camera array to perform color correction on multiple original images in the image sequence; wherein the color correction matrix between the multiple camera units represents the color consistency between the multiple camera units.

[0149] Fig.10 The device shown can execute the method shown in the above embodiment, and its implementation principle and technical effect will not be described in detail. Fig.10 The specific manner in which each module and unit of the device shown performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0150] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 401 to 403 can be device A; for another example, the execution subject of steps 401 and 402 can be device A, and the execution subject of step 403 can be device B; and so on.

[0151] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 401, 402, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0153] Fig.11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.11 As shown, the electronic device includes: a memory 11 and a processor 12;

[0154] The memory 11 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc.

[0155] The memory 11 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0156] The processor 12 is coupled to the memory 11 and is used to execute the computer program in the memory 11 to execute the steps in the data acquisition method, the three-dimensional reconstruction method or the cutout method.

[0157] Optional, such as Fig.11 As shown, the electronic device also includes: a communication component 13, a display 14, a power component 15, an audio component 16 and other components. Fig.11 Only some components are shown schematically, which does not mean that the electronic device only includes Fig.11 In addition, Fig.11 The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the electronic device. The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT (Internet of Things) device, or a server device such as a conventional server, a cloud server, or a server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Fig.11 If the electronic device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Fig.11 Components within the dashed box.

[0158] For the detailed implementation process of the processor executing each action, please refer to the relevant description in the aforementioned method embodiment or device embodiment, which will not be repeated here.

[0159] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be performed by an electronic device in the above method embodiment.

[0160] Accordingly, an embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is enabled to implement each step in the above method embodiment that can be executed by an electronic device.

[0161] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi (Wireless Fidelity), 2G (2 Generation, 2 Generation), 3G (3 Generation, 3 Generation), 4G (4 Generation, 4 Generation) / LTE (long Term Evolution), 5G (5 Generation, 5 Generation) and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (Near Field Communication, NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0162] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0163] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0164] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0165] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0166] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0167] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0168] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0169] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0170] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0171] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, Phase Change RAM (PRAM), Static Random-Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0172] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0173] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data acquisition system, characterized in that: include: A light source array, a multi-camera array and a control device; the light source array and the multi-camera array are deployed around a target object; The light source array and the multi-camera array meet lighting consistency requirements at the deployment position, and are used to provide consistent lighting information for the multi-camera array from multiple viewing angles for the target object; The control device is used to send a hardware synchronization signal to the multi-camera array to control the multi-camera array to synchronously shoot the target object; The multi-camera array is used to synchronously shoot the target object from multiple viewing angles according to the hardware synchronization signal sent by the control device under the condition of consistent lighting information provided by the light source array, so as to obtain image sequences under the multiple viewing angles; The control device is further used to receive image sequences at multiple viewing angles sent by the multi-camera array; and perform cutout processing and / or three-dimensional reconstruction on the target object according to the image sequences at multiple viewing angles.

2. The system according to claim 1, characterized in that The multi-camera array includes a first type of camera unit having a first focal length and a second type of camera unit having a second focal length, the first focal length is greater than the second focal length, and the first type of camera unit is used to capture a specific part of the target object.

3. The system according to claim 1, characterized in that Also includes: A shooting cavity, wherein the shooting cavity comprises a plurality of installation layers of different heights, the multi-camera array is dispersedly installed on the plurality of installation layers, and the lenses of the camera units in the multi-camera array are aimed at a target position in the shooting cavity, wherein the target position is the position where the target object is located.

4. The system according to claim 1, characterized in that The light source array comprises a control unit and a plurality of programmable light source units; The control unit is used to respond to the lighting configuration instruction and configure the lighting parameters of the multiple light source units according to the deployment positions of the multiple light source units, so that the multiple light source units provide consistent lighting information for the multi-camera array from multiple viewing angles.

5. The system according to any one of claims 1 to 4, characterized in that: The control device is also used for: Receive image sequences at multiple viewing angles sent by the multi-camera array; and perform cutout processing and / or three-dimensional reconstruction on the target object according to the image sequences at the multiple viewing angles.

6. The system according to any one of claims 1 to 4, characterized in that: Also includes: A calibration device, used for calibrating the internal and external parameters of each camera unit in the multi-camera array using a calibration plate that rotates at a set rotation angle; And color correction is performed on each camera unit in the multi-camera array using a standard light source and a standard color card.

7. A data collection method, characterized in that: include: Determine a target object, wherein a light source array and a multi-camera array are deployed around the target object, and the light source array and the multi-camera array meet the lighting consistency requirement at the deployment position; Controlling the light source array to provide consistent lighting information for the multi-camera array from multiple viewing angles with respect to the target object; Sending a hardware synchronization signal to the multi-camera array to control the multi-camera array to synchronously shoot the target object from multiple viewing angles according to the hardware synchronization signal and consistent lighting information provided by the light source array to obtain image sequences under the multiple viewing angles; According to the image sequences under the multiple viewing angles, the target object is subjected to a cutout process and / or a three-dimensional reconstruction.

8. The method according to claim 7, characterized in that Also includes: In response to the lighting configuration instruction, the lighting parameters of the plurality of programmable light source units in the light source array are configured according to the deployment positions of the plurality of light source units, so that the plurality of light source units provide consistent lighting information for the multi-camera array from multiple viewing angles.

9. The method according to claim 7, characterized in that: The image sequence at any viewing angle includes a plurality of original images captured at a plurality of capturing moments at any viewing angle; and performing three-dimensional reconstruction of the target object according to the image sequence at the plurality of viewing angles includes: Reconstructing the target object in three dimensions according to the original images corresponding to the same shooting time in the image sequences under the multiple viewing angles to obtain multiple initial mesh models of the target object corresponding to the multiple shooting times; According to the surface attachment information of the target object, mesh data adapted to the surface attachment information is extracted from the multiple initial mesh models to obtain multiple local mesh models; A plurality of target parameterized object models corresponding to the target object at the plurality of shooting moments are generated according to the plurality of local grid models and the basic parameterized object model.

10. The method according to claim 9, characterized in that According to the original images corresponding to the same shooting time in the image sequences under the multiple viewing angles, the target object is three-dimensionally reconstructed to obtain multiple initial mesh models corresponding to the target object at the multiple shooting times, including: According to the image sequences under the multiple viewing angles, the target object is subjected to a cutout process to obtain a foreground image sequence under the multiple viewing angles, wherein the foreground image sequence includes a plurality of foreground images corresponding to the plurality of shooting moments, and the foreground images include the target object under the corresponding viewing angles; The target object is three-dimensionally reconstructed according to the foreground images corresponding to the same shooting time in the foreground image sequence under the multiple viewing angles to obtain multiple initial mesh models of the target object corresponding to the multiple shooting times.

11. The method according to claim 10, characterized in that Generating a plurality of target parameterized object models corresponding to the target object at the plurality of shooting moments according to the plurality of local grid models and the basic parameterized object model, comprising: Determining, according to the foreground images corresponding to the same shooting moment in the foreground image sequence under the multiple viewing angles, multiple multi-dimensional parameterized representation information corresponding to the target object at the multiple shooting moments; Adjusting the basic parameterized object model according to the multiple multi-dimensional parameterized representation information to obtain multiple intermediate parameterized object models; The plurality of target parameterized object models are generated according to the plurality of intermediate parameterized object models and the plurality of local mesh models.

12. The method according to claim 11, characterized in that The target object is a human object, and according to the foreground images corresponding to the same shooting moment in the foreground image sequence under the multiple perspectives, multiple multi-dimensional parameterized representation information corresponding to the target object at the multiple shooting moments is determined, including: Determine at least two parameters of the face shape parameter, expression coefficient, head posture parameter, hand shape parameter, hand posture parameter, body shape parameter and body posture parameter corresponding to the human object at each shooting moment according to the foreground image corresponding to the same shooting moment in the foreground image sequence under the multiple viewing angles; At least two parameters of the facial parameters, expression coefficients, head posture parameters, hand shape parameters, hand posture parameters, body shape parameters and body posture parameters corresponding to the human object at each shooting moment are used as multi-dimensional parameterized representation information corresponding to the human object at each shooting moment.

13. The method according to any one of claims 7 to 12, characterized in that: The image sequence at any viewing angle includes a plurality of original images captured at a plurality of capturing moments at any viewing angle; performing a cutout process on the target object according to the image sequence at the plurality of viewing angles, including: For an image sequence at any viewing angle, a plurality of original images in the image sequence are subjected to a cutout process using a cutout model to obtain a plurality of initial cutout images; Performing morphological processing on the multiple initial cutout images to obtain multiple intermediate cutout images; Foreground detection is performed on the multiple intermediate state cutout images to obtain a foreground image sequence under any viewing angle, wherein the foreground image sequence includes multiple foreground images, and the foreground image includes the target object under any viewing angle.

14. The method according to claim 13, characterized in that The cutout model includes a backbone network, a semantic segmentation network and an edge extraction network; The multiple original images in the image sequence are subjected to a cutout process by using a cutout model to obtain a plurality of initial cutout images corresponding to the multiple shooting moments, including: For any original image, the backbone network is used to extract image features of multiple channels of the original image, and feature aggregation processing is performed on the image features of the multiple channels to obtain aggregated image features, and the image features of the multiple channels and the aggregated image features are fused to obtain fused image features; Inputting the fused image features into a semantic segmentation network for processing to obtain a semantic segmentation image; Inputting the image features of the multiple channels into an edge detection network for processing to obtain edge information of the target object, and generating a mask image according to the edge information of the target object; The semantic segmentation image and the mask image are fused to obtain an initial cutout image corresponding to the original image.

15. The method according to claim 13, characterized in that Before performing a cutout process on the target object according to the image sequences under the multiple viewing angles, the method further includes: For an image sequence of any viewing angle, color correction is performed on a plurality of original images in the image sequence using respective color correction matrices of a plurality of camera units in the multi-camera array; The color correction matrix between the multiple camera units represents that the color consistency between the multiple camera units is met.

16. A three-dimensional reconstruction method, characterized in that: include: Acquire an image sequence of the target object at multiple viewing angles, wherein the image sequence at any viewing angle includes multiple original images captured at multiple capturing moments at any viewing angle; Reconstructing the target object in three dimensions according to the original images corresponding to the same shooting time in the image sequences under the multiple viewing angles to obtain multiple initial mesh models of the target object corresponding to the multiple shooting times; According to the surface attachment information of the target object, mesh data adapted to the surface attachment information is extracted from the multiple initial mesh models to obtain multiple local mesh models; A plurality of target parameterized object models corresponding to the target object at the plurality of shooting moments are generated according to the plurality of local grid models and the basic parameterized object model.

17. A method for cutting out a picture, characterized in that: include: Acquire an image sequence of the target object at multiple viewing angles, wherein the image sequence at any viewing angle includes multiple original images captured at multiple capturing moments at any viewing angle; For an image sequence at any viewing angle, a plurality of original images in the image sequence are subjected to a cutout process using a cutout model to obtain a plurality of initial cutout images; Performing morphological processing on the multiple initial cutout images to obtain multiple intermediate cutout images; Foreground detection is performed on the multiple intermediate state cutout images to obtain a foreground image sequence under any viewing angle, wherein the foreground image sequence includes multiple foreground images, and the foreground image includes the target object under any viewing angle.

18. An electronic device, characterized in that: include: Memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to perform the steps in the method of any one of claims 7 to 15, claim 16 and claim 17.

19. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is enabled to implement the steps of the method according to any one of claims 7 to 15, claim 16 and claim 17.

20. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, enables the processor to implement the steps of the method as claimed in any one of claims 7 to 15, claim 16 and claim 17.

Images