Target recognition model training method, system and device based on depth map guidance and medium
Through the depth map-guided target recognition model training method, combined with UNet network and video timing characteristics, the problem of difficult extraction of portraits and objects in the existing technology is solved, and high-quality recognition and processing of videos and images is achieved. Improved recognition accuracy and stability.
Patent Information
- Application Number
- CN202310613194.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-29
- Publication Date
- 2024-11-29
AI Technical Summary
The existing cutout technology is difficult to accurately extract portraits and objects in images without relying on user guidance information, especially in video processing, and cannot meet the complete extraction needs of portrait plus objects.
Using the depth map-guided target recognition model training method, by acquiring RGB images and depth images, using preset distance parameters to generate a training set, combining UNet network structure and Skip-Connection, the model is alternately trained to identify human bodies and interactive objects, and through Video timing characteristics and loss function optimization, ultimately generating high-quality and high-resolution images.
Real-time processing of videos and images can be realized, and the human body and interactive objects can be fully and effectively recognized, and the recognition accuracy and stability can be improved. It meets the needs of people and objects, and improves the robustness and refinement of the cutout algorithm. Prospect prediction ability.
Smart Images

Figure CN119048790A_ABST
Abstract
Description
Background Art
[0002] Image matting (image background removal) refers to extracting the foreground object of interest from an image or video (image sequence) while filtering out the background. An image can be simply considered to consist of two parts, namely the foreground and the background. In short, image matting is to distinguish the foreground from the background of a given image. It is one of the key technologies in many image editing. The research on this technology has a history of more than 20 years. The mathematical definition of image background removal was first proposed by foreign scholars Porter and Duff in 1984. They first introduced the concept of transparent (alpha) channel to control the linear interpolation ratio of the foreground and background colors when the foreground and background are merged. In the end, the image matting problem is defined as the task of estimating the alpha value (foreground-background color ratio) of each image pixel. In this task, the input is the original image, and the output will be the alpha value of each pixel. Since the known information is the color information of each pixel, the unknown information is alpha and the foreground and background colors, and the number of unknown variables is greater than the number of known variables, it is a difficult problem with insufficient constraints in mathematics. Most matting methods rely on the user to provide guidance information or prior assumptions to constrain the problem and obtain good estimates of unknown variables. The use of guidance information and prior assumptions is destined to have weak generalization capabilities of the model, and the prediction accuracy will be very poor under certain conditions. Therefore, providing high-confidence prior information in the matting process can significantly improve its effect. A more classic method is an interactive matting technology called Bayesian-based matting, that is, humans participate in this process and manually mark the foreground and background to achieve foreground extraction. In the present invention, there is no need to rely on other information, and the algorithm can independently judge and achieve the separation of foreground and background.
[0003] In addition, currently common cutout algorithms, such as those provided by Tencent Conference and Zoom, can only meet the needs of portrait cutouts, that is, extracting the foreground of the portrait, while other foregrounds associated with the portrait cannot be completely extracted, such as handheld objects, attached tables, computers, etc. For users who have needs such as live broadcasts and displaying items in meetings, the existing methods cannot meet expectations. However, the present invention can extract the foreground of the portrait or the portrait plus the object based on the depth map according to user needs on the basis of completely extracting the foreground of the portrait. The depth image is also called a range image, which refers to an image that uses the distance (depth) from the image collector to each point in the scene as the pixel value, which directly reflects the geometry of the visible surface of the scene.
[0004] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention
[0005] To this end, the present invention uses a preset distance to obtain a first target range, generates a first training set consisting of images and a second training set consisting of videos, and alternately uses the first training set and the second training set to train the initial model, thereby obtaining a final target recognition model that has good recognition effects on both videos and images, can completely and effectively identify the human body and interactive objects, and generate high-quality and high-resolution images, and can realize real-time processing of videos.
[0006] In a first aspect, the present invention provides a method for training a target recognition model based on depth map guidance, characterized in that it comprises the following steps:
[0007] Step S1: Acquire a first image in a video; wherein the first image includes an RGB image and a depth image;
[0008] Step S2: on the depth image, using a preset distance parameter, obtaining a first target range, thereby obtaining a first training set and a second training set; wherein the first training set is composed of a first image containing only the first target range; and the second training set is composed of a first video composed of the first image;
[0009] Step S3: using the first training set to train the initial model;
[0010] Step S4: using the second training set to train the initial model;
[0011] Step S5: Repeat steps S3-S4 to obtain the final target recognition model;
[0012] The initial model includes the following steps during training:
[0013] Step S31: Based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first image, the first target range is identified to obtain the second target range;
[0014] Step S32: using the preset distance parameter to adjust the second target range to obtain a third target range.
[0015] Optionally, the target recognition model training method based on depth map guidance is characterized in that step S2 comprises:
[0016] Step S21: filtering the depth image to obtain a second depth image;
[0017] Step S22: extracting a first target range from the second depth image according to a preset distance parameter;
[0018] Step S23: performing an erosion operation on the first target range, and marking the eroded area as 1, the eroded area as 0.5, and other areas as 0;
[0019] Step S24: the images corresponding to the first target are classified into a first training set, and the images corresponding to the first target are integrated into a video in chronological order and classified into a second training set.
[0020] Optionally, the target recognition model training method guided by a depth map is characterized in that the corrosion operation is determined according to the pixel offset between the second depth map and the RGB image.
[0021] Optionally, the target recognition model training method based on depth map guidance is characterized in that step S31 comprises:
[0022] Step S311: using an encoder to perform feature extraction and semantic information concentration on the first target range;
[0023] Step S312: using a decoder to restore image pixels;
[0024] Step S313: using Skip-Connection with different dimensional features to supplement information;
[0025] Step S314: iterating through the video timing correlation loss function to obtain a second target range.
[0026] Optionally, the depth map-guided target recognition model training method is characterized in that, in step S312, the decoder includes a GRU for inter-frame information transmission.
[0027] Optionally, in the depth map-guided target recognition model training method, the initial model also alternately trains a third training set and a fourth training set; wherein the third training set is composed of a second image including a portrait; and the fourth training set is a video composed of the second image.
[0028] Optionally, the depth map-guided target recognition model training method is characterized in that the initial model also trains the changes of hair over time to obtain the change pattern of hair shape.
[0029] In a second aspect, the present invention provides a target recognition model training system based on depth map guidance, which is used to implement any of the above-mentioned target recognition model training methods based on depth map guidance, and is characterized by comprising:
[0030] An acquisition module, used to acquire a first image in a video; wherein the first image includes an RGB image and a depth image;
[0031] A pre-selection module, used to obtain a first target range on the depth image using a preset distance parameter, thereby obtaining a first training set and a second training set; wherein the first training set is composed of a first image containing only the first target range; and the second training set is composed of a first video composed of the first image;
[0032] An image training module, used for training an initial model using the first training set;
[0033] A video training module, used for training the initial model using the second training set;
[0034] A convergence module, used for alternately running the image training module and the video training module to obtain a final object recognition model;
[0035] Wherein, the initial model includes:
[0036] A second target unit is used to identify the first target range based on the Skip-Connection of the UNet network structure and different dimensional features and the temporal features of the first image to obtain a second target range;
[0037] The third target unit is used to adjust the second target range by using the preset distance parameter to obtain a third target range.
[0038] In a third aspect, the present invention provides a target recognition model training device based on depth map guidance, characterized in that it includes:
[0039] processor;
[0040] a memory storing executable instructions of the processor;
[0041] Wherein, the processor is configured to execute the steps of any one of the above-mentioned depth map-guided target recognition model training methods by executing the executable instructions.
[0042] In a fourth aspect, the present invention provides a computer-readable storage medium for storing a program, characterized in that when the program is executed, the steps of any one of the aforementioned depth map-guided target recognition model training methods are implemented.
[0043] Compared with the prior art, the present invention has the following beneficial effects:
[0044] The present invention pre-processes the first image with the depth value as a guide, preliminarily locates the target area, reduces the amount of data for initial model training, improves learning efficiency, and is conducive to improving recognition accuracy.
[0045] The model trained by the present invention improves the stability of video cutout through the temporal characteristics of the image, and integrates the human segmentation function. Through the multi-task training strategy, the cutout algorithm can take into account the integrity and robustness of the human foreground while achieving refined foreground prediction. It can realize real-time foreground cutout based on RGB images and depth images, and meet the needs of cutout of people and objects at the same time.
[0046] The present invention adopts image and video alternating training model, which can make the model have high recognition quality for both images and videos, and the recognition effect is better than the effect of simple image or video training. In particular, when the trained model is used for real-time processing of video, it has a much better effect than the prior art. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:
[0048] Figure 1 This is a flowchart of the steps of a method for training a target recognition model guided by a depth map in an embodiment of the present invention;
[0049] Figure 2 This is a flow chart of steps for obtaining a first target range in an embodiment of the present invention;
[0050] Figure 3 This is a schematic diagram of a target range in an embodiment of the present invention;
[0051] Figure 4 A flowchart of steps for obtaining a second target range in an embodiment of the present invention;
[0052] Figure 5 This is another target range schematic diagram in an embodiment of the present invention;
[0053] Figure 6A schematic diagram of a process of adjusting a hair area in an embodiment of the present invention;
[0054] Figure 7 Schematic diagram of the structure of a target recognition model training system based on depth map guidance in an embodiment of the present invention;
[0055] Figure 8 is a schematic diagram of the structure of a target recognition model training device based on depth map guidance in an embodiment of the present invention; and
[0056] Fig. 9 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION
[0057] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.
[0058] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0059] An embodiment of the present invention provides a depth map-guided target recognition model training method, which aims to solve the problems existing in the prior art.
[0060] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.
[0061] The present invention uses a preset distance to obtain a first target range, generates a first training set consisting of images and a second training set consisting of videos, and alternately uses the first training set and the second training set to train the initial model, thereby obtaining a final target recognition model that has good recognition effects on both videos and images, can completely and effectively identify the human body and interactive objects, and generate high-quality and high-resolution images, and can realize real-time processing of videos.
[0062] Figure 1 FIG. 1 is a flowchart of a method for training a target recognition model based on depth map guidance in an embodiment of the present invention. Figure 1 As shown, the steps of a target recognition model training method based on depth map guidance in an embodiment of the present invention include:
[0063] Step S1: Acquire the first image in the video.
[0064] In this step, the first image includes an RGB image and a depth image. In some application scenarios, the RGB image and the depth image will be described as RGBD images, which have the same function as in this embodiment. The RGB image and the depth image in this step are aligned at the pixel level, and they are described separately to more clearly illustrate the processing of the image data. The image is acquired by a depth camera. Subsequent steps are performed on the mask to obtain the final target area range. Figure 3 a is an RGB image, Figure 3 b is the depth image.
[0065] Step S2: On the depth image, using a preset distance parameter, a first target range is obtained, thereby obtaining a first training set and a second training set.
[0066] In this step, the preset distance parameter is a constant used to extract the effective area. For example, if the preset distance parameter is 1.2m, all pixels whose depth values are less than 1.2m on the depth image constitute the first target range. For the same video, the preset distance parameter is a fixed value. For different videos, the preset distance parameter can be different values. The setting of the preset distance can be selected according to the depth camera and the specific application scenario. The first training set consists of a first image containing only the first target range; the second training set consists of a first video consisting of the first image. The processed first image clarifies the first target range and can be used to train the model. The first training set consists of independent first images and is used to train the model on a single image. The second training set consists of multiple videos and is used to train the model on videos. The videos in the second training set consist of the first images in the first training set. The frames of each video in the second training set exist independently in the first training set.
[0067] Step S3: using the first training set to train the initial model.
[0068] In this step, the initial model is a Skip-Connection based on the UNet network structure and features of different dimensions, and has the temporal feature parameters of the first image. The UNet structure used in the initial model includes three components: an encoder, a decoder, and a Skip-Connection on the same layer. The encoder performs downsampling. For example, the input image size is 572*572, the feature map size after two rounds of 3*3 convolution is 568*568, and the output size after 22 maximum pooling is 284*284. The decoder performs upsampling. For example, the decoder is composed of 4 groups of 2*2 transposed convolutions, 3*3 convolutions and a ReLU activation function, and a 1*1 convolution is added to the final output layer. Finally, the Skip-Connection on the same layer is to crop the output of each layer during downsampling and connect it to the upsampling layer on the same layer for fusion. In this step, the initial model is trained using an image so that the initial model converges on the recognition of a single image.
[0069] Step S4: using the second training set to train the initial model.
[0070] In this step, unlike step S3, since the initial model has temporal feature parameters, the initial model can be trained using videos. Any frame of any video in the second training set may not be in the first training set, and the image in the first training set may not be in any video in the second training set, that is, the number of images in the first training set may be different from the number of images in all videos in the second training set.
[0071] Step S5: Repeat steps S3-S4 to obtain the final target recognition model.
[0072] In this step, steps S3-S4 are repeated, and the model is repeatedly trained using the first training set and the second training set until the parameter change is less than the preset value after a certain training, thereby obtaining the final target recognition model.
[0073] The initial model includes the following steps during training:
[0074] Step S31: Based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first image, the first target range is identified to obtain the second target range.
[0075] In this step, for a single image, the temporal feature is set to a constant, such as 0; for a video, each frame is numbered in chronological order, and the frequency or time interval parameter is input into the initial model. This step processes the image in combination with the temporal features of the image to ensure the stability of the target recognition between different frames. To some extent, the temporal features can also be regarded as a kind of deep supervision.
[0076] Step S32: using the preset distance parameter to adjust the second target range to obtain a third target range.
[0077] In this step, different from the screening performed in step S2, this step resets the area within part of the depth value range to increase the selected area and obtain a third target range. The third target range is larger than the second target range. When determining the area to be reset, a fixed depth value range or a depth value range related to the target area can be used. When using a fixed depth value range, the depth value within 1m can be reset to obtain the third target range. When using a depth value range related to the target area, if the average depth value of the target area is 0.8m, the depth value of 0.6m-1m is reset to obtain the third target range.
[0078] Figure 2 FIG. 1 is a flow chart of steps for obtaining a first target range in an embodiment of the present invention. Figure 2 As shown, in an embodiment of the present invention, a step of obtaining a first target range includes:
[0079] Step S21: filtering the depth image to obtain a second depth image.
[0080] In this step, the filtering process includes flicker removal, flying spot removal, missing area filling and other processes. Figure 3 b is the TOF depth map obtained by the 3D camera. As can be seen from the figure, there are flying spots and missing areas in the depth map. The differences between different frame images will cause flickering and other phenomena. This step can improve the quality of the depth image and ensure the integrity of the image content by processing the single frame image and the previous image, so that the extraction of the foreground is more complete.
[0081] Step S22: extracting a first target range from the second depth image according to a preset distance parameter.
[0082] In this step, the preset distance parameter is a constant. By comparing the depth value of the pixel point with the preset distance parameter, all the pixels whose depth values are less than the preset distance parameter can be obtained, that is, the first target range. The first target range is the set of all the pixels whose depth values are less than the preset distance parameter.
[0083] Step S23: performing an erosion operation on the first target range, and marking the eroded area as 1, the eroded area as 0.5, and the other areas as 0.
[0084] In this step, the image is marked in three categories according to the first target range. The first target range is eroded to obtain the core area, which is marked as 1, and has a high credibility of foreground. The eroded part of the first target range is marked as 0.5, and has a low credibility of foreground. The other areas are marked as 0, which are background. The part marked as 0.5 may be either foreground or background, and needs to be judged in subsequent steps.
[0085] In some embodiments, the erosion operation is determined according to the pixel offset between the second depth map and the RGB image. Different from the erosion operation using a convolutional template for a single RGB image in the prior art, this embodiment uses the offset between the depth map and the RGB map to perform the erosion operation, and uses the depth change to further confirm the confidence of the target object, which can greatly improve the scope and accuracy of the erosion, which is of great significance for obtaining a determined target object and can greatly improve the recognition effect.
[0086] In some embodiments, the area filtered in step S21 is also marked as 0.5. The area processed in step S21 is not the original data, but a low confidence area, which is also marked as 0.5. A more accurate result can be obtained by performing a one-step judgment on it in subsequent steps.
[0087] Step S24: the images corresponding to the first target are classified into a first training set, and the images corresponding to the first target are integrated into a video in chronological order and classified into a second training set.
[0088] In this step, individual images are classified into the first training set, and multiple images with temporal correlation are classified into the second training set as a video as a whole. Although an image may exist in both the first training set and the second training set, their existence forms and functions are different. In the first training set, the image exists as an independent image, and the impact on the initial model mainly lies in the parameters of the image itself. In the second training set, the image exists as a frame of the video, which not only affects the parameters of the image itself in the initial model, but also adjusts the timing-related parameters of the initial model together with other images.
[0089] In some embodiments, after processing the video in step S1 , the video is directly included in the second training set, and then all frames of the video are separately included in the first training set, so as to generate the first training set and the second training set as quickly as possible.
[0090] This embodiment performs filtering processing on the image, then extracts the first target range according to the preset distance parameter, and then marks the image area as 1, 0.5 and 0 according to the corrosion operation, and directly uses the depth value to determine part of the foreground and background, thereby reducing the amount of calculation of the subsequent algorithm, improving efficiency, and avoiding the determined foreground or background being misidentified by the algorithm, thereby ensuring quality.
[0091] Figure 4 FIG. 1 is a flow chart of steps for obtaining a second target range in an embodiment of the present invention. Figure 4 As shown, in an embodiment of the present invention, a step of obtaining a second target range includes:
[0092] Step S311: using an encoder to perform feature extraction and semantic information concentration on the first target range.
[0093] In this step, the encoder is a downsampling module consisting of two 3x3 convolutional layers (RELU) plus a 2x2 maxpooling layer. After multiple downsampling, the low-resolution information obtains the target features and can provide the contextual semantic information of the segmented target in the entire image.
[0094] Step S312: Use a decoder to restore image pixels.
[0095] In this step, the decoder consists of an upsampling convolutional layer (deconvolutional layer), a feature concatenation layer, and two 3x3 convolutional layers (ReLU).
[0096] Step S313: Use Skip-Connection with different dimensional features to supplement information.
[0097] In this step, the U-Net with skipconnection structure is added, which enables the network to fuse the feature maps of the corresponding position of the encoder on the channel during each level of upsampling. By fusing the underlying features with the high-level features, the network can retain more high-resolution detail information contained in the high-level feature maps, thereby improving the image segmentation accuracy. While the cross-layer feature reconstruction enhances the transmission of feature information between layers, it further utilizes the rich detail information in the high-level convolutional feature layer, thereby maximizing the utilization of feature information in each layer of the network. This step introduces the feature information at the corresponding scale into the upsampling or deconvolution process, providing multi-scale and multi-level information for the later image segmentation, thereby obtaining a more refined segmentation effect.
[0098] Step S314: iterating through the video timing correlation loss function to obtain a second target range.
[0099] In this step, the information of the previously processed images is compared with the image, and the loss function is iteratively optimized to obtain the final second target range.
[0100] In some embodiments, the decoder includes a GRU for inter-frame information transmission. The input to the GRU is a three-dimensional variable, and the conv layer input variable changes, which can be well used for inter-frame information transmission and greatly reduces defects and improves the perceived quality by predicting more coherent results. At the same time, the GRU can also improve the robustness of extinction and better guess the boundaries when individual frames may be blurred.
[0101] This embodiment uses UNet to process the first target range, and combines it with Skip-Connection for supplementation, and uses the video timing-related loss function for iterative optimization, so that the final second target range can obtain a high-accuracy and high-precision image in a single frame, and can ensure the continuity of the target object in the video, and prevent problems such as video jitter.
[0102] In some embodiments, the initial model is further trained alternately on a third training set and a fourth training set; wherein the third training set is composed of second images including portraits; and the fourth training set is a video composed of the second images. Figure 5 As shown, compared with the previous embodiment, the second image in this embodiment includes a portrait. The processing of the image itself is the same as that in the previous embodiment, and will not be repeated here. Compared with the previous embodiment, the initial model in this embodiment also includes a hair detection module, and the initial model is trained by the third training set and the fourth training set, so that the initial model can detect the hair area in the image.
[0103] In some embodiments, the initial model is also trained on the change of hair over time to obtain the change law of hair shape. This embodiment obtains the change law of different types of hair over time through the time-series change of hair shape, so that when performing subsequent operations such as background replacement, the hair can be changed according to the background type, so that the state of the hair adapts to the background, and does not have to be the same as the hair state in the first image. Figure 6 As shown, the hair area is divided into a fixed area and a moving area according to the hairstyle, and then the position of the moving area is determined, so that the curvature of the moving area can be adjusted according to the moving area, the type of the background and the temporal characteristics of the image.
[0104] Figure 7 FIG. 1 is a schematic diagram of a target recognition model training system based on depth map guidance in an embodiment of the present invention. Figure 7 As shown, a target recognition model training system based on depth map guidance in an embodiment of the present invention includes:
[0105] An acquisition module, used to acquire a first image in a video; wherein the first image includes an RGB image and a depth image;
[0106] A pre-selection module, used to obtain a first target range on the depth image using a preset distance parameter, thereby obtaining a first training set and a second training set; wherein the first training set is composed of a first image containing only the first target range; and the second training set is composed of a first video composed of the first image;
[0107] An image training module, used for training an initial model using the first training set;
[0108] A video training module, used for training the initial model using the second training set;
[0109] A convergence module, used for alternately running the image training module and the video training module to obtain a final object recognition model;
[0110] Wherein, the initial model includes:
[0111] A second target unit is used to identify the first target range based on the Skip-Connection of the UNet network structure and different dimensional features and the temporal features of the first image to obtain a second target range;
[0112] The third target unit is used to adjust the second target range by using the preset distance parameter to obtain a third target range.
[0113] Specifically, the acquisition module obtains a first image with depth information, and retains single image and video information at the same time. The pre-selection module uses the depth information to process the image and video to obtain the first target range, greatly reducing the range to be processed. The image training module uses the first training set to train the initial model, so that the training of the image is more accurate. The video training module uses the second training set to train the initial model, so that the training of the video is more accurate. The convergence module controls the image training module and the video training module to alternately train the initial model, so that the final target recognition model has good recognition effect stability for both single images and videos. The second target unit uses the UNet-based network structure to perform target recognition within the first target range while ensuring different dimensional features and inter-frame information to obtain the second target range. The third target unit then uses the depth information to fine-tune the second target range to ensure the display of the core foreground and the hiding of the background.
[0114] This embodiment uses a preset distance to obtain a first target range, generates a first training set consisting of images and a second training set consisting of videos, and alternately uses the first training set and the second training set to train the initial model to obtain a final target recognition model that has good recognition effects on both videos and images. It can completely and effectively identify the human body and interactive objects, generate high-quality and high-resolution images, and realize real-time processing of videos.
[0115] The present invention also provides a depth map guided target recognition model training device, including a processor and a memory, wherein executable instructions of the processor are stored. The processor is configured to execute the steps of a depth map guided target recognition model training method by executing the executable instructions.
[0116] As described above, this embodiment uses a preset distance to obtain a first target range, generates a first training set consisting of images and a second training set consisting of videos, and alternately uses the first training set and the second training set to train the initial model, thereby obtaining a final target recognition model that has good recognition effects on both videos and images, can completely and effectively identify the human body and interactive objects, and generate high-quality and high-resolution images, and can realize real-time processing of videos.
[0117] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".
[0118] Figure 8 Schematic diagram of the structure of a target recognition model training device based on depth map guidance in an embodiment of the present invention. Figure 8 The electronic device 600 according to this embodiment of the present invention is described. Figure 8 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0119] like Figure 8 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.
[0120] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various exemplary embodiments of the present invention described in the above-mentioned method for training a target recognition model guided by a depth map. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .
[0121] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .
[0122] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0123] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0124] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although Figure 8 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.
[0125] In an embodiment of the present invention, a computer-readable storage medium is also provided for storing a program, and when the program is executed, the steps of a method for training a target recognition model guided by a depth map are implemented. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above-mentioned method for training a target recognition model guided by a depth map section of this specification.
[0126] As shown above, this embodiment uses a preset distance to obtain a first target range, generates a first training set consisting of images and a second training set consisting of videos, and alternately uses the first training set and the second training set to train the initial model to obtain a final target recognition model that has good recognition effects on both videos and images. It can completely and effectively identify the human body and interactive objects, generate high-quality and high-resolution images, and realize real-time processing of videos.
[0127] Fig. 9 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Fig. 9 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.
[0128] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0129] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.
[0130] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).
[0131] This embodiment uses a preset distance to obtain a first target range, generates a first training set consisting of images and a second training set consisting of videos, and alternately uses the first training set and the second training set to train the initial model to obtain a final target recognition model that has good recognition effects on both videos and images. It can completely and effectively identify the human body and interactive objects, generate high-quality and high-resolution images, and realize real-time processing of videos.
[0132] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.
[0133] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.
Claims
1. A method for training a target recognition model based on depth map guidance. It is characterized in that The steps include: Step S1: Acquire a first image in a video; wherein the first image includes an RGB image and a depth image; Step S2: on the depth image, using a preset distance parameter, obtaining a first target range, thereby obtaining a first training set and a second training set; wherein the first training set is composed of a first image containing only the first target range; and the second training set is composed of a first video composed of the first image; Step S3: using the first training set to train the initial model; Step S4: using the second training set to train the initial model; Step S5: Repeat steps S3-S4 to obtain the final target recognition model; The initial model includes the following steps during training: Step S31: Based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first image, the first target range is identified to obtain the second target range; Step S32: using the preset distance parameter to adjust the second target range to obtain a third target range.
2. According to claim 1, a method for training a target recognition model based on depth map guidance, It is characterized in that Step S2 includes: Step S21: filtering the depth image to obtain a second depth image; Step S22: extracting a first target range from the second depth image according to a preset distance parameter; Step S23: performing an erosion operation on the first target range, and marking the eroded area as 1, the eroded area as 0.5, and other areas as 0; Step S24: the images corresponding to the first target are classified into a first training set, and the images corresponding to the first target are integrated into a video in chronological order and classified into a second training set.
3. According to claim 2, a method for training a target recognition model based on depth map guidance, It is characterized in that The erosion operation is determined according to a pixel offset between the second depth map and the RGB image.
4. According to the method for training a target recognition model based on depth map guidance according to claim 1, It is characterized in that Step S31 includes: Step S311: using an encoder to perform feature extraction and semantic information concentration on the first target range; Step S312: using a decoder to restore image pixels; Step S313: using Skip-Connection with different dimensional features to supplement information; Step S314: Iterate the video timing-related loss function to obtain a second target range.
5. According to claim 4, a method for training a target recognition model based on depth map guidance, It is characterized in that In step S312, the decoder includes a GRU for inter-frame information transmission.
6. According to the depth map guided target recognition model training method of claim 1, the initial model is further trained alternately on the third training set and the fourth training set; in, The third training set is composed of second images including portraits; and the fourth training set is videos composed of the second images.
7. The method for training a target recognition model based on depth map guidance according to claim 6, It is characterized in that The initial model is also trained on the changes of hair over time to obtain the change rules of hair shape.
8. A depth map-guided target recognition model training system, used to implement the depth map-guided target recognition model training method according to any one of claims 1 to 7, It is characterized in that include: An acquisition module, used to acquire a first image in a video; wherein the first image includes an RGB image and a depth image; A pre-selection module, used to obtain a first target range on the depth image using a preset distance parameter, thereby obtaining a first training set and a second training set; wherein the first training set is composed of a first image containing only the first target range; and the second training set is composed of a first video composed of the first image; An image training module, used for training an initial model using the first training set; A video training module, used for training the initial model using the second training set; A convergence module, used for alternately running the image training module and the video training module to obtain a final object recognition model; Wherein, the initial model includes: A second target unit is used to identify the first target range based on the Skip-Connection of the UNet network structure and different dimensional features and the temporal features of the first image to obtain a second target range; The third target unit is used to adjust the second target range by using the preset distance parameter to obtain a third target range.
9. A target recognition model training device based on depth map guidance, It is characterized in that include: processor; a memory storing executable instructions of the processor; Wherein, the processor is configured to execute the steps of the depth map guided target recognition model training method described in any one of claims 1 to 7 by executing the executable instructions.
10. A computer-readable storage medium for storing a program, It is characterized in that When the program is executed, the steps of the depth map-guided target recognition model training method described in any one of claims 1 to 7 are implemented.
