Target region determination method and apparatus, device, and storage medium

Abstract

This application relates to a method, apparatus, device, and storage medium for determining a target region, and pertains to the field of image processing technology. The method includes: acquiring a first image; obtaining first estimated depth data based on the first color image data; obtaining first calibrated depth data based on the first estimated depth data and the first depth image data; performing weighted processing on the first color image data and the first calibrated depth data to obtain a first fused feature map; and determining a target region corresponding to the first image based on the first fused feature map. This scheme improves the accuracy of target region determination by estimating depth information in the first image through a color image, correcting the corresponding depth image, and then obtaining the target region based on the corrected depth image data and color image data.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a method, apparatus, device, and storage medium for determining a target region. Background Technology

[0002] Salience target detection is a series of related techniques that process image data to detect the most important target region in the image and segment the target region to improve the recognition accuracy of the image.

[0003] Among related technologies, salient object detection based on RGB images has good detection performance, but its performance is still limited when the object has a similar appearance to its surroundings or when the background scene is severely cluttered. As a compensation, depth data is added to RGB-D salient object detection, because the rich spatial structure and 3D layout information embedded in the depth map greatly improves the model's performance in challenging scenarios.

[0004] In the above-mentioned technical solutions, the quality of the depth map is often low, and it may contain a lot of noise and misleading information, resulting in poor target detection performance based on RGB-D images. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for determining a target region, which can improve the accuracy of target region determination. The technical solution is as follows:

[0006] On the one hand, a method for determining a target region is provided, the method comprising:

[0007] Acquire a first image; the first image contains first color image data and first depth image data;

[0008] Based on the first color image data, first estimated depth data is obtained; the first estimated depth data is used to indicate the depth information corresponding to the first color image data.

[0009] Based on the first estimated depth data and the first depth image data, the first calibrated depth data is obtained;

[0010] Based on the first color image data and the first calibration depth data, a weighted processing is performed to obtain the first fused feature map;

[0011] Based on the first fused feature map, the target region corresponding to the first image is determined.

[0012] On another front, a method for determining a target region is provided, the method comprising:

[0013] Acquire a third sample image; the third sample image includes third sample color image data and third sample depth image data;

[0014] Based on the third sample color image data, third sample estimated depth data is obtained; the third sample estimated depth data is used to indicate the depth information corresponding to the third sample color image data.

[0015] Based on the estimated depth data of the third sample and the depth image data of the third sample, the calibrated depth data of the third sample is obtained;

[0016] Based on the third sample color image data and the third sample calibration depth data, the feature fusion branch in the target region determination model is weighted and processed to obtain the third sample fusion feature map.

[0017] Based on the third sample fusion feature map and the target region corresponding to the third sample image, the target region determination model is trained;

[0018] The trained target region determination model is used to process the color image data and calibration depth data corresponding to the first image to obtain the target region corresponding to the first image.

[0019] Furthermore, a target area determination device is provided, the device comprising:

[0020] A first image acquisition module is used to acquire a first image; the first image includes first color image data and first depth image data.

[0021] The depth estimation module is used to obtain first estimated depth data based on the first color image data; the first estimated depth data is used to indicate the depth information corresponding to the first color image data.

[0022] The calibration depth acquisition module is used to obtain first calibration depth data based on the first estimated depth data and the first depth image data;

[0023] The fusion feature acquisition module is used to perform weighted processing based on the first color image data and the first calibration depth data to obtain a first fusion feature map;

[0024] The target region determination module is used to determine the target region corresponding to the first image based on the first fused feature map.

[0025] In one possible implementation, the device further includes:

[0026] The confidence level acquisition module is used to acquire the confidence level corresponding to the first depth image data based on the first depth image data; the confidence level corresponding to the first depth image data is used to indicate the accuracy of the image data corresponding to the target region in the first depth image data;

[0027] The calibration depth acquisition module is also used for,

[0028] The first estimated depth data and the first depth image data are weighted based on the confidence level corresponding to the first depth image data to obtain the first calibrated depth data.

[0029] In one possible implementation, the confidence level acquisition module is further configured to:

[0030] Based on the first depth image data, a confidence discrimination model is used to process the data to obtain the confidence level corresponding to the first depth image data.

[0031] The depth estimation module is also used for,

[0032] Based on the first color image data, data processing is performed using a depth estimation model to obtain the first estimated depth data;

[0033] The confidence discrimination model is a machine learning model trained using the first sample image as a sample and the confidence type corresponding to the first sample image as a label.

[0034] The depth estimation model is a machine learning model trained using the second sample image as a sample and the depth image data corresponding to the second sample image as a label; the second sample image is a sample image whose confidence level meets the first specified condition.

[0035] In one possible implementation, the device further includes:

[0036] The first sample set acquisition module is used to acquire a first training sample set; the first training sample set includes a first sample image and the confidence type corresponding to the first sample image.

[0037] The first confidence probability acquisition module is used to process data based on the first sample image using the confidence discrimination model to obtain the confidence probability corresponding to the first sample image; the confidence probability is used to indicate the probability that the first sample image is a positive sample;

[0038] The confidence discrimination model training module is used to train the confidence discrimination model based on the confidence probability and the confidence type corresponding to the first sample image.

[0039] In one possible implementation, the first sample set acquisition module includes:

[0040] The second sample set acquisition submodule is used to acquire a second training sample set; the second training sample set includes training sample images and target regions corresponding to the training sample images; the training sample images include training color sample data and training depth sample data.

[0041] The color prediction region acquisition submodule is used to determine the color image processing branch in the model through the target region, process the training color sample data, and obtain the prediction region corresponding to the training color sample data.

[0042] The depth prediction region acquisition submodule is used to determine the depth image processing branch in the model through the target region, process the training depth sample data, and obtain the prediction region corresponding to the training depth sample data.

[0043] The confidence score acquisition submodule is used to determine the confidence score of the training sample image based on the prediction region corresponding to the training color sample data, the prediction region corresponding to the training depth sample data, and the target region corresponding to the training sample image.

[0044] The first sample image determination submodule is used to determine the confidence type of the training sample image and determine the training sample image as the first sample image in response to the confidence score of the training sample image meeting a specified condition.

[0045] The color image processing branch in the target region determination model is a machine learning model pre-trained using sample color images as samples and the target regions corresponding to the sample color images as annotations.

[0046] The depth image processing branch in the target region determination model is a machine learning model pre-trained using sample depth images as samples and the target regions corresponding to the sample depth images as annotations.

[0047] In one possible implementation, the confidence score includes a color confidence score and a depth confidence score;

[0048] The confidence score acquisition submodule includes:

[0049] The color confidence score acquisition unit is used to determine the color confidence score corresponding to the training sample image based on the overlap between the prediction region corresponding to the training color sample data and the target region corresponding to the training sample image.

[0050] The depth confidence score acquisition unit is used to determine the depth confidence score corresponding to the training sample image based on the degree of overlap between the predicted region corresponding to the training depth sample data and the target region corresponding to the training sample image.

[0051] In one possible implementation, the fusion feature acquisition module includes:

[0052] The attention weighting submodule is used to perform attention-based weighting processing on the feature fusion branch in the target region determination model based on the first color image data and the first calibration depth data to obtain the first fused feature map.

[0053] The device further includes:

[0054] The color feature map acquisition module is used to process data based on the first color image data and through the depth image processing branch in the target region determination model to obtain the first color feature map.

[0055] The depth feature map acquisition module is used to process data based on the first depth image data by determining the depth image processing branch in the target region model to obtain the first depth feature map.

[0056] The target region determination module is also used for,

[0057] Based on the first fused feature map, the first depth feature map, and the first color feature map, the target region corresponding to the first image is determined.

[0058] The target region determination model is a machine learning model trained using a third sample image as a sample and the target region corresponding to the third sample image as an annotation.

[0059] In one possible implementation, the feature fusion branch includes a first pooling layer, a second pooling layer, a first fully connected layer, and a second fully connected layer;

[0060] The fusion feature acquisition module includes:

[0061] The first pooling submodule is used to perform global pooling through the first pooling layer based on the first color image data to obtain the first color pooling data.

[0062] The first fully connected submodule is used to process the first color pooling data through the first fully connected layer to obtain the first color vector.

[0063] The second pooling submodule is used to perform global pooling through the second pooling layer based on the first depth image data to obtain the first depth pooled data.

[0064] The second fully connected submodule is used to process the data through the second fully connected layer based on the first deep pooling data to obtain the first depth vector.

[0065] The fusion feature acquisition submodule is used to obtain the first fusion feature map by performing channel attention weighting processing on the first color image data and the first calibration depth data through the first color vector and the first depth vector; the first color vector is used to indicate the weight corresponding to the first color image data; the first depth vector is used to indicate the weight corresponding to the first depth image data.

[0066] In one possible implementation, the device further includes:

[0067] The third image acquisition module is used to acquire a third sample image; the third sample image includes third sample color image data and third sample depth image data.

[0068] The third estimation data acquisition module is used to obtain the third sample estimated depth data based on the third sample color image data;

[0069] The third calibration data acquisition module is used to obtain third sample calibration depth data based on the third sample estimated depth data and the third sample depth image data.

[0070] The third fusion feature acquisition module is used to perform weighted processing on the feature fusion branch in the target region determination model based on the third sample color image data and the third sample calibration depth data to obtain the third sample fusion feature map.

[0071] The region determination model training module is used to train the target region determination model based on the third sample fusion feature map and the target region corresponding to the third sample image.

[0072] In one possible implementation, the device further includes:

[0073] The third color feature acquisition module is used to process data based on the third sample color image data and through the color image processing branch in the target region determination model to obtain the third sample color feature map.

[0074] The third depth feature acquisition module is used to process data based on the third sample depth image data and through the depth image processing branch in the target region determination model to obtain the third sample color feature map.

[0075] The region determination model training module is also used for,

[0076] The target region determination model is trained based on the third sample color feature map, the third sample depth feature map, the third sample fusion feature map, and the target region corresponding to the third sample image.

[0077] In another aspect, a target area determination device is provided, the device comprising:

[0078] The third sample image acquisition module is used to acquire a third sample image; the third sample image includes third sample color image data and third sample depth image data.

[0079] The third sample estimation acquisition module is used to obtain third sample estimated depth data based on the third sample color image data; the third sample estimated depth data is used to indicate the depth information corresponding to the third sample color image data.

[0080] The third sample calibration acquisition module is used to obtain third sample calibration depth data based on the third sample estimated depth data and the third sample depth image data.

[0081] The third sample fusion feature acquisition module is used to obtain the third sample fusion feature map by performing weighted processing on the feature fusion branch in the target region determination model based on the third sample color image data and the third sample calibration depth data.

[0082] The region determination model training module is used to train the target region determination model based on the third sample fusion feature map and the target region corresponding to the third sample image.

[0083] The trained target region determination model is used to process the color image data and calibration depth data corresponding to the first image to obtain the target region corresponding to the first image.

[0084] In another aspect, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the above-described target region determination method.

[0085] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned target area determination method.

[0086] The beneficial effects of the technical solutions provided in this application include at least the following:

[0087] By using the first color image data in the first image, estimated depth data corresponding to the first image is obtained. Then, the first depth image data corresponding to the first image is corrected based on the estimated depth data to obtain calibrated depth data. This calibrated depth data is then fused with the color image data, and the target region is determined based on the fused feature map. This scheme improves the accuracy of target region determination by estimating the depth information corresponding to the first image using the color image, correcting the corresponding depth image, and then obtaining the target region based on the corrected depth image data and color image data. Attached Figure Description

[0088] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0089] Figure 1 A schematic diagram of a computer system provided in an exemplary embodiment of this application is shown;

[0090] Figure 2 This is a flowchart illustrating a method for determining a target region according to an exemplary embodiment;

[0091] Figure 3 This is a flowchart illustrating a method for determining a target region according to an exemplary embodiment;

[0092] Figure 4 This is a flowchart illustrating a method for determining a target region according to an exemplary embodiment;

[0093] Figure 5 It shows Figure 4 The illustrated embodiment is a schematic diagram of an RGB-D image channel;

[0094] Figure 6 It shows Figure 4 The illustrated embodiment is a schematic diagram of channel attention weighting.

[0095] Figure 7 It shows Figure 4 The illustrated embodiment is a schematic diagram of cross-modal feature fusion.

[0096] Figure 8 It shows Figure 4 The illustrated embodiment involves a model network framework diagram;

[0097] Figure 9This is a flowchart illustrating a method for determining a target region according to an exemplary embodiment;

[0098] Figure 10 This is a structural block diagram of a target area determination device according to an exemplary embodiment;

[0099] Figure 11 This is a structural block diagram of a target area determination device according to an exemplary embodiment;

[0100] Figure 12 This is a schematic diagram of the structure of a computer device according to an exemplary embodiment. Detailed Implementation

[0101] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0102] First, the terms used in the embodiments of this application will be introduced.

[0103] 1) Artificial Intelligence (AI)

[0104] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0105] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0106] 2) Computer Vision (CV)

[0107] Computer vision is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, and map building.

[0108] 3) Machine Learning (ML)

[0109] Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learn-by-doing.

[0110] 4) Depth image (RGB-D, Red Green Blue-Depth)

[0111] In 3D computer graphics, the RGB color model is an industry-standard color system. It generates a wide variety of colors by varying the red (R), green (G), and blue (B) color channels and superimposing them. RGB represents the colors of these three channels and encompasses almost all colors perceptible to human vision, making it one of the most widely used color systems. The RGB-D image mode adds depth map information to the RGB color model. A depth map is an image or image channel containing information about the distances to the surfaces of objects in a scene from the viewpoint. Similar to a grayscale image, a depth map's pixel value represents the actual distance from the sensor to the object. Typically, RGB images and depth images are registered, resulting in a one-to-one correspondence between pixels.

[0112] The target region determination method provided in this application can be applied to computer devices with strong data processing capabilities. In one possible implementation, the target region determination method provided in this application can be applied to a personal computer, workstation, or server. In one possible implementation, the confidence discrimination model, depth estimation model, and target region determination model involved in this application are trained using a personal computer, workstation, or server. In one possible implementation, the confidence discrimination model, depth estimation model, and target region determination model trained using the training method provided in this application can be applied to application scenarios requiring salient target detection, enabling the determination of the target region corresponding to the acquired RGB-D image, thereby improving the accuracy of target region determination in RGB-D images.

[0113] Please refer to Figure 1 The diagram illustrates a computer system provided in an exemplary embodiment of this application. The computer system includes a terminal 110 and a server 120, wherein the terminal 110 and the server 120 communicate via a communication network. Optionally, the communication network can be a wired network or a wireless network, and the communication network can be at least one of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0114] The terminal 110 is equipped with an application with image processing capabilities. This application may be a virtual reality application, a game application, an image processing application, or an artificial intelligence (AI) application with image processing capabilities. This application embodiment does not limit the scope of the application.

[0115] Optionally, the terminal 110 can be a mobile terminal such as a smartphone, tablet computer, or laptop computer, or a terminal such as a desktop computer or projector computer, or a smart terminal with RGB-D image acquisition components and data processing components. This application embodiment does not limit this.

[0116] Server 120 can be implemented as a single server or as a server cluster consisting of a group of servers. It can be a physical server or a cloud server. In one possible implementation, server 120 is the backend server for the application in terminal 110.

[0117] In one possible implementation of this embodiment, server 120 trains the target region determination model using a pre-set training sample set, which may include RGB-D sample images. After server 120 completes the training process for the target region determination model, it sends the trained model to terminal 110 via a wired or wireless connection. Terminal 110 receives the trained target region determination model and inputs the corresponding data information into an application with image processing capabilities. This allows the user to use the application to invoke image processing functions, enabling the determination of the target region based on the trained model, thus achieving all or part of the steps of the target region determination function.

[0118] Figure 2 This is a flowchart illustrating a target region determination method according to an exemplary embodiment. The method can be executed by a computer device, which may be an image processing device, wherein the image processing device can be the one described above. Figure 1 Terminal 110 in the illustrated embodiment. For example... Figure 2 As shown, the process of this target area determination method may include the following steps:

[0119] Step 201: Obtain a first image; the first image contains first color image data and first depth image data.

[0120] Step 202: Based on the first color image data, obtain first estimated depth data; the first estimated depth data is used to indicate the depth information corresponding to the first color image data.

[0121] Step 203: Based on the first estimated depth data and the first depth image data, obtain the first calibrated depth data.

[0122] Step 204: Based on the first color image data and the first calibration depth data, perform weighted processing to obtain the first fused feature map.

[0123] Step 205: Based on the first fused feature map, determine the target region corresponding to the first image.

[0124] In summary, the solution presented in this application obtains estimated depth data corresponding to the first image from the first color image data in the first image, then corrects the first depth image data corresponding to the first image based on the estimated depth data to obtain calibrated depth data, then fuses the calibrated depth data with the color image data, and determines the target region based on the fused feature map. This solution improves the accuracy of target region determination by estimating the depth information corresponding to the first image from the color image, correcting the corresponding depth image, and then obtaining the target region corresponding to the first image based on the corrected depth image data and the color image data.

[0125] Figure 3 This is a flowchart illustrating a target region determination method according to an exemplary embodiment. The method can be executed by a computer device, which can be a model training device, wherein the model training device can be the one described above. Figure 1 Server 120 in the illustrated embodiment. For example... Figure 3 As shown, the process of this target area determination method may include the following steps:

[0126] Step 301: Obtain the third sample image; the third sample image contains third sample color image data and third sample depth image data.

[0127] Step 302: Based on the third sample color image data, obtain the third sample estimated depth data.

[0128] Step 303: Based on the estimated depth data of the third sample and the depth image data of the third sample, obtain the calibration depth data of the third sample.

[0129] Step 304: Based on the third sample color image data and the third sample calibration depth data, the feature fusion branch in the target region determination model is weighted and processed to obtain the third sample fusion feature map.

[0130] Step 305: Based on the fusion feature map of the third sample and the target region corresponding to the third sample image, train the target region determination model.

[0131] The trained target region determination model is used to process the color image data and calibration depth data corresponding to the first image to obtain the target region corresponding to the first image.

[0132] In summary, the solution presented in this application obtains estimated depth data corresponding to the first image from the first color image data in the first image, then corrects the first depth image data corresponding to the first image based on the estimated depth data to obtain calibrated depth data, then fuses the calibrated depth data with the color image data, and determines the target region based on the fused feature map. This solution improves the accuracy of target region determination by estimating the depth information corresponding to the first image from the color image, correcting the corresponding depth image, and then obtaining the target region corresponding to the first image based on the corrected depth image data and the color image data.

[0133] Figure 4 This is a flowchart illustrating a target region determination method according to an exemplary embodiment. The method can be executed jointly by a model processing device and an image processing device, wherein the model processing device can be the one described above. Figure 1 The server 120 in the illustrated embodiment, the image matching device can be as described above. Figure 1 Terminal 110 in the illustrated embodiment. For example... Figure 4 As shown, the process of this target area determination method may include the following steps:

[0134] Step 401: Obtain the first image.

[0135] The first image includes first color image data and first depth image data.

[0136] In one possible implementation, the first image is an RGB-D image, that is, the first image contains an RGB image and a depth map. Specifically, the first color image data is the image data corresponding to the RGB image contained in the first image, and the first depth image data is the image data corresponding to the depth map contained in the first image.

[0137] In one possible implementation, the first image contains at least two image channels, including a color (RGB) image channel and a depth image channel.

[0138] Please refer to Figure 5 This illustrates a schematic diagram of an RGB-D image channel according to an embodiment of this application. Figure 5 As shown, when the first image is an RGB-D image, the color image data corresponding to the RGB image 501 in the first image is as follows: Figure 5As shown in section 502, the color image data 502 contains three image channels, each corresponding to the pixel data of the three primary colors: red, green, and blue. The depth image data corresponding to the depth image 503 in the first image is as follows: Figure 5 As shown in section 504, in the depth image data 504, there is a depth image channel, and each data in the depth image channel is used to indicate the pixel data of each pixel point in the depth image.

[0139] In one possible implementation, the first color image data is an RGB image contained in the first image; the first depth image data is a depth image contained in the first image.

[0140] In another possible implementation, the first color image data is obtained by feature extraction based on the RGB image contained in the first image; the first depth image data is obtained by feature extraction based on the depth image contained in the first image.

[0141] Step 402: Based on the first depth image data, obtain the confidence level corresponding to the first depth image data.

[0142] The confidence level is used to indicate the accuracy of the image data corresponding to the target region in the first depth image data.

[0143] Because depth acquisition devices typically have low accuracy and are easily affected by the natural environment, the accuracy of the first color image data contained in the first image acquired by such devices is usually low. Therefore, the accuracy of the first depth image data can be judged by its confidence level to determine its authenticity. A high confidence level indicates that the data in the first depth image data has high reliability, and thus the result obtained after processing the first depth image data is also relatively realistic. Conversely, a low confidence level indicates that the data in the first depth image data does not have sufficient reliability, and thus the result obtained after processing the first depth image data lacks sufficient credibility.

[0144] In one possible implementation, the confidence level of the first depth image data is obtained by processing it through a confidence discrimination model.

[0145] The confidence discrimination model is a machine learning model trained using the first sample image as the sample and the confidence type corresponding to the first sample image as the label.

[0146] In one possible implementation, the confidence type corresponding to the first sample image can include positive and negative samples. When the confidence type of the first sample image is positive, it indicates that the first sample image is a sample image with high accuracy. When the confidence type of the first sample image is negative, it indicates that the first sample image is a sample image with low accuracy.

[0147] In one possible implementation, the first depth image data is input into the confidence discrimination model to obtain the confidence probability distribution corresponding to the first depth image data. The confidence probability distribution includes the probability that the first depth image data is a positive sample and the probability that the first depth image data is a negative sample. The probability that the first depth image data is a positive sample is obtained as the confidence level corresponding to the first depth image data.

[0148] In this context, a positive sample of the first depth image data indicates that the first depth image data is reliable image data; a negative sample of the first depth image data indicates that the first depth image data is unreliable image data.

[0149] In one possible implementation, a first training sample set is obtained; the first training sample set includes a first sample image and the confidence type corresponding to the first sample image; based on the first sample image, data processing is performed through the confidence discrimination model to obtain the confidence probability corresponding to the first sample image; the confidence probability is used to indicate the probability that the first sample image is a positive sample; based on the confidence probability corresponding to the first sample image and the confidence type corresponding to the first sample image, the confidence discrimination model is trained.

[0150] During the training of the confidence discrimination model, a first sample image from the first training sample set can be input into the model to obtain the probability distribution corresponding to the first sample image. This probability distribution includes the probability of the first sample image corresponding to a positive sample and the probability of the first sample image corresponding to a negative sample. The confidence probability of the first sample image is the probability of it corresponding to a positive sample. Based on the confidence type and the confidence probability of the first sample image, the confidence discrimination model is trained. The trained model can then process the input image data to obtain the confidence probability of the input image data.

[0151] In one possible implementation, the first sample image is a depth image, in which case the confidence discrimination model trained based on the first sample image can determine the confidence level of the depth image.

[0152] In another possible implementation, the first sample image includes a depth image and an RGB color image. Therefore, the confidence discrimination model trained based on the first sample image can determine the confidence of the depth image and also determine the confidence of the RGB color image.

[0153] In one possible implementation, the confidence discrimination model can be a machine learning model that includes convolutional layers and fully connected layers.

[0154] In this confidence judgment model, the convolutional layer extracts features from the first input sample image, and the fully connected layer performs a linear transformation on the extracted features to obtain a two-dimensional confidence vector corresponding to the first sample image. Based on the confidence vector, the probability distribution corresponding to the first sample image is obtained.

[0155] In one possible implementation, in response to the first sample image comprising a depth image and an RGB color image, the confidence discrimination model may further comprise a depth confidence discrimination branch and a color confidence discrimination branch.

[0156] When the first sample image contains both a depth image and an RGB color image, since the depth image and the RGB color image belong to different modalities, training the machine learning model to simultaneously evaluate the confidence scores of both the depth image and the RGB color image using the same machine learning model structure may result in poor actual confidence score discrimination performance. Therefore, the confidence score discrimination model can include a depth confidence discrimination branch and a color confidence discrimination branch. The depth confidence discrimination branch is trained based on the depth image in the first sample image and the corresponding confidence category of the depth image in the first sample image, and the color confidence discrimination branch is trained based on the color image in the first sample image and the corresponding confidence category of the color image in the first sample image.

[0157] The trained confidence discrimination model can perform well in discriminating the confidence of both color images and depth images.

[0158] In one possible implementation, when the first sample image contains a depth image and an RGB color image, the confidence type of the first sample image can respectively include the confidence type corresponding to the depth image in the first sample image and the confidence type corresponding to the color image in the first sample image.

[0159] The confidence type corresponding to the depth image in the first sample image can be the same as or different from the confidence type corresponding to the color image in the first sample image. For example, when both the color image and the depth image in the first sample image have high precision, the confidence type corresponding to the color image in the first sample image can be a positive sample, and the confidence type corresponding to the depth image in the first sample image can also be a positive sample; when the precision of the color image in the first sample image is high, but the precision of the depth image in the first sample image is low, the confidence type corresponding to the depth image in the first sample image is a negative sample, and the confidence type corresponding to the color image in the first sample image is also a negative sample.

[0160] In one possible implementation, a second training sample set is obtained; the second training sample set includes training sample images and target regions corresponding to the training sample images; the training sample images include training color sample data and training depth sample data; a color image processing branch in the model is determined through the target region, and the training color sample data is processed to obtain a prediction region corresponding to the training color sample data; a depth image processing branch in the model is determined through the target region, and the training depth sample data is processed to obtain a prediction region corresponding to the training depth sample data; based on the prediction region corresponding to the training color sample data, the prediction region corresponding to the training depth sample data, and the target region corresponding to the training sample image, a confidence score for the training sample image is determined; in response to the confidence score of the training sample image satisfying a specified condition, the confidence type of the training sample image is determined, and the training sample image is identified as the first sample image.

[0161] Specifically, the color image processing branch in the target region determination model is a machine learning model pre-trained using sample color images as samples and the target regions corresponding to the sample color images as annotations; the depth image processing branch in the target region determination model is a machine learning model pre-trained using sample depth images as samples and the target regions corresponding to the sample depth images as annotations.

[0162] The second training sample set includes various training sample images and target regions corresponding to each training sample image. For one of the training sample images in the second training sample set, the training sample image includes training color sample data and training depth sample data, and the training color sample data and the training depth sample data both correspond to the target region of the training sample image.

[0163] The training color sample data is processed by the color image processing branch in the target region determination model to obtain the predicted region corresponding to the training color sample data; the training depth sample data is processed by the depth image processing branch in the target region determination model to obtain the predicted region corresponding to the training depth sample data. Based on the predicted region corresponding to the training color sample data, the predicted region corresponding to the training depth sample data, and the target region, the confidence type of the training sample image can be determined. The first sample image determined by the training sample image is then used to train the confidence discrimination model.

[0164] In one possible implementation, the sample color image is used as a training sample and input into the color image processing branch of the target region determination model to obtain the predicted region corresponding to the sample color image. Based on the predicted region and the target region corresponding to the sample color image, the color image processing branch of the target region determination model is trained to obtain a pre-trained color image processing branch. This pre-trained color image processing branch can process the input color image to obtain the corresponding predicted region in the color image. At this point, it can be assumed that when the quality of the input color image is good, the predicted region of the input color image should have a high degree of overlap with the actual target region corresponding to the color image. Therefore, based on the degree of overlap between the predicted region and the actual target region corresponding to the color image, it can be determined whether the color image is a high-quality color image, and the confidence level of the color image can be obtained.

[0165] The sample depth image is used as a training sample and input into the depth image processing branch of the target region determination model to obtain the predicted region corresponding to the sample depth image. Based on the predicted region and the target region corresponding to the sample depth image, the depth image processing branch of the target region determination model is trained to obtain a pre-trained depth image processing branch. This pre-trained depth image processing branch can process the input depth image to obtain the predicted region in the depth image. It can be assumed that when the quality of the input depth image is good, the overlap between the predicted region and the actual target region corresponding to the depth image should be high. Therefore, the confidence level of the depth image can be obtained based on the overlap between the predicted region and the actual target region.

[0166] In one possible implementation, the confidence score includes a color confidence score and a depth confidence score; the color confidence score of the training sample image is determined based on the overlap between the predicted region corresponding to the training color sample data and the target region corresponding to the training sample image; the depth confidence score of the training sample image is determined based on the overlap between the predicted region corresponding to the training depth sample data and the target region corresponding to the training sample image.

[0167] The confidence scores mentioned above include color confidence scores and depth confidence scores. The color confidence score is determined based on the degree of overlap between the predicted region corresponding to the training color sample data and the target region corresponding to the training sample image. A higher degree of overlap between the predicted region corresponding to the training color sample data and the target region corresponding to the training sample image results in a higher color confidence score. Similarly, the depth confidence score is determined based on the degree of overlap between the predicted region corresponding to the training depth sample data and the target region corresponding to the training sample image. A higher degree of overlap between the predicted region corresponding to the training depth sample data and the target region corresponding to the training sample image results in a higher depth confidence score.

[0168] In one possible implementation, in response to the depth confidence score of the training sample image satisfying a specified condition, the confidence type of the training sample image is obtained, and the training sample image is obtained as the first sample image.

[0169] In one possible implementation, in response to the depth confidence score of the training sample image being greater than a first confidence threshold, the confidence type of the training sample image is determined to be a positive sample, and the training sample image is acquired as the first sample image.

[0170] In one possible implementation, in response to the depth confidence score of the training sample image being less than a second confidence threshold, the confidence type of the training sample image is determined to be a negative sample, and the training sample image is acquired as the first sample image.

[0171] When the depth confidence score of the training sample image is greater than the first confidence threshold, meaning that the predicted region obtained after data processing by the depth image processing branch in the target region determination model has a high degree of overlap with the target region, the training depth sample data in the training sample image can be considered to be highly reliable data. Therefore, the confidence type of the training sample image is determined as a positive sample, and the training sample image is designated as the first sample image to train the confidence discrimination model. When the depth confidence score of the training sample image is less than the second confidence threshold, meaning that the predicted region obtained after data processing by the depth image processing branch in the target region determination model has a high degree of overlap with the target region, the training depth sample data in the training sample image can be considered to be highly reliable data. Therefore, the confidence type of the training sample image is determined as a positive sample, and the training sample image is designated as the first sample image to train the confidence discrimination model. The predicted region obtained after processing the training depth sample data through the depth image processing branch of the target region determination model has a low overlap with the target region. Therefore, the training depth sample data in the training sample image can be considered as data with low confidence. Thus, the confidence type of the training sample image is determined as a negative sample, and the training sample image is determined as the first sample image to train the confidence discrimination model. At this time, the confidence discrimination model obtained after training with a sufficient number of training sample images can determine the confidence of the input sample image to determine the quality of the depth image data in the input sample image.

[0172] In one possible implementation, based on each training sample image in the second training sample set, the depth image processing branch in the model is determined by the target region, and the depth confidence score corresponding to each training sample image is determined. The depth confidence scores corresponding to each training sample image are sorted from largest to smallest, and the confidence type of the training sample images with a depth confidence score of the top a% is determined as positive samples and obtained as the first sample images, where a is greater than 0.

[0173] In another possible implementation, the confidence type of the training sample images with a depth confidence score of b% is determined as negative samples and obtained as the first sample image, where b is greater than 0.

[0174] At this point, the training sample images in the second training sample set with a depth confidence score in the top a% can be considered as training sample images with high accuracy. Therefore, the depth image data of the top a% training sample images is relatively accurate. At this point, the top a% training sample images can be obtained as the first sample images and determined as positive samples. The training sample images in the second training sample set with a depth confidence score in the bottom b% can be considered as training sample images with low accuracy. Therefore, the bottom b% training sample images can also be obtained as the first sample images and determined as negative samples, so as to achieve the training of the confidence discrimination model.

[0175] In one possible implementation, in response to the depth confidence score and color confidence score of the training sample image satisfying a specified condition, the confidence type of the training sample image is obtained, and the training sample image is obtained as the first sample image.

[0176] The confidence type of the training sample image can also be determined based on the confidence scores of the training depth sample data and the training color image data of the training sample image.

[0177] In one possible implementation, in response to the depth confidence score of the training sample image being greater than the color confidence score of the training sample image, the confidence type of the training sample image is obtained as a positive sample, and the training sample image is obtained as the first sample image, so as to train the confidence discrimination model.

[0178] Since the accuracy of RGB images is usually higher than that of depth images in training sample images (i.e., RGB-D images), when the depth confidence score corresponding to the depth image is greater than the color confidence score corresponding to the RGB image, it means that the training depth sample data in the training sample image has more accurate target region information than the training color sample data. In this case, the training sample image can also be taken as the first sample image to train the model.

[0179] Step 403: Based on the first color image data, obtain the first estimated depth data.

[0180] The first estimated depth data is obtained based on the first color image data, that is, the first estimated depth data is the depth information of the first image contained in the first color image.

[0181] In one possible implementation, the first estimated depth data is obtained by processing the first color image data using a depth estimation model.

[0182] The depth estimation model is a machine learning model trained using the second sample image as a sample and the depth image data corresponding to the second sample image as a label; the second sample image is a sample image whose confidence level meets the first specified condition.

[0183] In one possible implementation, the second sample image is a sample image with a confidence level greater than a third confidence threshold.

[0184] In another possible implementation, an estimation model training sample set is obtained, which contains at least two estimation sample images; the estimation sample images contain color image data and depth image data; based on the depth image data of each estimation sample image in the estimation model training sample set, the data is processed by the confidence discrimination model to obtain the depth confidence score corresponding to each estimation sample image, and the estimation sample image whose depth confidence score meets the confidence condition is obtained as the second sample image.

[0185] In one possible implementation, the depth confidence scores corresponding to each estimated sample image in the training sample set of the estimation model are obtained, and the depth confidence scores corresponding to each estimated sample image are sorted from largest to smallest. The estimated sample images with the top c% confidence scores are then obtained as the second sample images.

[0186] In other words, by using the above method, estimated sample images with high depth confidence scores can be selected and obtained as second sample images. These second sample images contain both depth image data and color image data. A high depth confidence score indicates that the depth image data corresponding to the second sample image is of good quality. Therefore, by using the second sample image as a sample (i.e., using the color image data corresponding to the second sample image as a sample) and the depth image data corresponding to the second sample image as a label, the trained depth estimation model can obtain high-quality depth image data based on the input color image data after data processing. This enables the estimation of the depth image data of the second sample image using the color image data corresponding to the second sample image.

[0187] Step 404: Perform weighted processing on the first estimated depth data and the first depth image data based on the confidence level corresponding to the first depth image data to obtain the first calibrated depth data.

[0188] In one possible implementation, the confidence level corresponding to the first depth image is obtained by processing the first depth image data through the confidence level discrimination model. That is, the confidence level corresponding to the first depth image data can be used to indicate the accuracy of the first depth image data in the first image. The first estimated depth data is the depth image data obtained by processing the first color image data corresponding to the first image based on the depth estimation model. Therefore, the first calibrated depth data is obtained by weighted summing the first estimated depth data and the first depth image data based on the confidence level corresponding to the first depth image data. Compared with the directly obtained first depth image data, it takes into account the accuracy of the first depth image data itself and combines the depth information contained in the first color image. Therefore, the first calibrated depth image data can more accurately represent the depth information of the first image.

[0189] Step 405: Based on the first color image data and the first calibration depth data, perform weighted processing to obtain the first fused feature map.

[0190] In one possible implementation, based on the first color image data and the first calibration depth data, the first fused feature map is obtained by performing attention-based weighted processing on the feature fusion branch in the target region determination model.

[0191] In one possible implementation, the feature fusion branch includes a first pooling layer, a second pooling layer, a first fully connected layer, and a second fully connected layer. Based on the first color image data, global pooling is performed through the first pooling layer to obtain first color pooled data. Based on the first color pooled data, data processing is performed through the first fully connected layer to obtain a first color vector. Based on the first depth image data, global pooling is performed through the second pooling layer to obtain first depth pooled data. Based on the first depth pooled data, data processing is performed through the second fully connected layer to obtain a first depth vector. Based on the first color image data and the first calibration depth data, channel attention weighting is performed using the first color vector and the first depth vector to obtain the first fused feature map.

[0192] The first color vector is used to indicate the weight corresponding to the first color image data; the first depth vector is used to indicate the weight corresponding to the first depth image data.

[0193] In the process of fusing the first color image data and the first calibration depth data to obtain the first fused feature map, the first color image data can first be globally pooled through a first pooling layer to obtain the first color pooled data. This first color pooled data represents the overall data size of the first color image data, that is, it represents the importance of the first color image data to the target region. Then, based on the first color pooled data, data processing is performed through a first fully connected layer, that is, through linear transformation, it is transformed into a first color vector. This first color vector can be used to indicate the weight ratio of each channel in the first color image data (this weight ratio indicates the importance of the features of each channel). Alternatively, the first depth image data can be globally pooled through a second pooling layer to obtain the first depth pooled data. This first depth pooled data represents the overall data size of the first depth pooled data, that is, it represents the importance of the first depth image data to the target region. Based on the first depth pooling data, the data is processed through the second fully connected layer, that is, linearly converted into the first depth vector. The second depth vector can indicate the weight ratio of the image channel corresponding to the first depth image data. Then, the first color vector and the first depth vector are fused after channel attention weighting processing to obtain the first fused feature map.

[0194] Please refer to Figure 6 This illustrates a channel attention weighting diagram related to an embodiment of this application. For example... Figure 6 As shown, for a feature map 601 with channel C and size W×H, global pooling is first performed on the feature maps of all channels to obtain an average feature map. This average feature map is then transformed through a fully connected layer to form channel attention values. Finally, the channel attention values ​​are multiplied by all channel feature maps to form a channel attention feature map 602. Since this channel attention mechanism maps the mean values ​​of each channel to channel attention values ​​through a fully connected layer, the attention feature map obtained by weighting these channel attention values ​​will pay more attention to the channel with the larger average value (i.e., the image features of the channel with the larger average value have greater weights).

[0195] Please refer to Figure 7 This illustration shows a schematic diagram of cross-modal feature fusion according to an embodiment of this application. Figure 7As shown, for the first color image data 701 corresponding to the first image, the first color image data 701 can be input into the first pooling layer 702 of GAP (Global Average Pooling) for global pooling. The value obtained after global pooling can be input into the first fully connected layer 703 of FC (Fully connected) and then processed by the activation function 704 to obtain the color vector 705 corresponding to the first color image data.

[0196] For the first depth image data 706 corresponding to the first image, the first depth image data 706 can be input into the GAP second pooling layer 707 for global pooling. The value obtained after global pooling can be input into the FC second fully connected layer 708 and then processed by the activation function 709 to obtain the depth vector 710 corresponding to the first depth image data.

[0197] The first color image data can be image features obtained by feature extraction from the RGB image of the first image; the first depth image data can be image features obtained by feature extraction from the depth image of the first image.

[0198] Among them, the color vector 705 and the depth vector 710 can be simultaneously input into the maximum value acquisition module 711, which takes the larger value of each dimension of the color vector and the depth vector and obtains the maximum value vector 712.

[0199] For the first color image data 701, channel attention weighting is performed using the color vector 705 as the weight to obtain a first color-weighted feature map. The first color image data 701 is then further weighted using the extremum vector as the weight to obtain a second color-weighted feature map. The first and second color-weighted feature maps are then fused to obtain the color feature map 713. For the first depth image data 706, channel attention weighting is performed using the depth vector 710 as the weight to obtain a first depth-weighted feature map. The first depth image data 706 is then weighted using the extremum vector as the weight to obtain a second depth-weighted feature map. The first and second depth-weighted feature maps are then fused to obtain the depth feature map 714. Finally, the color feature map 713 and the depth feature map 714 are input into the convolution module C to achieve the fusion of the color feature map and the depth feature map, obtaining the fused feature map 715.

[0200] That is, the above Figure 7This application illustrates a cross-reference module involved in an embodiment of the present application. After correcting the depth image data, the corrected depth map and the RGB image are input together into a dual-stream feature extraction network (i.e., the color image processing branch and the depth image processing branch in the pre-training stage) to generate multi-level features. Features extracted from the RGB channels contain rich semantic and texture information, while features from the depth channels contain more discriminative scene layout cues, complementing the RGB features. This application proposes a cross-reference module as a cross-modal feature fusion strategy, and in... Figure 7 The proposed cross-reference module aims to mine and combine the most distinctive channels (i.e., feature detectors) from depth and RGB features to generate more informative features. Given two input features from the RGB and depth streams, global average pooling is first used to obtain global statistics for the RGB and depth views. These two feature vectors are then fed into a fully connected layer and a softmax activation function, respectively, to obtain channel attention vectors that reflect the importance of RGB and depth features. The attention vectors are then applied to the input features via channel multiplication. In this way, the cross-reference module explicitly focuses on important features while suppressing unnecessary scene understanding features. Based on the decoded RGB and depth features, a convolutional layer generates cross-modal fusion features. Furthermore, a triplet loss constrains the cross-modal fusion features, making them closer to the foreground while increasing the distance between foreground and background features.

[0201] Step 406: Based on the first fused feature map, determine the target region corresponding to the first image.

[0202] In one possible implementation, based on the first color image data, data processing is performed through the depth image processing branch in the target region determination model to obtain a first color feature map; based on the first depth image data, data processing is performed through the depth image processing branch in the target region determination model to obtain a first depth feature map; based on the first fused feature map, the first depth feature map, and the first color feature map, the target region corresponding to the first image is determined.

[0203] The target region determination model is a machine learning model trained using a third sample image as a sample and the target region corresponding to the third sample image as an annotation.

[0204] The target region corresponding to the first image can be obtained by simultaneously considering the first depth feature map, the first color feature map, and the first fusion feature map corresponding to the first image. The first depth feature map is obtained by inputting the first depth image data into the depth image processing branch of the target region determination model; therefore, the first depth feature map can indicate the predicted region corresponding to the first depth image data. The first color feature map is obtained by inputting the first color image data into the color image processing branch of the target region determination model; therefore, the first color feature map can indicate the predicted region corresponding to the first color image data. By considering the first color feature map and the first depth feature map based on the first fusion feature map, the target region is obtained. This allows for the consideration of the original features in the color feature map and the original features in the depth feature map while obtaining the first fusion feature map after calibrating the depth image data, thus improving the accuracy of target region recognition.

[0205] In one possible implementation, a third sample image is acquired; the third sample image includes third sample color image data and third sample depth image data; based on the third sample color image data, third sample estimated depth data is obtained; based on the third sample estimated depth data and the third sample depth image data, third sample calibrated depth data is obtained; based on the third sample color image data and the third sample calibrated depth data, a weighted processing is performed through the feature fusion branch in the target region determination model to obtain a third sample fusion feature map; based on the third sample fusion feature map and the target region corresponding to the third sample image, the target region determination model is trained.

[0206] In the process of determining the target region model, the third sample image and the target region corresponding to the third sample image are acquired. The third sample image contains the third sample color image data and the third sample depth image data. Therefore, the third sample depth image data can be calibrated first based on the third sample color image data and the third sample depth image data to obtain the third sample calibration depth data. Based on the third sample calibration depth data and the third sample color image data, the image features of different modalities are fused to obtain the third sample fusion feature map. Based on the third sample fusion feature map, the prediction region corresponding to the third sample image is determined. Then, the prediction region corresponding to the third sample image and the target region corresponding to the third sample image are used to train the target region determination model.

[0207] In one possible implementation, based on the third sample color image data, data processing is performed through the color image processing branch in the target region determination model to obtain a third sample color feature map; based on the third sample depth image data, data processing is performed through the depth image processing branch in the target region determination model to obtain a third sample color feature map; based on the third sample color feature map, the third sample depth feature map, the third sample fusion feature map, and the target region corresponding to the third sample image, the target region determination model is trained.

[0208] Since the target region determination model includes color image processing, depth image processing, and feature fusion branches, the color image processing branch can be used to process the third sample color image data to obtain the corresponding third sample color feature map; then, the depth image processing branch can be used to process the third sample depth image data to obtain the corresponding third sample depth feature map; and simultaneously, based on the third sample color feature map, the third sample depth feature map, the third sample fused feature map, and the target region corresponding to the third sample image, the target region determination model can be trained. The trained target region determination model is then... The system can simultaneously process color image data to obtain the corresponding feature map, process depth image data to obtain the corresponding feature map, and fuse the color image data and calibrated depth image data to obtain a fused feature map. Each feature map is used to indicate the corresponding prediction region. Therefore, the target region determination model can obtain the prediction region corresponding to the color image data, the prediction region corresponding to the depth image data, and the prediction region corresponding to the fused feature map based on the color image data and calibrated depth image data, and obtain the target region of the input image based on the three prediction regions.

[0209] In one possible implementation, the overall optimization objective of the network is... It consists of four parts, including the cross-entropy loss of the RGB, depth, and fusion branches, as well as the triplet loss in the cross-reference module, as shown below:

[0210]

[0211] in, , as well as These are the loss functions corresponding to the outputs of the three decoders mentioned above. This represents the triplet loss function corresponding to each convolutional layer, where N=3 represents the number of convolutional layers involved in the triplet loss function. 0.2 is acceptable.

[0212] In one possible implementation, the cross-entropy loss function could also be edge-guided BCE of boundary information to improve the prediction results' learning of object boundaries.

[0213] Please refer to Figure 8 This illustrates a model network framework diagram related to an embodiment of this application. Figure 8 As shown, it illustrates the framework of the solution embodiment of this application:

[0214] For the color image data 801 of the input first image, it is processed by the color image processing branch 802 in the target region determination model to obtain the color feature map corresponding to the first image; for the depth image data 803 of the input first image, it is processed by the depth image processing branch 804 in the target region determination model to obtain the depth feature map corresponding to the first image; then the features extracted from some convolutional layers in the color image processing branch 802 and the features extracted from some convolutional layers in the depth image processing branch 804 are input into the feature fusion branch composed of the cross-reference module CRM to obtain the fused feature map; then the color feature map, depth feature map and fused feature map are decoded by three decoders respectively, and the corresponding outputs are summed to obtain the final saliency map.

[0215] That is, the above Figure 8 Based on a two-stream feature extraction network, this system consists of two core components: a depth calibration strategy and a fusion strategy. First, a depth calibration strategy is proposed to correct potential noise from unreliable original depth maps. The corrected depth better reflects scene layout and identifies foreground regions than the original depth. Given corrected RGB-D data, the RGB image and the corrected depth are simultaneously input into the two-stream feature extraction network to generate multi-level features. Then, a fusion strategy cross-reference module is designed to integrate effective cues from RGB and depth features to cross-modal fusion features. This allows three decoding branches to process RGB, depth, and fused features separately. All features are processed individually, and the corresponding outputs are summed to obtain the final saliency map.

[0216] Salient object detection (SOD) is an important computer vision problem aimed at identifying and segmenting the most salient objects in a scene. It has been successfully applied to various tasks such as object recognition, image retrieval, SLAM (simultaneous localization and mapping), and video analysis. To address the inherent challenges in handling difficult scenes with low texture contrast or cluttered backgrounds, depth information is incorporated as a supplementary input source. Adding depth information to the RGB image as additional input enables the localization of salient objects in challenging scenes.

[0217] In this embodiment of the application, based on, as follows Figure 8 The dual-stream feature extraction network shown comprises two core components: depth calibration and a fusion strategy. Based on Figure 8 The dual-stream feature extraction network shown in this application also proposes a depth calibration (DC) strategy to correct potential noise caused by unreliable original depth maps and obtain calibrated depth. The calibrated depth better reflects the scene layout and identifies foreground regions than the original depth. Now, given calibrated RGB-D paired data, the RGB image... and corrected depth image The input is fed into a two-stream feature extraction network to generate hierarchical features. For each stream, a codec network is used as the backbone. This is followed by a fusion strategy: Cross-Reference Modules (CRMs) are designed to integrate features from RGB and depth features to cross-modal fused features; this results in three decoding branches processing RGB, depth features, and fused hierarchical features. These features are processed individually, and the corresponding outputs are summed to the final saliency map. .

[0218] Effective spatial information from depth maps plays a crucial role in helping to locate salient regions in challenging scenes, such as cluttered backgrounds and low-contrast situations. However, unreliable raw depth and potential depth acquisition errors, due to factors such as observation distance, occlusion, or reflection, can hinder models from extracting accurate information from depth maps. To address the performance bottleneck caused by depth map noise, this application calibrates the raw depth to better represent the scene layout. Two key issues addressed in this application are: 1) how the model learns to distinguish between poor-quality depth maps (negative cases) and high-quality depth maps (positive cases); and 2) how to create calibrated / corrected depth maps that retain useful cues from high-quality depth maps while correcting unreliable information from low-quality depth maps. Therefore, this application proposes a depth calibration (DC) strategy, a core component of DCF. This requires two consecutive steps to select representative samples and generate calibrated depth maps.

[0219] To address the first key issue, a difficulty-aware selection strategy is proposed, aiming to select the most representative positive and negative samples from the training database. These samples are then used to train a discriminator / classifier to predict the quality of the depth map, reflecting its reliability. First, this application pre-trains two model branches using the same architecture, taking RGB data and depth data as inputs under saliency mapping supervision, respectively, denoted as... , Then, based on the saliency of the predictions from the two baseline models, a selection scheme is designed to measure whether the depth map can provide reliable information. Specifically, based on the saliency results generated by the RGB stream and the depth stream, the intersection (IoU) metric between the predicted saliency of the two streams and the ground-truth saliency is first calculated, denoted as follows: and For each training sample, then, for all training samples... The scores are sorted from largest to smallest. Based on the score ranking, the top 20% of training samples will be considered as the typical positive sample set. (That is to say, the quality of the depth map is acceptable) and the bottom 20% will be considered a typical negative sample set. (That is to say, the quality of the depth map is bad and unacceptable). Furthermore, when > In this case, these samples are also considered positive samples, which shows that the raw depth data provides richer global cues than the RGB input when identifying foreground regions.

[0220] Based on selected representative positive and negative samples, a binary discriminator / classifier based on the ResNet-18 model architecture is trained to evaluate the reliability of the depth map. Therefore, the trained discriminator is able to predict a reliability score. This indicates the probability of the depth map being positive or negative. The higher the value, the better the quality of the original depth map.

[0221] Furthermore, a depth estimator containing multiple convolutional blocks was established. This depth estimator was trained using RGB images and high-quality depth data from the positive set to mitigate the inherent noise from inaccurate raw depth data. In the depth calibration module, directly using the raw depth map may be unreliable; therefore, the raw depth map was replaced with a weighted sum of the raw depth map and the estimated depth, which was obtained based on this depth estimator. Thus, the following was obtained: The calibrated depth map is shown in the following formula:

[0222]

[0223] in," "and" "" represents the depth estimated by the depth estimator and the original depth map, respectively.

[0224] After the depth correction process is completed, the calibrated depth map will be... The RGB images are fed into a two-stream feature extraction network to generate hierarchical features, respectively. and Note that in this embodiment, the last three convolutional blocks with rich semantic features are retained, while the first two high-resolution convolutional blocks are removed to balance computational cost. Generally, features extracted from the RGB channels contain rich semantic and texture information; meanwhile, features from the depth channels contain more discriminative scene layout cues, complementing the RGB features. To integrate cross-modal information, a fusion strategy called Cross Reference Module (CRM) is designed, as shown in the embodiment of this application. Figure 7 As shown.

[0225] The proposed CRM aims to mine and combine the most discriminative channels (i.e., feature detectors) from depth and RGB features to generate more informative features. Specifically, it generates two input features for the i-th convolutional block of the RGB steam and depth steam. and First, global average pooling is used to obtain global statistics for the RGB and depth views. Then, these two feature vectors are fed into a fully connected layer (FC) and a softmax activation function, respectively, to obtain the channel attention vectors. and This reflects the importance of RGB features and depth features, respectively. Then, the attention vector is applied to the input features using channel multiplication. In this way, CRM explicitly focuses on important features while suppressing unnecessary scene-based feature understanding. The entire process can be defined as:

[0226]

[0227] in and These are the parameters in the fully connected layer that correspond to the features of the i-th layer. This indicates that average pooling is performed. Then, based on the channel attention characteristics, further processing is performed as follows: Channel attention weighting operation, in which This represents a multiplication operation based on a channel.

[0228] Additionally, from the attention vector as well as By aggregating through a maxima function, more prominent feature channels are obtained from the RGB and depth streams. These channels are then fed into a normalization operation, normalizing the output to the range of 0 to 1, thus obtaining cross-referenced channel attention vectors. This step can be defined as:

[0229]

[0230] Based on the fused channel attention vector Input and Weighting is performed to obtain the enhanced features. and The enhanced features from the RGB and Depth branches are further connected and fed into a 1×1 convolutional layer to generate cross-modal fusion features. This process can be defined as:

[0231]

[0232]

[0233] Subsequently, the cross-modal fusion features were analyzed using a triplet loss function. The process involves merging features closer to the foreground while simultaneously increasing the distance between foreground and background features. This is achieved by... The anchor features are set as follows: features corresponding to salient regions are used as positive features, and features corresponding to background regions are used as negative features.

[0234]

[0235]

[0236] Where S represents the labeled salient image region.

[0237] The triplet loss function can be calculated using the following formula:

[0238]

[0239] Where d represents the Euclidean distance and m represents the margin parameter, which is set to 1.0.

[0240] Based on CRM, cross-modal features can be obtained. And simultaneously include RGB stream features and depth flow characteristics The data is decoded using three decoders, and the outputs of the three decoders are summed to obtain the final salient region. .

[0241] The overall optimization objective of this scheme can be described as follows: ,in, , as well as These are the loss functions corresponding to the outputs of the three decoders mentioned above. N=3 represents the number of convolutional layers involved in the triplet loss function. In this application... 0.2 is acceptable.

[0242] In salient object detection, complex backgrounds and the similarity between objects and their surroundings are generally considered challenging scenarios. This naturally leads to the introduction of additional depth information beyond the traditional RGB image as input, a process known as depth-induced (RGB-D) salient object detection. However, this emerging research direction is largely hampered by the prevalent noise and blurring in the original depth image. To address these issues, this application proposes a depth calibration fusion framework comprising two components: a novel learning strategy to calibrate latent biases in the original depth image to improve detection performance; and an efficient cross-reference module to fuse complementary features from both the RGB and depth map modalities. Extensive experiments demonstrate that this method outperforms other state-of-the-art methods.

[0243] Saliency object detection (SOD) has significant value in real-world applications. It aims to identify the most attention-grabbing target regions within a scene. Unlike gaze point prediction, which originated in cognitive and psychological research, SOD is widely used across various fields. In computer vision, applications include image understanding, image caption generation, object detection, unsupervised video object segmentation, and semantic segmentation. In computer graphics, SOD is widely used in VR (Virtual Reality) rendering, automatic image cropping, image retargeting, and video summarization. In robotics, applications such as human-computer interaction and object discovery, as well as scene understanding for obstacle-avoiding robots, also benefit from SOD. However, mainstream SOD methods are typically based on single-input RGB images, resulting in mediocre performance in complex scenes. Therefore, the introduction of depth images has significantly improved the localization capabilities of SOD in challenging scenes. However, due to the influence of depth acquisition devices and environmental conditions, some depth maps may contain noticeable noise. Therefore, it is necessary to introduce a depth calibration strategy in the current field of RGB-D salient target detection to improve the utilization efficiency of depth information and thus improve the detection accuracy.

[0244] This application proposes a solution for salient object detection based on depth map quality calibration. First, two salient discrimination networks are pre-trained, one using RGB and the other using depth maps as input. Then, by comparing the performance of the two pre-trained networks, a depth calibration learning strategy is designed to improve and enhance the quality of the depth map. Furthermore, a cross-reference module is introduced to effectively fuse information from the complementary features of depth and RGB, greatly enhancing the utilization of depth information for salient object detection.

[0245] Furthermore, this application proposes a general depth map calibration framework. This framework can be utilized in other advanced RGB-D salient object detection methods, and all of them offer significant performance gains.

[0246] Table 1 shows the performance of other schemes and the scheme described in the embodiments of this application on the SIP dataset, verifying the superior performance of the proposed method. Furthermore, the method proposed in the embodiments of this application has achieved excellent performance on multiple large-scale publicly available salient object detection datasets.

[0247]

[0248] Table 1

[0249] Table 2 validates the performance gains of each component in the proposed method. Table 2 uses RGB data and the original depth map as inputs, respectively. It can be seen that the RGB branch outperforms the depth branch using the original depth map, indicating that the RGB input contains more semantic and texture information than the depth input. To evaluate the effectiveness of the depth calibration strategy, benchmark networks using the original depth and the calibrated depth were compared. As shown in Table 2, the calibrated depth reduced the MAE error by an average of 14.51% across the four datasets. Furthermore, to verify the generalization ability of the proposed depth calibration module, the generated calibrated depth was applied to two state-of-the-art models, D3Net and DMRA. As shown in Table 3, training D3Net and DMRA with the calibrated depth instead of the original depth map achieved significant performance improvements on both the DUT-D and NJU2K datasets. The MAE scores for D3Net and DMRA decreased by 12.5% ​​and 9.1%, respectively. Therefore, extensive experiments demonstrate the advantages of the proposed depth calibration strategy.

[0250]

[0251] Table 2

[0252] Furthermore, for the cross-modal fusion module to integrate RGB and depth features, a simple solution is to use concatenation followed by convolution to fuse complementary features. Table 2 shows that, by comparing (d) and (f), the proposed cross-reference module can better fuse the complementary information of RGB and depth features compared to direct feature fusion. Meanwhile, compared to (f), removing the triplet loss function resulted in a decrease in performance for all experiments, demonstrating the effectiveness of the three loss functions in enhancing feature representation.

[0253] Table 3 shows the impact of the calibration depth scheme involved in the embodiments of this application on the determination of the target area.

[0254]

[0255] Table 3

[0256] In summary, the solution described in this application involves obtaining estimated depth data corresponding to the first image using the first color image data in the first image, then correcting the first depth image data corresponding to the first image based on the estimated depth data to obtain calibrated depth data, and finally fusing the calibrated depth data with the color image data. The target region is then determined based on the fused feature map. This solution improves the accuracy of target region determination by estimating the depth information corresponding to the first image using the color image, correcting the corresponding depth image, and then obtaining the target region based on the corrected depth image data and the color image data.

[0257] Figure 9 This is a flowchart illustrating a method for determining a target region according to an exemplary embodiment. Figure 9 As shown, the flowchart of the target area determination method in this embodiment of the application is as follows: Figure 9 The parts shown, 900, 910, and 920, together constitute the structure, wherein... Figure 9 The 900, 910, and 920 sections shown can be implemented in different devices or in the same device, such as... Figure 9 As shown, the method for determining the target area includes the following steps.

[0258] like Figure 9As shown in section 900, the training sample set may include a color image set 901 and a depth image set 903, wherein the color image set 901 contains at least two sample color images; the depth image set 903 contains at least two sample depth images; and the images in the color image set 901 and the depth image set 903 are in one-to-one correspondence. The color image processing branch 902 in the target region determination model processes each sample color image in the color image set to obtain the predicted region corresponding to each sample color image. The depth image processing branch 904 in the target region determination model processes each sample depth image in the depth image set to obtain the predicted region 905 corresponding to each sample depth image. The confidence score corresponding to each sample depth image can be obtained by comparing the predicted region 905 and the target region corresponding to each sample depth image. The sample depth images are then sorted from largest to smallest according to their confidence scores to obtain the sorted sample depth image set 906. The top a% of the confidence scores are determined as positive samples, and the bottom b% are determined as negative samples.

[0259] like Figure 9 As shown in section 910, in Figure 9 In part 910, there exists a confidence level discriminant model 912, which is based on... Figure 9 The confidence discrimination model is obtained by training the positive and negative samples in the sorted sample depth image set 906 in part 900. It can determine the confidence level of the input sample depth image. Therefore, for each sample depth image 911 in the training sample set, it is input into the confidence discrimination model 912 to obtain the confidence scores corresponding to each sample depth image in the training sample set and sort them to obtain sorted sample depth images 913. Then, the sample depth images with the top c% confidence scores in the sorted sample depth images 913 and their corresponding sample color images are used to train the depth estimation model 915 so that the trained depth estimation model 915 can process the input color image 914 to obtain the estimated depth data 916 corresponding to the color image 914. Then, the estimated depth data 916 and the depth image corresponding to the color image 914 are weighted based on the confidence scores of the depth images corresponding to the color images to obtain the corrected depth image 917. The confidence scores of the depth images corresponding to the color images can be obtained from the confidence discrimination model 912.

[0260] exist Figure 9In part 920, the color image set 921 contains color images, and the corrected depth image set 922 contains corrected depth images corresponding to each color image in the color image set 921. The color image corresponding to the first image in the color image set 921 is input into the color image processing branch of the target region determination model to obtain the color feature map corresponding to that color image. The depth image corresponding to the first image in the depth image set 922 is input into the depth image processing branch of the target region determination model to obtain the depth feature map corresponding to that depth image. Then, the data extracted from the N convolutional layers in the color image processing branch and the data extracted from the N convolutional layers in the depth image processing branch are respectively input as follows: Figure 7 The N cross-reference modules (CRMs) shown realize feature fusion between depth and color images to obtain a fused feature map. Based on the fused feature map, depth feature map, and color feature map, the target region of the first image is obtained.

[0261] like Figure 9 As shown, the goal is to select the most typical easy and difficult samples from the training database. These samples are then used to train a discriminator / classifier to predict the quality of the depth map, thus reflecting its reliability. Based on the selected representative positive and negative samples, a basic binary classifier is trained to evaluate the reliability of the depth map. Therefore, the trained discriminator can predict a reliability score for each data point, representing the probability that the depth map is a positive or negative example. Furthermore, this embodiment also establishes a depth estimator containing multiple convolutional operations. The depth estimator is trained using RGB images and high-quality depth data pairs to mitigate the inherent noise caused by the inaccuracy of the original depth data. In the depth calibration module, the potentially unreliable original depth map is not used directly; instead, a weighted sum of the original and estimated depth maps is used as input to improve the utilization of depth information.

[0262] Figure 10 This is a structural block diagram illustrating a target region determination device according to an exemplary embodiment. The target region determination device can realize the function of... Figure 2 or Figure 4 The method provided in the illustrated embodiment includes all or part of the steps, and the target area determination device includes:

[0263] The first image acquisition module 1001 is used to acquire a first image; the first image includes first color image data and first depth image data.

[0264] The depth estimation acquisition module 1002 is used to obtain first estimated depth data based on the first color image data; the first estimated depth data is used to indicate the depth information corresponding to the first color image data.

[0265] The calibration depth acquisition module 1003 is used to obtain first calibration depth data based on the first estimated depth data and the first depth image data;

[0266] The fusion feature acquisition module 1004 is used to perform weighted processing based on the first color image data and the first calibration depth data to obtain a first fusion feature map;

[0267] The target region determination module 1005 is used to determine the target region corresponding to the first image based on the first fused feature map.

[0268] In one possible implementation, the device further includes:

[0269] The confidence level acquisition module is used to acquire the confidence level corresponding to the first depth image data based on the first depth image data; the confidence level corresponding to the first depth image data is used to indicate the accuracy of the image data corresponding to the target region in the first depth image data;

[0270] The calibration depth acquisition module 1003 is also used for,

[0271] The first estimated depth data and the first depth image data are weighted based on the confidence level corresponding to the first depth image data to obtain the first calibrated depth data.

[0272] In one possible implementation, the confidence level acquisition module is further configured to:

[0273] Based on the first depth image data, a confidence discrimination model is used to process the data to obtain the confidence level corresponding to the first depth image data.

[0274] The estimated depth acquisition module 1002 is also used for,

[0275] Based on the first color image data, data processing is performed using a depth estimation model to obtain the first estimated depth data;

[0276] The confidence discrimination model is a machine learning model trained using the first sample image as a sample and the confidence type corresponding to the first sample image as a label.

[0277] The depth estimation model is a machine learning model trained using the second sample image as a sample and the depth image data corresponding to the second sample image as a label; the second sample image is a sample image whose confidence level meets the first specified condition.

[0278] In one possible implementation, the device further includes:

[0279] The first sample set acquisition module is used to acquire a first training sample set; the first training sample set includes a first sample image and the confidence type corresponding to the first sample image.

[0280] The first confidence probability acquisition module is used to process data based on the first sample image using the confidence discrimination model to obtain the confidence probability corresponding to the first sample image; the confidence probability is used to indicate the probability that the first sample image is a positive sample;

[0281] The confidence discrimination model training module is used to train the confidence discrimination model based on the confidence probability and the confidence type corresponding to the first sample image.

[0282] In one possible implementation, the first sample set acquisition module includes:

[0283] The second sample set acquisition submodule is used to acquire a second training sample set; the second training sample set includes training sample images and target regions corresponding to the training sample images; the training sample images include training color sample data and training depth sample data.

[0284] The color prediction region acquisition submodule is used to determine the color image processing branch in the model through the target region, process the training color sample data, and obtain the prediction region corresponding to the training color sample data.

[0285] The depth prediction region acquisition submodule is used to determine the depth image processing branch in the model through the target region, process the training depth sample data, and obtain the prediction region corresponding to the training depth sample data.

[0286] The confidence score acquisition submodule is used to determine the confidence score of the training sample image based on the prediction region corresponding to the training color sample data, the prediction region corresponding to the training depth sample data, and the target region corresponding to the training sample image.

[0287] The first sample image determination submodule is used to determine the confidence type of the training sample image and determine the training sample image as the first sample image in response to the confidence score of the training sample image meeting a specified condition.

[0288] The color image processing branch in the target region determination model is a machine learning model pre-trained using sample color images as samples and the target regions corresponding to the sample color images as annotations.

[0289] The depth image processing branch in the target region determination model is a machine learning model pre-trained using sample depth images as samples and the target regions corresponding to the sample depth images as annotations.

[0290] In one possible implementation, the confidence score includes a color confidence score and a depth confidence score;

[0291] The confidence score acquisition submodule includes:

[0292] The color confidence score acquisition unit is used to determine the color confidence score corresponding to the training sample image based on the overlap between the prediction region corresponding to the training color sample data and the target region corresponding to the training sample image.

[0293] The depth confidence score acquisition unit is used to determine the depth confidence score corresponding to the training sample image based on the degree of overlap between the predicted region corresponding to the training depth sample data and the target region corresponding to the training sample image.

[0294] In one possible implementation, the fusion feature acquisition module 1004 includes:

[0295] The attention weighting submodule is used to perform attention-based weighting processing on the feature fusion branch in the target region determination model based on the first color image data and the first calibration depth data to obtain the first fused feature map.

[0296] The device further includes:

[0297] The color feature map acquisition module is used to process data based on the first color image data and through the depth image processing branch in the target region determination model to obtain the first color feature map.

[0298] The depth feature map acquisition module is used to process data based on the first depth image data by determining the depth image processing branch in the target region model to obtain the first depth feature map.

[0299] The target area determination module 1005 is also used for,

[0300] Based on the first fused feature map, the first depth feature map, and the first color feature map, the target region corresponding to the first image is determined.

[0301] The target region determination model is a machine learning model trained using a third sample image as a sample and the target region corresponding to the third sample image as an annotation.

[0302] In one possible implementation, the feature fusion branch includes a first pooling layer, a second pooling layer, a first fully connected layer, and a second fully connected layer;

[0303] The fusion feature acquisition module 1004 includes:

[0304] The first pooling submodule is used to perform global pooling through the first pooling layer based on the first color image data to obtain the first color pooling data.

[0305] The first fully connected submodule is used to process the first color pooling data through the first fully connected layer to obtain the first color vector.

[0306] The second pooling submodule is used to perform global pooling through the second pooling layer based on the first depth image data to obtain the first depth pooled data.

[0307] The second fully connected submodule is used to process the data through the second fully connected layer based on the first deep pooling data to obtain the first depth vector.

[0308] The fusion feature acquisition submodule is used to obtain the first fusion feature map by performing channel attention weighting processing on the first color image data and the first calibration depth data through the first color vector and the first depth vector; the first color vector is used to indicate the weight corresponding to the first color image data; the first depth vector is used to indicate the weight corresponding to the first depth image data.

[0309] In one possible implementation, the device further includes:

[0310] The third image acquisition module is used to acquire a third sample image; the third sample image includes third sample color image data and third sample depth image data.

[0311] The third estimation data acquisition module is used to obtain the third sample estimated depth data based on the third sample color image data;

[0312] The third calibration data acquisition module is used to obtain third sample calibration depth data based on the third sample estimated depth data and the third sample depth image data.

[0313] The third fusion feature acquisition module is used to perform weighted processing on the feature fusion branch in the target region determination model based on the third sample color image data and the third sample calibration depth data to obtain the third sample fusion feature map.

[0314] The region determination model training module is used to train the target region determination model based on the third sample fusion feature map and the target region corresponding to the third sample image.

[0315] In one possible implementation, the device further includes:

[0316] The third color feature acquisition module is used to process data based on the third sample color image data and through the color image processing branch in the target region determination model to obtain the third sample color feature map.

[0317] The third depth feature acquisition module is used to process data based on the third sample depth image data and through the depth image processing branch in the target region determination model to obtain the third sample color feature map.

[0318] The region determination model training module is also used for,

[0319] The target region determination model is trained based on the third sample color feature map, the third sample depth feature map, the third sample fusion feature map, and the target region corresponding to the third sample image.

[0320] In summary, the solution presented in this application obtains estimated depth data corresponding to the first image from the first color image data in the first image, then corrects the first depth image data corresponding to the first image based on the estimated depth data to obtain calibrated depth data, then fuses the calibrated depth data with the color image data, and determines the target region based on the fused feature map. This solution improves the accuracy of target region determination by estimating the depth information corresponding to the first image from the color image, correcting the corresponding depth image, and then obtaining the target region corresponding to the first image based on the corrected depth image data and the color image data.

[0321] Figure 11 This is a structural block diagram illustrating a target region determination device according to an exemplary embodiment. The target region determination device can realize the function of... Figure 2 or Figure 4 The method provided in the illustrated embodiment includes all or part of the steps, and the target area determination device includes:

[0322] The third sample image acquisition module 1101 is used to acquire a third sample image; the third sample image includes third sample color image data and third sample depth image data.

[0323] The third sample estimation acquisition module 1102 is used to obtain third sample estimated depth data based on the third sample color image data; the third sample estimated depth data is used to indicate the depth information corresponding to the third sample color image data.

[0324] The third sample calibration acquisition module 1103 is used to obtain third sample calibration depth data based on the third sample estimated depth data and the third sample depth image data.

[0325] The third sample fusion feature acquisition module 1104 is used to obtain the third sample fusion feature map by performing weighted processing on the feature fusion branch in the target region determination model based on the third sample color image data and the third sample calibration depth data.

[0326] The region determination model training module 1105 is used to train the target region determination model based on the third sample fusion feature map and the target region corresponding to the third sample image.

[0327] The trained target region determination model is used to process the color image data and calibration depth data corresponding to the first image to obtain the target region corresponding to the first image.

[0328] In summary, the solution presented in this application obtains estimated depth data corresponding to the first image from the first color image data in the first image, then corrects the first depth image data corresponding to the first image based on the estimated depth data to obtain calibrated depth data, then fuses the calibrated depth data with the color image data, and determines the target region based on the fused feature map. This solution improves the accuracy of target region determination by estimating the depth information corresponding to the first image from the color image, correcting the corresponding depth image, and then obtaining the target region corresponding to the first image based on the corrected depth image data and the color image data.

[0329] Figure 12This is a schematic diagram illustrating the structure of a computer device according to an exemplary embodiment. The computer device can be implemented as a model processing device and / or a text-image matching device in the various method embodiments described above. The computer device 1200 includes a central processing unit (CPU) 1201, a system memory 1204 including random access memory (RAM) 1202 and read-only memory (ROM) 1203, and a system bus 1205 connecting the system memory 1204 and the central processing unit 1201. The computer device 1200 also includes a basic input / output system 1206 to facilitate information transfer between various devices within the computer, and a mass storage device 1207 for storing an operating system 1213, application programs 1214, and other program modules 1215.

[0330] The mass storage device 1207 is connected to the central processing unit 1201 via a mass storage controller (not shown) connected to the system bus 1205. The mass storage device 1207 and its associated computer-readable media provide non-volatile storage for the computer device 1200. That is, the mass storage device 1207 may include computer-readable media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0331] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM, or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1204 and the mass storage device 1207 described above can be collectively referred to as memory.

[0332] Computer device 1200 can be connected to the Internet or other network devices via network interface unit 1211 connected to the system bus 1205.

[0333] The memory also includes one or more programs, which are stored in the memory, and the central processing unit 1201 implements these programs by executing them. Figure 2 , Figure 3or Figure 4 All or part of the steps of the method shown.

[0334] In exemplary embodiments, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including a computer program (instructions) that can be executed by a processor of a computer device to perform the methods shown in the various embodiments of this application. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0335] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods shown in the various embodiments described above.

[0336] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0337] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for determining a target region, characterized in that, The method includes: Acquire a first image; the first image contains first color image data and first depth image data; Based on the first depth image data, the confidence level corresponding to the first depth image data is obtained; the confidence level corresponding to the first depth image data is used to indicate the accuracy of the image data corresponding to the target region in the first depth image data. Based on the first color image data, first estimated depth data is obtained; the first estimated depth data is used to indicate the depth information corresponding to the first color image data. The first estimated depth data and the first depth image data are weighted based on the confidence level corresponding to the first depth image data to obtain the first calibrated depth data. Based on the first color image data and the first calibration depth data, a weighted processing is performed to obtain the first fused feature map; Based on the first fused feature map, the target region corresponding to the first image is determined.

2. The method according to claim 1, characterized in that, The step of obtaining the confidence level corresponding to the first depth image data based on the first depth image data includes: Based on the first depth image data, a confidence discrimination model is used to process the data to obtain the confidence level corresponding to the first depth image data. The step of obtaining the first estimated depth data based on the first color image data includes: Based on the first color image data, data processing is performed using a depth estimation model to obtain the first estimated depth data; The confidence discrimination model is a machine learning model trained using the first sample image as a sample and the confidence type corresponding to the first sample image as a label. The depth estimation model is a machine learning model trained using the second sample image as a sample and the depth image data corresponding to the second sample image as a label; the second sample image is a sample image whose confidence level meets the first specified condition.

3. The method according to claim 2, characterized in that, The method further includes: Obtain a first training sample set; the first training sample set includes a first sample image and the confidence type corresponding to the first sample image; Based on the first sample image, data processing is performed using the confidence discrimination model to obtain the confidence probability corresponding to the first sample image; the confidence probability is used to indicate the probability that the first sample image is a positive sample; The confidence discrimination model is trained based on the confidence probability and confidence type corresponding to the first sample image.

4. The method according to claim 3, characterized in that, The process of obtaining the first training sample set includes: Obtain a second training sample set; the second training sample set includes training sample images and target regions corresponding to the training sample images; the training sample images include training color sample data and training depth sample data. The color image processing branch in the model is determined by the target region, and the training color sample data is processed to obtain the prediction region corresponding to the training color sample data. The depth image processing branch in the model is determined by the target region, and the training depth sample data is processed to obtain the prediction region corresponding to the training depth sample data. Based on the predicted region corresponding to the training color sample data, the predicted region corresponding to the training depth sample data, and the target region corresponding to the training sample image, the confidence score of the training sample image is determined. In response to the confidence score of the training sample image satisfying a specified condition, the confidence type of the training sample image is determined, and the training sample image is identified as the first sample image; The color image processing branch in the target region determination model is a machine learning model pre-trained using sample color images as samples and the target regions corresponding to the sample color images as annotations. The depth image processing branch in the target region determination model is a machine learning model pre-trained using sample depth images as samples and the target regions corresponding to the sample depth images as annotations.

5. The method according to claim 4, characterized in that, The confidence scores include color confidence scores and depth confidence scores; The step of determining the confidence score of the training sample image based on the prediction region corresponding to the training color sample data, the prediction region corresponding to the training depth sample data, and the target region corresponding to the training sample image includes: Based on the overlap between the predicted region corresponding to the training color sample data and the target region corresponding to the training sample image, the color confidence score corresponding to the training sample image is determined. Based on the overlap between the predicted region corresponding to the training depth sample data and the target region corresponding to the training sample image, the depth confidence score corresponding to the training sample image is determined.

6. The method according to claim 1, characterized in that, The step of performing weighted processing based on the first color image data and the first calibration depth data to obtain the first fused feature map includes: Based on the first color image data and the first calibration depth data, the feature fusion branch in the target region is determined and weighted based on an attention mechanism to obtain the first fused feature map. Before determining the target region corresponding to the first image based on the first fused feature map, the method further includes: Based on the first color image data, data processing is performed through the depth image processing branch in the target region determination model to obtain the first color feature map; Based on the first depth image data, the depth image processing branch in the target region determination model is used to process the data to obtain the first depth feature map. The step of determining the target region corresponding to the first image based on the first fused feature map includes: Based on the first fused feature map, the first depth feature map, and the first color feature map, the target region corresponding to the first image is determined. The target region determination model is a machine learning model trained using a third sample image as a sample and the target region corresponding to the third sample image as an annotation.

7. The method according to claim 6, characterized in that, The feature fusion branch includes a first pooling layer, a second pooling layer, a first fully connected layer, and a second fully connected layer; The step of obtaining the first fused feature map by performing attention-based weighted processing on the feature fusion branch of the target region determination model based on the first color image data and the first calibration depth data includes: Based on the first color image data, global pooling is performed through the first pooling layer to obtain the first color pooling data; Based on the first color pooling data, the data is processed through the first fully connected layer to obtain the first color vector; Based on the first depth image data, global pooling is performed through the second pooling layer to obtain the first depth pooled data; Based on the first deep pooled data, the data is processed through the second fully connected layer to obtain the first depth vector; Based on the first color image data and the first calibration depth data, channel attention weighting is performed using the first color vector and the first depth vector to obtain the first fused feature map; the first color vector is used to indicate the weight corresponding to the first color image data; the first depth vector is used to indicate the weight corresponding to the first depth image data.

8. The method according to claim 6, characterized in that, The method further includes: Acquire a third sample image; the third sample image includes third sample color image data and third sample depth image data; Based on the third sample color image data, the estimated depth data of the third sample is obtained; Based on the estimated depth data of the third sample and the depth image data of the third sample, the calibrated depth data of the third sample is obtained; Based on the third sample color image data and the third sample calibration depth data, the feature fusion branch in the target region determination model is weighted and processed to obtain the third sample fusion feature map. Based on the third sample fusion feature map and the target region corresponding to the third sample image, the target region determination model is trained.

9. The method according to claim 8, characterized in that, Before training the target region determination model based on the third sample fusion feature map and the target region corresponding to the third sample image, the method further includes: Based on the third sample color image data, data processing is performed through the color image processing branch in the target region determination model to obtain the third sample color feature map. Based on the third sample depth image data, data processing is performed through the depth image processing branch in the target region determination model to obtain the third sample color feature map. The step of training the target region determination model based on the fused feature map of the third sample and the target region corresponding to the third sample image includes: The target region determination model is trained based on the third sample color feature map, the third sample depth feature map, the third sample fusion feature map, and the target region corresponding to the third sample image.

10. A method for determining a target region, characterized in that, The method includes: Acquire a third sample image; the third sample image includes third sample color image data and third sample depth image data; Based on the third sample color image data, third sample estimated depth data is obtained; the third sample estimated depth data is used to indicate the depth information corresponding to the third sample color image data. Based on the estimated depth data of the third sample and the depth image data of the third sample, the calibrated depth data of the third sample is obtained; Based on the third sample color image data and the third sample calibration depth data, the feature fusion branch in the target region determination model is weighted and processed to obtain the third sample fusion feature map. Based on the third sample fusion feature map and the target region corresponding to the third sample image, the target region determination model is trained; The trained target region determination model processes the first color image data and the first calibration depth data contained in the first image to obtain the target region corresponding to the first image. The first calibration depth data is obtained by acquiring the confidence level corresponding to the first depth image data based on the first depth image data contained in the first image, and obtaining the first estimated depth data based on the first color image data. The first estimated depth data and the first depth image data are then weighted based on the confidence level corresponding to the first depth image data. The confidence level corresponding to the first depth image data is used to indicate the accuracy of the image data corresponding to the target region in the first depth image data. The first estimated depth data is used to indicate the depth information corresponding to the first color image data.

11. A target area determination device, characterized in that, The device includes: A first image acquisition module is used to acquire a first image; the first image includes first color image data and first depth image data. The confidence level acquisition module is used to acquire the confidence level corresponding to the first depth image data based on the first depth image data; the confidence level corresponding to the first depth image data is used to indicate the accuracy of the image data corresponding to the target region in the first depth image data; The depth estimation module is used to obtain first estimated depth data based on the first color image data; the first estimated depth data is used to indicate the depth information corresponding to the first color image data. The calibration depth acquisition module is used to perform weighted processing on the first estimated depth data and the first depth image data based on the confidence level corresponding to the first depth image data to obtain the first calibration depth data. The fusion feature acquisition module is used to perform weighted processing based on the first color image data and the first calibration depth data to obtain a first fusion feature map; The target region determination module is used to determine the target region corresponding to the first image based on the first fused feature map.

12. A target area determination device, characterized in that, The device includes: The third sample image acquisition module is used to acquire a third sample image; the third sample image includes third sample color image data and third sample depth image data. The third sample estimation acquisition module is used to obtain third sample estimated depth data based on the third sample color image data; the third sample estimated depth data is used to indicate the depth information corresponding to the third sample color image data. The third sample calibration acquisition module is used to obtain third sample calibration depth data based on the third sample estimated depth data and the third sample depth image data. The third sample fusion feature acquisition module is used to obtain the third sample fusion feature map by performing weighted processing on the feature fusion branch in the target region determination model based on the third sample color image data and the third sample calibration depth data. The region determination model training module is used to train the target region determination model based on the third sample fusion feature map and the target region corresponding to the third sample image. The trained target region determination model processes the first color image data and the first calibration depth data contained in the first image to obtain the target region corresponding to the first image. The first calibration depth data is obtained by acquiring the confidence level corresponding to the first depth image data based on the first depth image data contained in the first image, and obtaining the first estimated depth data based on the first color image data. The first estimated depth data and the first depth image data are then weighted based on the confidence level corresponding to the first depth image data. The confidence level corresponding to the first depth image data is used to indicate the accuracy of the image data corresponding to the target region in the first depth image data. The first estimated depth data is used to indicate the depth information corresponding to the first color image data.

13. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, code set, or instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the target region determination method as described in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or instruction set is loaded and executed by a processor to implement the target region determination method as described in any one of claims 1 to 10.

15. A computer program product, characterized in that, The computer program product includes computer instructions that are executed by a processor of a computer device to implement the target region determination method as described in any one of claims 1 to 10.

Images