Target detection model training method and related device

By combining information from training images and depth images, and utilizing label information and depth images to supervise boundary response and depth response information, the problem of low target detection accuracy in existing technologies is solved, achieving higher detection accuracy and robustness.

CN118982698APending Publication Date: 2024-11-19TENCENT TECHNOLOGY (SHENZHEN) CO LTD +1

Patent Information

Application Number
CN202310531878.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-11
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Existing salient object detection algorithms and datasets primarily use visible light camera images, resulting in low object detection accuracy.

Method used

The model is trained using paired training images and depth images. Label information is used to supervise boundary response information, and depth image is used to supervise depth response information. The boundary response and depth response information are used to assist in result prediction. The complementarity of high-order and low-order features is fully utilized to enhance the depth information extraction capability of the object detection model.

Benefits of technology

It improves the accuracy and robustness of the object detection model, especially in complex scenes, and can maintain a high object recognition effect without the need for additional depth image feature extraction operations, which helps to miniaturize the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982698A_ABST
    Figure CN118982698A_ABST
Patent Text Reader

Abstract

The invention discloses a target detection model training method and a related device, and relates to the fields of artificial intelligence, computer vision technology, machine learning and the like. Specifically, an initial model can be trained through a training image to obtain a target detection model, the initial model can be utilized to perform feature extraction on the training image to obtain low-order features and high-order features, and boundary extraction can be performed on the low-order features to obtain boundary response information. And depth estimation can be carried out on the high-order features to obtain depth response information, so that in the initial model training process, label information is utilized to supervise boundary response information, and a depth image is utilized to supervise the depth response information, thereby obtaining accurate boundary response information and depth response information for assisting target detection. The complementarity of the high-order features and the low-order features is fully utilized, the depth information extraction capability of the target detection model is enhanced in combination with the depth image, the data expression potential of different modalities of the image is mined, and the performance of the target detection model is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a target detection model method and related devices. Background Art

[0002] Salient object detection (also known as foreground segmentation), as an emerging image processing technology, is one of the important research directions in the field of computer vision and can be widely used in scenarios such as self-driving cars, medical aided judgment, and robot environmental perception. In recent years, driven by the availability of large-scale datasets and the rapid development of deep convolutional networks and segmentation models, salient object detection has made significant progress. However, current algorithms and most datasets use images generated by visible light cameras as model inputs to perform salient object detection on the input images, but there is a problem of low object detection accuracy. Summary of the invention

[0003] In order to solve the above technical problems, the present application provides a training method and related devices for a target detection model, which uses paired training images and depth images to train the model, uses label information to supervise boundary response information, uses depth images to supervise depth response information, and uses boundary response information and depth response information to assist result prediction, which is beneficial to improving the target detection accuracy of the target detection model.

[0004] The embodiments of the present application disclose the following technical solutions:

[0005] On the one hand, the present application provides a method for training a target detection model, the method comprising:

[0006] Acquire a pair of training images and depth images, wherein the training images have label information for indicating a target object in the training images;

[0007] Using the initial model to extract features from the training image to obtain low-order features and high-order features with a higher hierarchy than the low-order features;

[0008] Performing boundary extraction on the low-order features to obtain boundary response information, and performing depth estimation on the high-order features to obtain depth response information;

[0009] Predicting a prediction result of the training image according to the low-order features, the high-order features, the boundary response information and the depth response information;

[0010] A prediction loss function is obtained by performing loss calculation based on the prediction result and the label information, a boundary loss function is obtained by performing loss calculation based on the boundary information of the label information and the boundary response information, and a depth loss function is obtained by performing loss calculation based on the depth image and the depth response information;

[0011] The initial model is trained according to the boundary loss function, the depth loss function and the prediction loss function, so that the initial model is converted into a target detection model, and the target detection model is used to perform target detection on the image to be detected.

[0012] On the other hand, the present application provides a training device for a target detection model, the device comprising:

[0013] A training data acquisition unit, used to acquire a pair of training images and depth images, wherein the training images have label information for indicating a target object in the training images;

[0014] A feature extraction unit, configured to extract features from the training image using an initial model to obtain low-order features and high-order features with a higher hierarchy than the low-order features;

[0015] A feature processing unit, configured to extract boundaries of the low-order features to obtain boundary response information, and to estimate depth of the high-order features to obtain depth response information;

[0016] A prediction unit, configured to predict a prediction result of the training image according to the low-order features, the high-order features, the boundary response information and the depth response information;

[0017] A loss function determination unit, configured to perform loss calculation according to the prediction result and the label information to obtain a prediction loss function, perform loss calculation according to the boundary information of the label information and the boundary response information to obtain a boundary loss function, and perform loss calculation according to the depth image and the depth response information to obtain a depth loss function;

[0018] A training unit is used to train the initial model according to the boundary loss function, the depth loss function and the prediction loss function, so that the initial model is converted into a target detection model, and the target detection model is used to perform target detection on the image to be detected.

[0019] In another aspect, the present application provides a computer device, the device comprising a processor and a memory:

[0020] The memory is used to store a computer program and transmit the computer program to the processor;

[0021] The processor is used to execute the target detection model training method described in the above aspects according to the instructions in the computer program.

[0022] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the training method of the target detection model described in the above aspects.

[0023] On the other hand, an embodiment of the present application provides a computer program product including a computer program, which, when executed on a computer device, enables the computer device to execute the training method of the target detection model.

[0024] It can be seen from the above technical solution that a pair of training images and depth images can be obtained as training data, and the initial model can be trained to obtain a target detection model. The training image can have label information for indicating the target object in the training image. The initial model can be used to extract features from the training image to obtain low-order features and high-order features with a higher hierarchy than the low-order features. The low-order features can be subjected to boundary extraction to obtain boundary response information. The high-order features can be subjected to depth estimation to obtain depth response information. The prediction results of the training image can be predicted based on the low-order features, high-order features, boundary response information and depth response information. In this way, the boundary response information can assist the prediction process of the results, and the detailed characterization of the prediction results can be enhanced through boundary constraints. The depth response information can also assist the prediction process of the results, and the contrast of the prediction results can be enhanced through depth constraints.

[0025] Afterwards, the prediction loss function can be obtained by performing loss calculation based on the prediction result and label information, the boundary loss function can be obtained by performing loss calculation based on the boundary information and boundary response information of the label information, and the depth loss function can be obtained by performing loss calculation based on the depth image and depth response information. The initial model is trained according to the boundary loss function, the depth loss function and the prediction loss function, so that the initial model is converted into a target detection model, so that the obtained target detection model can be used for target detection of the image to be detected. In this way, by training the initial model, the prediction result is closer to the label information, the depth response information is closer to the depth image, and the boundary response information is closer to the boundary information of the label information. In the initial model training process, the label information is used to supervise the boundary response information, and the depth image is used to supervise the depth response information, which improves the ability to accurately extract the boundary response information and the depth response information, and obtains accurate boundary corresponding information and depth corresponding information for auxiliary target detection, further improving the auxiliary role of boundary constraints and depth constraints on the prediction results, making full use of the complementarity of high-order features and low-order features, and by combining the depth image to enhance the depth information extraction ability of the target detection model, the data performance potential of different image modalities is excavated, and the performance of the target detection model is improved. In addition, there is no need to perform operations such as feature extraction on the depth image, which is conducive to the miniaturization of the target detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A schematic diagram of an application scenario of a method for training a target detection model provided in an embodiment of the present application;

[0028] Figure 2 A flowchart of a method for training a target detection model provided in an embodiment of the present application;

[0029] Figure 3 A schematic diagram of training data provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of the data processing process of an initial model provided in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of the overall structure of an initial model provided in an embodiment of the present application;

[0032] Figure 6 A schematic diagram of a high-level initial feature processing process provided in an embodiment of the present application;

[0033] Figure 7 A schematic diagram of a data processing process of a global perception module provided in an embodiment of the present application;

[0034] Figure 8 A schematic diagram of response characteristics provided in an embodiment of the present application;

[0035] Fig. 9 A schematic diagram of the working process of a knowledge collector provided in an embodiment of the present application;

[0036] Fig.10 A structural block diagram of a training device for a target detection model provided in an embodiment of the present application;

[0037] Fig.11 A structural diagram of a terminal device provided in an embodiment of the present application;

[0038] Fig.12 A structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The embodiments of the present application are described below in conjunction with the accompanying drawings.

[0040] Current algorithms and most datasets use images produced by visible light cameras as model input and perform salient object detection on the input images. However, there is a problem of low object detection accuracy.

[0041] In order to solve the above technical problems, the embodiments of the present application provide a training method and related devices for a target detection model, which use paired training images and depth images for model training, use label information to supervise boundary response information, use depth images to supervise depth response information, and use boundary response information and depth response information to assist result prediction, thereby making full use of the complementarity of high-order features and low-order features, and by combining depth images to enhance the depth information extraction capability of the target detection model, tapping the data expression potential of different image modalities, and helping to improve the target detection accuracy of the target detection model.

[0042] A method for training a target detection model provided in an embodiment of the present application can be implemented by a computer device, which can be a terminal device or a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Terminal devices include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0043] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0044] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0045] The solution provided in the embodiment of the present application involves technologies such as computer vision technology (Computer Vision, CV) of artificial intelligence. Computer vision is a science that studies how to make machines "see". To put it more concretely, it refers to machine vision such as using cameras and computers to replace human eyes to identify, monitor and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0046] The solution provided in the embodiment of the present application involves technologies such as machine learning of artificial intelligence. Machine learning (ML) is a multi-field interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning generally include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0047] Autonomous driving technology usually includes high-precision maps, environmental perception, behavioral decision-making, path planning, motion control and other technologies. Autonomous driving technology has broad application prospects.

[0048] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0049] In order to facilitate the understanding of the technical solution provided by the present application, the following will introduce a training method for a target detection model provided by an embodiment of the present application in combination with an actual application scenario.

[0050] Figure 1A schematic diagram of an application scenario of a method for training a target detection model provided in an embodiment of the present application is shown. In the scenario, a server 10 is included. The server 10 can be used as the aforementioned computer device to implement the method for training the target detection model.

[0051] The server 10 obtains a pair of training images and depth images as training data, trains the initial model to obtain a target detection model, and the training image may have label information for indicating the target object in the training image. The server 10 may use the initial model to extract features from the training image to obtain low-order features and high-order features with a higher hierarchy than the low-order features, perform boundary extraction on the low-order features to obtain boundary response information, perform depth estimation on the high-order features to obtain depth response information, and predict the prediction results of the training image based on the low-order features, high-order features, boundary response information, and depth response information, so that the boundary response information can assist the prediction process of the results, and enhance the detailed characterization of the prediction results through boundary constraints, and the depth response information can also assist the prediction process of the results, and enhance the contrast of the prediction results through depth constraints.

[0052] Afterwards, the server can perform loss calculation based on the prediction result and label information to obtain a prediction loss function, perform loss calculation based on the boundary information and boundary response information of the label information to obtain a boundary loss function, perform loss calculation based on the depth image and depth response information to obtain a depth loss function, and train the initial model based on the boundary loss function, the depth loss function, and the prediction loss function to convert the initial model into a target detection model, so that the obtained target detection model can be used for target detection in the image to be detected.

[0053] In this way, by training the initial model, the prediction results are closer to the label information, the depth response information is closer to the depth image, and the boundary response information is closer to the boundary information of the label information. In the initial model training process, the label information is used to supervise the boundary response information, and the depth image is used to supervise the depth response information, which improves the ability to accurately extract the boundary response information and the depth response information, and obtains accurate boundary response information and depth response information for auxiliary target detection, further improving the auxiliary role of boundary constraints and depth constraints on the prediction results, making full use of the complementarity of high-order features and low-order features, and by combining the depth image to enhance the depth information extraction ability of the target detection model, the data performance potential of different image modalities is explored to improve the performance of the target detection model. In addition, there is no need to perform feature extraction and other operations on the depth image, which is conducive to the miniaturization of the target detection model.

[0054] Figure 2A flowchart of a method for training a target detection model provided in an embodiment of the present application. In this embodiment, a server is used as the aforementioned computer device for illustration. The method for training the target detection model may include:

[0055] S101, obtaining a pair of training images and depth images, where the training images have label information.

[0056] In the embodiment of the present application, a training image can be used as training data of the model. The training image has a target object and a background of the target object. The target object is used as a foreground. Identifying the target object from the training image can be called target detection or foreground segmentation. The training image can be a color image (RGB Image). The color image has three channel features of RGB, which can provide the color, texture and context information of the object. Of course, the training image can also be a grayscale image. The grayscale image has a channel feature, which can provide the texture and context information of the object.

[0057] The training image may have label information for indicating the target object in the training image, that is, the true value annotation for the training image recognition. The label information may be reflected in the form of a graph as the gold standard (Mask or Ground-truth) in model training. Each training image may correspond to a label information, thereby providing a training instance for model training.

[0058] Traditional models based on training images as training data, such as convolutional neural networks (CNNs) and fully convolutional networks (FCNs), can perform feature processing on the images to be detected to achieve target detection in the images to be detected. However, when the amount of information in the images to be detected is small, the feature expression is insufficient, which affects the target detection performance of these models. For example, the images to be detected acquired under low light conditions have similar foregrounds and backgrounds, and there may be noisy environments. The target detection accuracy of such images to be detected is low. In other words, the target detection performance of the model trained only based on training images cannot meet actual needs.

[0059] In the embodiment of the present application, a depth image can be added to the training data, that is, a pair of training images and depth images are used as training data, and the training images and depth images correspond one to one. The depth image can provide additional depth information for the training image, so that the depth image can be used as complementary modal information of the training image. The depth information can reflect the three-dimensional spatial information of the scene corresponding to the image, for example, it can reflect the shape, size and position of the object in the image. The depth image can be obtained by shooting with a depth camera. The depth camera can use two or more cameras to shoot the same scene separately, and calculate the depth values ​​of different points in the scene through the parallax information between different cameras. In this way, the distance information of different objects or surfaces in the scene can be effectively obtained, thereby realizing the three-dimensional perception of the scene.

[0060] In specific implementation, the paired training images and depth images can be obtained by preprocessing the initial data. The preprocessing operation may include a normalization operation of subtracting the mean and dividing by the standard deviation, and may also include data expansion operations such as random data enhancement. The initial data is subjected to random data enhancement to obtain enhanced data corresponding to the initial data. The enhanced data and the initial data are used together as training data to expand the training data set and avoid potential overfitting. The random data enhancement may include one or more operations such as rotation, flipping or cropping. The resolution of the training image may be 256×256 pixels.

[0061] The initial data can be obtained through the public RGB-D salient object detection dataset DUTLF-Depth, which provides 1200 color image-depth image pairs captured in different scenes and corresponding fine annotations. The images in the dataset are divided into initial data for model training and test data for performance testing, where the data volume of training data and test data can be 800 and 400 respectively. Figure 3 FIG. 1 is a schematic diagram of a training data provided by an embodiment of the present application, which includes a training image, a depth image and label information of the training image, wherein Figure 3 A. Figure 3 D and Figure 3 G is three training images, Figure 3 B. Figure 3 E and Figure 3 H is the depth image corresponding to the three training images mentioned above. Figure 3 C. Figure 3 F and Figure 3 I are the label information corresponding to the three training images mentioned above.

[0062] S102, using the initial model to extract features from the training image to obtain low-order features and high-order features with a higher hierarchy than the low-order features.

[0063] In an embodiment of the present application, the initial model is a model to be trained, which has the same data processing flow as the target detection model and has low target detection accuracy. Therefore, it is necessary to improve the target detection accuracy of the initial model through model training.

[0064] At present, direct fusion of training images and depth images is performed through feature addition, feature cascade, result addition fusion, etc. However, these methods require the model to automatically integrate the complementarity between different modalities, and do not fully utilize the respective characteristics of multimodal data. There is still a lot of room for feature fusion between different modalities that is worth exploring. The embodiments of the present application achieve full interaction of cross-modal features by exploring the characteristics of different modalities, and explore the complementarity between low-order features and high-order features, so that the model can obtain ideal segmentation performance, improve the robustness of the model in complex scenes, and perform well, even in complex scenes such as target objects and backgrounds are similar and the environment is noisy. It can also have a high target recognition effect.

[0065] In an embodiment of the present application, features of the training image can be extracted through the feature extraction module in the initial model, and the feature extraction network (Feature Extraction Network) can include a backbone convolutional network, which does not classify the extracted features, but transmits the extracted features to other modules for other feature processing operations.

[0066] The backbone convolutional network can include convolutional layers, activation layers, pooling layers, etc. Each intermediate layer in the convolutional layer receives the output of the previous layer as its input. The starting layer of the backbone convolutional network is the input layer, which is directly connected to the training image. The number of its neurons is equal to the number of pixels of the training image. The convolutional layer of the backbone convolutional network convolves a certain number of filters (usually called convolution kernels) with the input data and extracts features as a result of its execution. The size of the filter is arbitrary and can be defined by the designer. Each convolution kernel only responds to a specific area of ​​the previous layer, which is called the receptive field (or receptive field). The output of each convolutional layer is considered to be an activation map, which emphasizes the effect of applying a specific filter to the input data. The activation layer is usually connected after the convolutional layer to apply nonlinearity to the activation layer. The activation layer can be connected after the pooling layer, which helps to reduce the dimensionality of the convolution output. During the training process of the initial model, the weights of the neural connections and the weights of the kernels in the backbone convolutional network are continuously optimized. As an example, the backbone convolutional network can be ResNet-50, in which the last fully connected layer is discarded to better adapt to the pixel-level object detection task.

[0067] The features obtained by feature extraction of the training image can be used as feature representation (feature map). After the training image passes through more layers of convolutional layers for feature extraction, the features obtained have a higher level and are more abstract. Therefore, low-order features and high-order features with higher levels than low-order features can be determined based on the output features of different convolutional layers. The high-order features and low-order features here are relative concepts. As long as the level of the high-order features is higher than that of the low-order features, there is no actual level limitation. Low-order features can also be called shallow features, and high-order features can also be called deep features.

[0068] Specifically, refer to Figure 4 As shown, it is a schematic diagram of the data processing process of an initial model provided in an embodiment of the present application, wherein S102 may include: S1021, extracting features of a training image through an initial model to obtain multiple initial features of different levels; S1022, determining low-order initial features and high-order initial features from multiple initial features according to preset conditions; S1023, performing feature processing on the low-order initial features to obtain low-order features; S1024, performing feature processing on the high-order initial features to obtain high-order features. In other words, low-order features and high-order features may be divided based on preset conditions. For example, initial features with a level higher than the preset level may be taken as high-order initial features, and initial features with a level lower than or equal to the preset level may be taken as low-order initial features. Wherein the low-order initial features have higher pixels and more details, and the high-order initial features have lower pixels and abstract features. Based on the characteristics of the two, data processing may be performed in a targeted manner later.

[0069] In S1023, in the process of performing feature processing on the low-order initial features to obtain low-order features, if the number of low-order initial features is one, the low-order initial feature can be used as the low-order feature, and if the number of low-order initial features is multiple, the multiple low-order initial features can be aggregated and convolved to obtain low-order features. The aggregation operation can be a concatenation operation. Before performing feature processing on the low-order initial features to obtain low-order features, the following can also be performed: S1023A, if the number of low-order initial features is multiple, based on a preset number of channels, the multiple low-order initial features can be converted respectively to change the number of channels of each low-order initial feature, and each low-order initial feature is converted to the preset number of channels. In the process of changing the number of channels, the size of the low-order initial features also changes accordingly, thereby unifying the various low-order initial features, which is beneficial to the subsequent aggregation operation of each low-order initial feature. The conversion operation can be implemented through a transition layer, which corresponds one-to-one to the low-order initial features. The transition layer can be a convolutional layer, for example, a 64-channel 1×1 convolutional layer, that is, the preset number of channels can be 64, and the 1×1 convolutional layer can implement point-by-point convolution to change the number of channels of the low-order initial features.

[0070] refer to Figure 5 As shown, it is a schematic diagram of the overall structure of an initial model provided in an embodiment of the present application. Multiple initial features have different levels, and the levels correspond from low to high: {f 1 ,f 2 ,f 3 ,f 4 ,f 5}, where the number of low-order initial features is 2, and the corresponding conversion layers are represented as trans1 and trans2. The multiple low-order initial features after the conversion operation are respectively recorded as f 1 ,f 2 , the low-order features obtained according to the low-order initial features are recorded as f l , the convolution kernel of the convolution operation is 3×3, denoted as CONV 3×3 .

[0071] In S1024, in the process of performing feature processing on the high-order initial features to obtain high-order features, if the number of high-order initial features is one, the high-order initial feature can be used as the high-order feature, and if the number of high-order initial features is multiple, feature fusion of multiple high-order initial features can be performed to make the high-order features have rich global semantics and reduce the loss of important information in the subsequent processing process. Figure 4As shown, the feature fusion process of multiple high-order initial features can be specifically as follows: S1024A, for the target initial feature among the multiple high-order initial features, according to other initial features in the multiple high-level initial features that are higher in hierarchy than the target initial feature, determine the additional features of the target initial feature, and the additional features reflect the information of other initial features other than the target initial feature; S1024B, integrate the target initial feature and the additional features to obtain the comprehensive feature corresponding to the target initial feature, so the comprehensive feature contains the information of the target initial feature and the other initial features; S1024C, perform feature processing on the comprehensive features corresponding to the multiple high-order initial features to obtain the high-order feature, because the high-level features have more abstract information and can better reflect the position of the target object, by integrating the additional features for the target initial feature, the information of the position of the target object can be emphasized in the target initial feature, so based on the global induced enhancement strategy, the high-order feature has richer global semantics, which is conducive to reducing the loss of important information.

[0072] Determining the additional features of the target initial features and the comprehensive features corresponding to the target initial features can be called the global guidance process, which is implemented by the Global Guidance Module (GGM). The global guidance module is applied to the high-order initial features to achieve contextual semantic enhancement of the high-order initial features.

[0073] For multiple high-order initial features, additional features can be determined in the form of cascade. Specifically, the number of multiple high-order initial features is recorded as n, and the multiple high-order initial features are sorted from high to low according to the hierarchy, that is, the highest hierarchy is the first high-order initial feature, and the additional features of the first high-order initial feature are determined to be empty. Then, based on the additional features of the previous i-1th high-order initial feature and the i-th high-order initial feature, the additional features of the i-th high-order initial feature are determined, where i is less than or equal to n. In this way, the information of other initial features with a hierarchy higher than the i-th high-order initial feature can be attached to the bottom i high-order initial features to obtain their corresponding comprehensive features. Since the higher-order features have more abstract features, it is easier to determine the area where the target object is located. The additional features determined in the form of cascade enable the features of higher hierarchies to be retained in each feature, reducing the loss of high-level features.

[0074] If there are multiple high-order initial features, before determining the additional features of the target initial features among the multiple high-order initial features, S102 may also include S1024D, performing conversion operations on the multiple high-order initial features based on the preset number of channels to change the number of channels of each high-order initial feature, and converting each high-order initial feature to the preset number of channels. In the process of changing the number of channels, the size of the high-order initial features also changes accordingly, thereby unifying the various low-order initial features, which is beneficial to the subsequent operation of each high-order initial feature. The conversion operation can be implemented by a conversion layer, and the conversion layer corresponds to the high-order initial feature one by one. The conversion layer can be a convolution layer, such as a 64-channel 1×1 convolution layer, that is, the preset number of channels can be 64, and the 1×1 convolution layer can implement point-by-point convolution to change the number of channels of the high-order initial feature.

[0075] refer to Figure 5 As shown, the number of high-order initial features is 3, and the corresponding conversion layers are represented as trans3, trans4 and trans5. The multiple high-order initial features after the conversion operation are respectively recorded as f 3 ,f 4 ,f 5 , the high-order features obtained based on multiple high-order initial features are recorded as f h .

[0076] refer to Figure 6 As shown in FIG. 1 , a schematic diagram of a high-order initial feature processing process provided in an embodiment of the present application is shown. The global induction module can be used for each high-order initial feature f 3 ,f 4 ,f 5 Determine the corresponding additional features and comprehensive features, wherein the additional features corresponding to the target initial features are determined based on other initial features with a higher hierarchy than the target initial features, and the comprehensive features corresponding to the target initial features are determined based on the target initial features and the additional features of the target initial features. 5 is the highest-level high-order initial feature. As the first high-order initial feature, its corresponding additional feature is empty, and the corresponding comprehensive feature is recorded as That is, according to f 5 Sure f 4 As the second high-order initial feature, its corresponding additional feature is determined according to the comprehensive feature of the first high-order initial feature, which can be The corresponding comprehensive feature is recorded as That is, according to f 5 and f 4 Sure f 3As the third high-order initial feature, its corresponding additional feature is determined according to the comprehensive features of the first high-order initial feature and the comprehensive features of the second high-order initial feature, which can be and The corresponding comprehensive feature is recorded as That is, according to and f 3 Sure

[0077] The comprehensive features corresponding to the target initial features are determined according to the target initial features and the additional features of the target initial features. Specifically, the target initial features and the additional features of the target initial features are integrated to obtain the input features corresponding to the target initial features, and then the input features corresponding to the target initial features are expanded and convolved to obtain the comprehensive features corresponding to the target initial features. Thus, the rich multi-scale contextual information of the high-order initial features can be captured through the expanded convolution without sacrificing the resolution of the training image.

[0078] The integration operation can be a pixel-based connection operation. The conversion from input features to comprehensive features can be achieved by the global perception module (GPM) in the global induction module. The global perception module is denoted as G. The input features of the target initial features can be used as input data, and the comprehensive features corresponding to the target initial features can be used as output data. The process of GPM is defined as Where F is the input feature map, The global perception module may include a dilated convolution layer and a convolution layer, so that a dilated convolution operation is performed through the dilated convolution layer, and a convolution operation is performed through the convolution layer.

[0079] refer to Figure 7 As shown, it is a schematic diagram of the data processing process of a global perception module in an embodiment of the present application. The input feature of the target initial feature is used as the input feature map F (Feature map). After the extended convolution operation, multiple feature maps are obtained. The size of the feature map can be W×H×C. Feature aggregation (Feature Aggregation) can be performed on the multiple feature maps, and then the convolution operation is performed on the aggregation result to obtain a comprehensive feature that reflects rich contextual semantic features (Contextfeature) The dilated convolution operation is implemented by dilated convolution through multiple parallel convolution units, the number of convolution units may be, for example, 14, the convolution kernel is 3×3, and the corresponding dilation ratios are 1 / 6 / 12 / 18. The convolution operation may be a 1×1 traditional convolution operation.

[0080] After determining the comprehensive features corresponding to the high-order initial features, the high-order features can be determined based on the comprehensive features. Specifically, each comprehensive feature can be subjected to aggregation, convolution and upsampling operations to obtain the high-order features. Figure 5 As shown, the aggregation operation can be a concatenation operation, and the convolution kernel of the convolution operation can be 3×3, denoted as CONV 3×3 , the upsampling operation can be 4 times bilinear interpolation upsampling (upsample×4), so the high-order feature f h and low-order features f l Have the same dimensions.

[0081] S103, performing boundary extraction on the low-order features to obtain boundary response information, and performing depth estimation on the high-order features to obtain depth response information.

[0082] In the embodiment of the present application, the initial model can be used to extract the boundary of the low-order features to obtain the boundary response information, and the depth of the high-order features can be estimated to obtain the depth response information. S103 may include S1031, l Perform boundary extraction to obtain boundary response information Att edge ,refer to Figure 4 , this is because the low-order features f l More details of the training image are retained, so the boundary information in the training image can be better reflected, that is, the texture information and boundary response information Att in the training image. edge In the subsequent prediction process, the characterization of object boundaries can be emphasized. For details, refer to Figure 5 As shown, 1×1 convolution (CONV 1×1 ) operation to obtain the boundary response information Att edge , for the boundary response information Att edge Perform softmax function operation to obtain the predicted boundary map M edge , the predicted boundary map reflects the boundary of the training image. Boundary response information Att edge It has a large amount of information and can be used as a reference in the prediction process, denoted as Att edge .

[0083] refer to Figure 8 As shown in the figure, a response feature schematic diagram provided in an embodiment of the present application is shown, wherein the training image is represented as Image, GT is the label information, and Ours is the prediction result. It can be seen from the figure that the predicted boundary map M edge It can reflect the boundary information of the target object. In addition, the predicted boundary map can also be expressed as M edge Complementary image of The two highlight different focuses respectively. The boundary response information can also be expressed as Att edge Complementary image of The two emphasize different focuses.

[0084] In the embodiment of the present application, S103 may further include S1032, performing depth prediction on the high-order features to obtain depth response information Att edge ,refer to Figure 4 This is because high-order features retain more abstract features of the training image and are more conducive to retaining global semantics. There is a strong consistency between global semantics and the geometric characteristics of objects. Therefore, the location of the object can be better reflected, and the saliency of the target can be reflected through high-order features.

[0085] Specifically, S1032 may include: S1032A, based on the spatial attention mechanism for high-order features f h Determine the significant response information Att sal ; S1032B, based on the significant response information Att sal For high-order features f h Perform saliency processing to obtain spatial attention information S1032C, for spatial attention information Perform depth estimation to obtain depth response information Att depth , that is, we can use the significant response information Att sal As a spatial attention map to refine the high-order features f h , emphasizing or suppressing high-order features f h For each pixel in, the high-order feature f h The salient targets (i.e., target objects) in the image are easier to identify.

[0086] When implementing it, refer to Figure 5 As shown, the high-order features f h Perform 1×1 convolution (CONV 1×1 ) operation to obtain significant high-order features, and determine significant response information Att based on the significant high-order features based on the spatial attention mechanism sal ; Determine the significant response information Att for high-order features sal This can be achieved through the spatial attention layer (SA); spatial attention information According to the significant response information Att sal and high-order features f h The pixel-wise multiplication operation is determined by, for example, the product and the high-order feature f h The sum can be used as spatial attention information This can alleviate the significant response information Att by identity mapping sal The error propagation in the network and accelerate the convergence of the network; for the spatial attention information Depth estimation can be performed by focusing on the spatial attention information The depth prediction branches of the three convolutional layers are used to constrain the features for depth estimation to obtain the depth response information Att depth .

[0087] refer to Figure 8 As shown, the significant response information Att sal , depth response information Att depth Reflecting the characteristics of the training image from different angles, the significant response information can also be expressed as Att sal Complementary image of The two highlight different focuses. Similarly, the depth response information can also be expressed as Att depth Complementary image of The two highlight different focuses. It should be noted that in order to ensure the unity of information, when using complementary images, each feature uses a complementary image, that is, features that reflect the same focus are selected at the same time.

[0088] S104, predicting a prediction result of the training image according to the low-order features, the high-order features, the boundary response information and the depth response information.

[0089] In the embodiment of the present application, the initial model can be used according to the low-order features, high-order features, and boundary response information Att edge and depth response Att depth The prediction result of the training image is obtained by information prediction. Specifically, S104 may include: S1041, aggregating the low-order features and the high-order features to obtain the aggregated features F g ; S1042, integrating boundary response information Att for aggregated features edge and depth response information Att depth Get the integrated result; S1043, predict the training image based on the integrated result. edge It can assist the prediction process of the results, strengthen the detailed description of the prediction results through boundary constraints, and the deep response information Att depth It can also assist the prediction process of the results, enhance the contrast of the prediction results through depth constraints, make full use of the complementarity of high-order features and low-order features according to the characteristics of features at different levels, and enhance the depth information extraction capability of the target detection model through depth images, thereby improving the performance of the target detection model.

[0090] In determining the significant response information Att sal In the case of, S1042 may be specifically, for the aggregate feature Fg Integrated significant response information Att sal , which can be the aggregate feature F g Integrated significant response information Att sal , boundary response information Att edge and depth response information Att edge Get the integrated result, and then predict the prediction result of the training image based on the integrated result, refer to Figure 4 . Such significant response information Att sal It can enhance the position description of salient targets, thus helping to improve the performance of target detection models.

[0091] Aggregate feature F g It can be obtained by directly aggregating high-order features and low-order features. For example, high-order features and low-order features are aggregated based on feature connection. The aggregate feature F g It can also be obtained by aggregating the processing results obtained after processing the high-order features and the low-order features based on feature connections, for example, in determining the deep response information Att based on the high-order features depth Afterwards, refer to Figure 4 As shown, S1041 may include: S1041A, based on the channel attention mechanism, deep response information Att depth Determine channel attention information; S1041B, based on the channel attention information, determine the spatial attention information Perform saliency processing to obtain enhanced features f hc ; S1041C, will enhance the feature f hc As the processing result obtained after processing the high-order features, the aggregated features are aggregated with the low-order features. In this way, the aggregated features contain more attention information, which can enhance the position description of the salient target. sal and depth response information Att edge They can work together to enhance the functional learning process of global semantics in a mutually effective and complementary manner.

[0092] is the depth response information Att depth Determining the channel attention information can be achieved through the channel attention layer (CA); enhancing the feature f hc Based on channel attention information and spatial attention information The pixel-based multiplication result is determined, for example, the product is multiplied with the spatial attention information The sum can be used as the enhanced feature f hc , which can alleviate the error propagation in channel attention information through identity mapping and accelerate network convergence.

[0093] is the aggregate feature Fg Integrated significant response information Att sal , boundary response information Att edge and depth response information Att edge In the process of obtaining the integrated results, refer to Figure 4 S1042 may be specifically as follows: S1042A, based on the significant response information Att sal and boundary response information Att edge Aggregation result of Att fuse Perform classification and obtain the classification attention information Att of the training image f , the classified attention information can effectively describe the location and boundary of the salient target; S1042B, is the aggregate feature F g Integrated depth response information Att depth and classification attention information Att f The integrated result is obtained, thereby enhancing the contrast between the area where the salient target is located and the area where the non-salient target is located. Figure 8 As shown, the aggregation result and classification attention information can be represented as Att fuse and Att f complementary information and Complementary information and itself reflects different focuses.

[0094] Among them, reference Figure 5 As shown, the aggregate feature F g Integrated significant response information Att sal , boundary response information and depth response information Att edge This can be achieved through the Knowledge Collector (KC), which significantly responds to the information Att sal , boundary response information Att edge and depth response information Att edge , and the aggregated features are unified into the knowledge collector, significantly responding to the information Att sal , boundary response information Att edge and depth response information Att edge As a triple attention and aggregation feature aggregation process, it summarizes rich boundary, depth and saliency knowledge, and assists in accurate target positioning through exploratory learning and timely interaction. It can emphasize the location area and detailed boundaries of the target object, and thus obtain a robust result prediction.

[0095] When implementing it, refer to Fig. 9 As shown, it is a schematic diagram of the working process of a knowledge collector provided in an embodiment of the present application, and the significant response information Att sal and boundary response information Att edgeAggregation can be achieved through pixel-based connection to obtain the aggregation result Att fuse , through the softmax function to aggregate the results Att fuse By classifying, we can get the classification attention information of the training image; aggregate feature F g can be sequentially combined with the depth response information Att depth and classification attention information Att f Based on pixel arrangement multiplication, and depth response information Att edge After multiplication, the multiplication result can be combined with the aggregate feature F g A pixel-wise addition operation is performed, and then the addition result is multiplied with the classification attention information, and then the multiplication result is added to the aforementioned addition result to obtain the integrated result. After that, the prediction result of the training image is obtained based on the integrated result, which can be obtained by performing a 1×1 convolution (CONV 1×1 ) operation and upsampling operation to generate the final prediction result S final .

[0096] S105, calculate the loss based on the prediction result and the label information to obtain the prediction loss function, and calculate the loss based on the boundary information and boundary response information Att of the label information. edge The loss is calculated to obtain the boundary loss function, based on the depth image and depth response information Att edge The loss is calculated to obtain the deep loss function.

[0097] In the embodiment of the present application, since the label information is used to reflect the target object in the training image, it can be used to supervise the prediction result. The closer the prediction result is to the label information, the stronger the prediction ability of the initial model is. Therefore, S105 may include: S1051, performing loss calculation based on the prediction result and the label information to obtain a prediction loss function, and the prediction loss function represents the overall prediction ability of the initial model. f This can be determined by the difference between the prediction result and the label information, which can be calculated using the binary cross entropy loss.

[0098] In the embodiment of the present application, due to the boundary response information Att edge It can reflect the boundary of the training image, while the label information reflects the target object. The boundary information determined according to the label information can reflect the real boundary of the target object, so the boundary response information Att can be constrained by the real boundary. edge , boundary response information Att edge The closer it is to the real boundary, the more the low-order features extracted by the initial model can reflect the real boundary of the target object, the stronger the extraction ability of the low-order features is, and the boundary response information Attedge The more accurate the label information is, the more reliable the constraint effect in the result prediction process is. Therefore, S105 may also include: S1052, according to the boundary information of the label information and the boundary response information Att edge The loss is calculated to obtain the boundary loss function, which characterizes the initial model's ability to extract accurate low-order features and obtain boundary information.

[0099] Among them, the real boundary information in the label information can be extracted by a boundary extractor, such as canny; the boundary response information Att edge It is obtained through convolution operation, which can be used to obtain the predicted boundary information M through softmax operation. edge ; Boundary loss function Loss e It can be determined by the difference between the predicted boundary information and the actual boundary information, and the difference between the two can be calculated by the binary cross entropy loss.

[0100] In the embodiment of the present application, due to the depth response information Att edge It can reflect the depth information of the training image, and the depth image reflects the real depth of the training image, so the depth response information Att can be constrained by the depth image edge , depth response information Att edge The closer it is to the true depth, the more the high-order features extracted by the initial model can reflect the true depth, the stronger the extraction ability of high-order features is, and the deeper the depth response information Att is. edge The more accurate, the richer the spatial clues and the stronger the distinguishing power, the more reliable the constraint effect in the result prediction process. Therefore, S105 may also include: S1053, according to the depth image and the depth response information Att edge The loss is calculated to obtain the deep loss function, which represents the initial model's ability to extract accurate high-order features and obtain deep information. d It can be determined by the difference between the new depth response and the true depth of the depth image, which can be calculated by a smooth L1 loss. In this way, the spatial information in the depth image is used to constrain the depth response information Att edge , instead of taking the depth image as input for feature extraction and processing, there is no need to use an additional deep network to process the depth image, making the model more lightweight and universal.

[0101] In addition, in the embodiment of the present application, the loss calculation can be performed based on the high-order features and label information to obtain a significant loss function. The high-order features are abstract features that can reflect the location of significant targets. Therefore, the significant result S can be determined based on the high-order features. coarse, the significant result represents the prediction result determined by the high-order features, for example, the high-order feature f h Perform 1×1 convolution operation to obtain significant high-order features, and perform softmax function processing on the significant high-order features to obtain the significant result S coarse Since the label information is used to reflect the target object in the training image, it can be used to supervise the saliency result S coarse , significant result S coarse The closer to the label information, the more the high-order features can reflect the target characteristics. Therefore, S105 may also include: S1054, classifying according to the high-order features to obtain a significant result; S1055, calculating the loss according to the significant result and the label information to obtain a significant loss function Loss s ,refer to Figure 4 As shown, the significant loss function Loss s Characterizes the feature expressiveness of high-order features. s The significant result S coarse The difference between the information in the dataset and the label information can be calculated by the binary cross entropy loss.

[0102] Significant results S coarse Not a significant response information sal , significant response information Att sal For high-order features f h Perform 1×1 convolution (CONV 1×1 ) operation obtains significant high-order features, and determines the significant high-order features based on the spatial attention mechanism, which has the characteristics of spatial attention and contains a larger amount of information, which is conducive to achieving more effective target constraints in the prediction process.

[0103] Among them, the execution of S1051, S1052, S1053 and S1054 are independent of each other and can be executed in any order or simultaneously, which is not limited here.

[0104] S106, training the initial model according to the boundary loss function, the depth loss function and the prediction loss function, so that the initial model is converted into a target detection model.

[0105] In the embodiment of the present application, the boundary loss function, the depth loss function and the prediction loss function respectively evaluate the initial model from different aspects, so the initial model can be trained based on the boundary loss function, the depth loss function and the prediction loss function, so that the initial model is converted into a target detection model. Specifically, the comprehensive loss function can be obtained by comprehensively calculating the boundary loss function, the depth loss function and the prediction loss function, and the comprehensive loss function can be used to train the initial model so that the initial model is converted into a target detection model.

[0106] When the significant loss function is determined, the initial model can also be trained according to the significant loss function. For details, refer to Figure 4 As shown, S106 can be specifically as follows: S1061, comprehensively calculating the boundary loss function, the depth loss function, the prediction loss function and the significant loss function to obtain a comprehensive loss function; S1062, training the initial model according to the comprehensive loss function. The comprehensive loss function can be the sum of the boundary loss function, the depth loss function, the prediction loss function and the significant loss function, that is, L = Loss e +Loss s +Loss d +Loss f ; Or the comprehensive loss function can be a weighted sum of the boundary loss function, the depth loss function, the prediction loss function and the significant loss function.

[0107] The target detection model can be used to perform target detection on the image to be detected, determine the target object in the image to be detected, and the target object is also used as the foreground. The image to be detected can be a color image or a grayscale image. Specifically, the target detection model can perform feature extraction on the image to be detected to obtain low-order features and high-order features with a higher hierarchy than the low-order features, perform boundary extraction on the low-order features to obtain boundary response information, perform depth estimation on the high-order features to obtain depth response information, and predict the prediction result of the image to be detected based on the low-order features, high-order features, boundary response information and depth response information. The above process can refer to the process of extracting features from the training image and obtaining the prediction result, which will not be repeated here. In addition, the significant response information can be determined based on the high-order features of the image to be detected, and the prediction result of the image to be detected can be predicted based on the significant response information, low-order features, high-order features, boundary response information and depth response information.

[0108] The target detection model has the ability to accurately obtain high-order features, low-order features, boundary response information, significant response information and depth response information. Therefore, in the prediction process, significant response information, boundary response information and depth response information can be aggregated as triple attention and aggregation features, summarizing rich boundary, depth and saliency knowledge, and assisting in accurate positioning of targets through exploratory learning and timely interaction. It can emphasize the location area and detail boundaries of the target object, and then obtain robust result prediction. Even if the amount of information in the image to be detected is small, features can be extracted through different dimensions, making full use of the complementarity between features at different levels, and using boundary depth, depth and saliency knowledge to predict the image to be detected and obtain accurate prediction results.

[0109] Based on a method for training a target detection model provided in an embodiment of the present application, an embodiment of the present application also provides a device for training a target detection model, referring to Fig.10 As shown, it is a structural block diagram of a training device for a target detection model provided in an embodiment of the present application. The training device 1300 for the target detection model includes:

[0110] The training data acquisition unit 1301 is used to acquire a pair of training images and depth images, wherein the training images have label information indicating a target object in the training images;

[0111] A feature extraction unit 1302 is used to extract features from the training image using an initial model to obtain low-order features and high-order features with a higher hierarchy than the low-order features;

[0112] A feature processing unit 1303 is used to extract the boundary of the low-order feature to obtain boundary response information, and to estimate the depth of the high-order feature to obtain depth response information;

[0113] A prediction unit 1304, configured to predict a prediction result of the training image according to the low-order features, the high-order features, the boundary response information and the depth response information;

[0114] A loss function determination unit 1305 is configured to perform loss calculation according to the prediction result and the label information to obtain a prediction loss function, perform loss calculation according to the boundary information of the label information and the boundary response information to obtain a boundary loss function, and perform loss calculation according to the depth image and the depth response information to obtain a depth loss function;

[0115] The training unit 1306 is used to train the initial model according to the boundary loss function, the depth loss function and the prediction loss function, so that the initial model is converted into a target detection model, and the target detection model is used to perform target detection on the image to be detected.

[0116] Wherein, the feature processing unit 1303 includes:

[0117] A boundary extraction unit, used for performing boundary extraction on the low-order features to obtain boundary response information;

[0118] The depth estimation unit is used to perform depth estimation on the high-order features to obtain depth response information.

[0119] Optionally, the depth estimation unit includes:

[0120] A spatial attention unit, configured to determine salient response information for the high-order features based on a spatial attention mechanism;

[0121] A spatial saliency unit, configured to perform saliency processing on the high-order features based on the salient response information to obtain spatial attention information;

[0122] The depth estimation subunit is used to perform depth estimation on the spatial attention information to obtain depth response information.

[0123] Optionally, the prediction unit 1304 includes:

[0124] A feature aggregation unit, used for aggregating the low-order features and the high-order features to obtain aggregated features;

[0125] A feature integration unit, configured to integrate the significant response information, the boundary response information and the depth response information for the aggregated feature to obtain an integration result;

[0126] A result prediction unit is used to predict the prediction result of the training image according to the integrated result.

[0127] Optionally, the feature aggregation unit includes:

[0128] A channel attention unit, configured to determine channel attention information for the deep response information based on a channel attention mechanism;

[0129] A spatial saliency unit, configured to perform saliency processing on the spatial attention information based on the channel attention information to obtain enhanced features;

[0130] The feature aggregation subunit is used to aggregate the low-order features and the enhanced features to obtain aggregated features.

[0131] Optionally, the feature integration unit includes:

[0132] a classification unit, configured to perform classification according to an aggregation result of the salient response information and the boundary response information to obtain classification attention information of the training image;

[0133] The feature integration subunit is used to integrate the deep response information and the classification attention information into the aggregated feature to obtain an integrated result.

[0134] Optionally, the device further comprises:

[0135] A feature classification unit, used for classifying according to the high-order features to obtain a significant result;

[0136] A significant loss determination unit, configured to perform loss calculation according to the significant result and the label information to obtain a significant loss function;

[0137] The training unit 1306 includes:

[0138] A comprehensive loss determination unit, configured to obtain a comprehensive loss function by comprehensive calculation based on the boundary loss function, the depth loss function, the prediction loss function and the significant loss function;

[0139] The training subunit is used to train the initial model according to the comprehensive loss function.

[0140] Optionally, the feature extraction unit includes:

[0141] A feature extraction subunit, used for extracting features from the training image using an initial model to obtain a plurality of initial features of different levels;

[0142] a distinguishing unit, configured to determine a low-order initial feature and a high-order initial feature from the plurality of initial features according to a preset condition, wherein the hierarchy of the high-order initial feature is higher than that of the low-order feature;

[0143] A low-order feature determination unit, used for performing feature processing on the low-order initial features to obtain low-order features;

[0144] The high-order feature determination unit is used to perform feature processing on the high-order initial features to obtain high-order features.

[0145] Optionally, the high-order feature determination unit includes:

[0146] an additional feature determination unit, configured to determine, for a target initial feature among the plurality of high-order initial features, additional features of the target initial feature according to other initial features of a higher level than the target initial feature among the plurality of high-order initial features if the number of the high-order initial features is multiple;

[0147] An integration unit, used for integrating the target initial feature and the additional feature to obtain a comprehensive feature corresponding to the target initial feature;

[0148] The high-order feature determination subunit is used to perform feature processing on the comprehensive features corresponding to the multiple high-order initial features to obtain high-order features.

[0149] Optionally, the number of the plurality of high-order initial features is n, and the additional feature determination unit includes:

[0150] A sorting unit, used to sort the plurality of high-order initial features from high to low according to the hierarchy, and determine that the additional feature of the first high-order initial feature is empty;

[0151] The additional feature determination subunit is used to determine the additional features of the i-th high-order initial feature based on the additional features of the previous i-1 high-order initial features and the i-th high-order initial features, where i is less than or equal to n.

[0152] Optionally, the integration unit includes:

[0153] An integration subunit, used for integrating the target initial feature and the additional feature to obtain an input feature corresponding to the target initial feature;

[0154] The operation unit is used to perform an extended convolution operation and a convolution operation on the input feature to obtain a comprehensive feature corresponding to the target initial feature.

[0155] Optionally, the device further comprises:

[0156] A first conversion unit is used for, before performing feature processing on the low-order initial features to obtain low-order features, if there are multiple low-order initial features, performing conversion operations on the multiple low-order initial features based on a preset number of channels respectively;

[0157] The device also includes:

[0158] The second conversion unit is used to perform conversion operations on the multiple high-order initial features based on a preset number of channels before determining the additional features of the target initial feature among the multiple high-order initial features according to other initial features in the multiple high-order initial features that have a higher hierarchy than the target initial feature.

[0159] It can be seen from the above technical solution that a pair of training images and depth images can be obtained as training data, and the initial model can be trained to obtain a target detection model. The training image can have label information for indicating the target object in the training image. The initial model can be used to extract features from the training image to obtain low-order features and high-order features with a higher hierarchy than the low-order features. The low-order features can be subjected to boundary extraction to obtain boundary response information. The high-order features can be subjected to depth estimation to obtain depth response information. The prediction results of the training image can be predicted based on the low-order features, high-order features, boundary response information and depth response information. In this way, the boundary response information can assist the prediction process of the results, and the detailed characterization of the prediction results can be enhanced through boundary constraints. The depth response information can also assist the prediction process of the results, and the contrast of the prediction results can be enhanced through depth constraints.

[0160] Afterwards, the prediction loss function can be obtained by performing loss calculation based on the prediction result and label information, the boundary loss function can be obtained by performing loss calculation based on the boundary information and boundary response information of the label information, and the depth loss function can be obtained by performing loss calculation based on the depth image and depth response information. The initial model is trained according to the boundary loss function, the depth loss function and the prediction loss function, so that the initial model is converted into a target detection model, so that the obtained target detection model can be used for target detection of the image to be detected. In this way, by training the initial model, the prediction result is closer to the label information, the depth response information is closer to the depth image, and the boundary response information is closer to the boundary information of the label information. In the initial model training process, the label information is used to supervise the boundary response information, and the depth image is used to supervise the depth response information, which improves the ability to accurately extract the boundary response information and the depth response information, and obtains accurate boundary corresponding information and depth corresponding information for auxiliary target detection, further improving the auxiliary role of boundary constraints and depth constraints on the prediction results, making full use of the complementarity of high-order features and low-order features, and by combining the depth image to enhance the depth information extraction ability of the target detection model, the data performance potential of different image modalities is excavated, and the performance of the target detection model is improved. In addition, there is no need to perform operations such as feature extraction on the depth image, which is conducive to the miniaturization of the target detection model.

[0161] The embodiment of the present application further provides a computer device, which is the computer device described above and may include a terminal device or a server, and the training device of the target detection model described above may be configured in the computer device. The computer device is described below in conjunction with the accompanying drawings.

[0162] If the computer device is a terminal device, see Fig.11 As shown, the embodiment of the present application provides a terminal device, taking the terminal device as a mobile phone as an example:

[0163] Fig.11 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal device provided in an embodiment of the present application. Fig.11 The mobile phone includes: a radio frequency (RF) circuit 1410, a memory 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a wireless fidelity (WiFi) module 1470, a processor 1480, and a power supply 1490. Those skilled in the art can understand that Fig.11 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0164] Combine the following Fig.11A detailed introduction to the various components of the mobile phone:

[0165] The RF circuit 1410 may be used for receiving and sending signals during information transmission or communication or during a call. In particular, after receiving downlink information from a base station, the information is sent to the processor 1480 for processing. In addition, the uplink data is sent to the base station.

[0166] The memory 1420 can be used to store software programs and modules. The processor 1480 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1420. The memory 1420 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1420 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0167] The input unit 1430 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the mobile phone. Specifically, the input unit 1430 may include a touch panel 1431 and other input devices 1432 .

[0168] The display unit 1440 may be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1440 may include a display panel 1441.

[0169] The mobile phone may also include at least one sensor 1450, such as a light sensor, a motion sensor, and other sensors.

[0170] The audio circuit 1460 , the speaker 1461 , and the microphone 1462 can provide an audio interface between the user and the mobile phone.

[0171] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1470. It provides users with wireless broadband Internet access.

[0172] Processor 1480 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. It executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in memory 1420 and calling data stored in memory 1420.

[0173] The mobile phone also includes a power source 1490 (such as a battery) for supplying power to various components.

[0174] In this embodiment, the processor 1480 included in the terminal device also has the following functions:

[0175] Acquire a pair of training images and depth images, wherein the training images have label information for indicating a target object in the training images;

[0176] Using the initial model to extract features from the training image to obtain low-order features and high-order features with a higher hierarchy than the low-order features;

[0177] Performing boundary extraction on the low-order features to obtain boundary response information, and performing depth estimation on the high-order features to obtain depth response information;

[0178] Predicting a prediction result of the training image according to the low-order features, the high-order features, the boundary response information and the depth response information;

[0179] A prediction loss function is obtained by performing loss calculation based on the prediction result and the label information, a boundary loss function is obtained by performing loss calculation based on the boundary information of the label information and the boundary response information, and a depth loss function is obtained by performing loss calculation based on the depth image and the depth response information;

[0180] The initial model is trained according to the boundary loss function, the depth loss function and the prediction loss function, so that the initial model is converted into a target detection model, and the target detection model is used to perform target detection on the image to be detected.

[0181] If the computer device is a server, the present application embodiment also provides a server, see Fig.12 As shown, Fig.12 The structural diagram of the server 1500 provided in the embodiment of the present application, the server 1500 may have relatively large differences due to different configurations or performances, and may include one or more processors 1522, such as a central processing unit (CPU), a memory 1532, and one or more storage media 1530 (e.g., one or more mass storage devices) storing application programs 1542 or data 1544. Among them, the memory 1532 and the storage medium 1530 can be short-term storage or permanent storage. The program stored in the storage medium 1530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the processor 1522 can be configured to communicate with the storage medium 1530 and execute a series of instruction operations in the storage medium 1530 on the server 1500.

[0182] The server 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input and output interfaces 1558, and / or one or more operating systems 1541, such as Windows Server 2003 or Windows Server 2008. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0183] The steps performed by the server in the above embodiment can be based on Fig.12 The server structure shown.

[0184] In addition, an embodiment of the present application further provides a storage medium, wherein the storage medium is used to store a computer program, and the computer program is used to execute the method provided in the above embodiment.

[0185] The embodiments of the present application also provide a computer program product including instructions, which, when executed on a computer, enables the computer to execute the method provided in the above embodiments.

[0186] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium can be at least one of the following media: read-only memory (English: Read-only Memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc. Various media that can store program codes.

[0187] It should be noted that each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0188] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a technician familiar with the technical field within the technical scope disclosed in the present application should be included in the protection scope of the present application. Moreover, based on the implementation methods provided in the above aspects, the present application can also be further combined to provide more implementation methods. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for training a target detection model, characterized in that: The method comprises: Acquire a pair of training images and depth images, wherein the training images have label information for indicating a target object in the training images; Extracting features from the training image using the initial model to obtain low-order features and high-order features with a higher hierarchy than the low-order features; Performing boundary extraction on the low-order features to obtain boundary response information, and performing depth estimation on the high-order features to obtain depth response information; Predicting a prediction result of the training image according to the low-order features, the high-order features, the boundary response information and the depth response information; A prediction loss function is obtained by performing loss calculation based on the prediction result and the label information, a boundary loss function is obtained by performing loss calculation based on the boundary information of the label information and the boundary response information, and a depth loss function is obtained by performing loss calculation based on the depth image and the depth response information; The initial model is trained according to the boundary loss function, the depth loss function and the prediction loss function, so that the initial model is converted into a target detection model, and the target detection model is used to perform target detection on the image to be detected.

2. The method according to claim 1, characterized in that The performing depth estimation on the high-order features to obtain depth response information includes: Determining salient response information for the high-order features based on a spatial attention mechanism; Based on the salient response information, the high-order features are saliently processed to obtain spatial attention information; Depth estimation is performed on the spatial attention information to obtain depth response information.

3. The method according to claim 2, characterized in that The predicting of the training image according to the low-order features, the high-order features, the boundary response information and the depth response information comprises: Aggregating the low-order features and the high-order features to obtain aggregated features; Integrating the significant response information, the boundary response information and the depth response information for the aggregated feature to obtain an integrated result; The prediction result of the training image is obtained according to the integrated result prediction.

4. The method according to claim 3, characterized in that The step of aggregating the low-order features and the high-order features to obtain aggregated features includes: Determining channel attention information for the deep response information based on a channel attention mechanism; Performing saliency processing on the spatial attention information based on the channel attention information to obtain enhanced features; The low-order features and the enhanced features are aggregated to obtain aggregated features.

5. The method according to claim 3, characterized in that: The step of integrating the significant response information, the boundary response information, and the depth response information for the aggregated feature to obtain an integrated result includes: Classify according to the aggregation result of the salient response information and the boundary response information to obtain classified attention information of the training image; The deep response information and the classification attention information are integrated into the aggregated feature to obtain an integrated result.

6. The method according to claim 2, characterized in that The method further comprises: Classifying according to the high-order features to obtain significant results; Performing loss calculation according to the significance result and the label information to obtain a significance loss function; The training of the initial model according to the boundary loss function, the depth loss function and the prediction loss function comprises: A comprehensive loss function is obtained by performing comprehensive calculation according to the boundary loss function, the depth loss function, the prediction loss function and the significant loss function; The initial model is trained according to a comprehensive loss function.

7. The method according to any one of claims 1 to 6, characterized in that: The extracting features of the training image using the initial model to obtain low-order features and high-order features with a higher hierarchy than the low-order features includes: Using the initial model to extract features from the training image to obtain a plurality of initial features of different levels; Determining a low-order initial feature and a high-order initial feature from the plurality of initial features according to a preset condition, wherein the hierarchy of the high-order initial feature is higher than that of the low-order feature; The low-order initial features are subjected to feature processing to obtain low-order features, and the high-order initial features are subjected to feature processing to obtain high-order features.

8. The method according to claim 7, characterized in that The step of performing feature processing on the high-order initial features to obtain high-order features includes: If there are multiple high-order initial features, for a target initial feature among the multiple high-order initial features, determine the additional features of the target initial feature according to other initial features with a higher hierarchy than the target initial feature among the multiple high-order initial features; Performing an integration operation on the target initial feature and the additional feature to obtain a comprehensive feature corresponding to the target initial feature; Feature processing is performed on the comprehensive features corresponding to the multiple high-order initial features to obtain high-order features.

9. The method according to claim 8, characterized in that The number of the plurality of high-order initial features is n, and the determining of the additional features of the target initial feature among the plurality of high-order initial features according to other initial features in the plurality of high-order initial features having a higher level than the target initial feature comprises: Sort the plurality of high-order initial features from high to low in terms of hierarchy, and determine that the additional feature of the first high-order initial feature is empty; The additional features of the i-th high-order initial feature are determined based on the additional features of the previous i-1-th high-order initial feature and the i-th high-order initial feature, where i is less than or equal to n.

10. The method according to claim 8, characterized in that The integrating operation of the target initial feature and the additional feature to obtain the comprehensive feature corresponding to the target initial feature includes: Performing an integration operation on the target initial feature and the additional feature to obtain an input feature corresponding to the target initial feature; Perform an extended convolution operation and a convolution operation on the input feature to obtain a comprehensive feature corresponding to the target initial feature.

11. The method according to claim 8, characterized in that Before performing feature processing on the low-order initial features to obtain low-order features, the method further includes: If there are multiple low-order initial features, converting the multiple low-order initial features based on a preset number of channels; Before determining the additional features of the target initial feature among the plurality of high-order initial features based on other initial features in the plurality of high-order initial features that are higher in hierarchy than the target initial feature, the method further includes: Based on a preset number of channels, conversion operations are performed on the plurality of high-order initial features respectively.

12. A training device for a target detection model, characterized in that: The device comprises: A training data acquisition unit, used to acquire a pair of training images and depth images, wherein the training images have label information for indicating a target object in the training images; A feature extraction unit, configured to extract features from the training image using an initial model to obtain low-order features and high-order features with a higher hierarchy than the low-order features; A feature processing unit, configured to extract boundaries of the low-order features to obtain boundary response information, and to estimate depth of the high-order features to obtain depth response information; A prediction unit, configured to predict a prediction result of the training image according to the low-order features, the high-order features, the boundary response information and the depth response information; A loss function determination unit, configured to perform loss calculation according to the prediction result and the label information to obtain a prediction loss function, perform loss calculation according to the boundary information of the label information and the boundary response information to obtain a boundary loss function, and perform loss calculation according to the depth image and the depth response information to obtain a depth loss function; A training unit is used to train the initial model according to the boundary loss function, the depth loss function and the prediction loss function, so that the initial model is converted into a target detection model, and the target detection model is used to perform target detection on the image to be detected.

13. A computer device, characterized in that: The computer device comprises a processor and a memory: The memory is used to store a computer program and transmit the computer program to the processor; The processor is used to execute the target detection model training method described in any one of claims 1-11 according to the instructions in the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the training method of the target detection model described in any one of claims 1-11.

15. A computer program product comprising a computer program, characterized in that When it runs on a computer device, the computer device executes the training method of the target detection model described in any one of claims 1-11.

Citation Information

Patent Citations

  • Saliency target detection method and device, computer equipment and storage medium

    CN111242143A

  • Image detection method and device

    CN113763405A

  • Significant target detection method and system based on boundary enhancement

    CN114821059A

  • Water surface and underwater target distinguishing method and equipment based on shallow water acoustic field correlation

    CN115687901A

Cited By

  • Face anti-counterfeiting identification method and device, equipment and storage medium

    CN122244929A

  • A method, device, equipment and storage medium for facial recognition anti-counterfeiting.

    CN122244929B