Training method and device of detection segmentation model, electronic equipment and storage medium

Abstract

The application relates to the computer technical field, and in particular to a training method and device of a detection segmentation model, an electronic device and a storage medium, which can be applied to intelligent traffic, auxiliary driving, automatic driving and the like, and is used for efficiently and accurately realizing detection and segmentation of different targets. The method comprises the following steps: obtaining a training sample set; selecting a training sample in the training sample set and inputting the training sample into a feature extraction network of a detection segmentation model to be trained; based on the feature extraction network, performing multi-scale feature fusion and attention feature extraction on a sample image in the training sample to obtain a multi-scale attention feature map; inputting the multi-scale attention feature map into a target detection head network and a target segmentation head network of the model respectively to obtain prediction position information of a sample detection target and prediction category information of a sample segmentation target; and based on the prediction position information and the prediction category information, adjusting parameters of the model, so that different targets can be efficiently and accurately detected and segmented.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and storage medium for training a detection segmentation model. Background Technology

[0002] Object detection and object segmentation are two important applications of computer vision technology. Object detection refers to detecting objects of a specified category in a given image using bounding boxes (such as rectangles); while object segmentation is the task of separating the parts corresponding to objects of a specified category.

[0003] Supervised deep learning-based object detection and segmentation methods have been successfully implemented in various fields, such as object detection and segmentation in autonomous driving scenarios. However, these methods typically use different network architectures to achieve object detection and segmentation tasks.

[0004] However, if different network structures are used for learning, on the one hand, a lot of computing resources will be consumed, which will not meet the system's requirements for high real-time performance; on the other hand, the target to be detected and the target to be segmented usually have a certain relationship at the level of visual semantic features. If target detection and target segmentation are treated as two independent sub-tasks, this relationship cannot be captured.

[0005] Therefore, how to efficiently and accurately complete the task of target detection and target segmentation of different categories of elements in an image is an urgent problem to be solved. Summary of the Invention

[0006] This application provides a method, apparatus, electronic device, and storage medium for training a detection and segmentation model, which can efficiently and accurately detect and segment different targets in an image.

[0007] This application provides a method for training a detection and segmentation model, comprising:

[0008] Obtain the training sample set;

[0009] Select training samples from the training sample set and input them into the feature extraction network of the detection and segmentation model to be trained. Based on the feature extraction network, perform multi-scale feature fusion and attention feature extraction on the sample images in the training samples to obtain the multi-scale attention feature map of the sample images.

[0010] The multi-scale attention feature map is input into the target detection head network and the target segmentation head network in the detection and segmentation model, respectively, to obtain the predicted location information of the target in the sample image and the predicted category information of the target in the sample segmentation.

[0011] The detection and segmentation model is adjusted by using a loss function constructed based on the predicted location information and the predicted category information.

[0012] This application provides a detection and segmentation model training device, comprising:

[0013] The sample acquisition unit is used to acquire the training sample set;

[0014] The first feature extraction unit is used to select training samples from the training sample set, input them into the feature extraction network of the detection and segmentation model to be trained, and perform multi-scale feature fusion and attention feature extraction on the sample images in the training samples based on the feature extraction network to obtain the multi-scale attention feature map of the sample images.

[0015] The second feature extraction unit is used to input the multi-scale attention feature map into the target detection head network and the target segmentation head network in the detection and segmentation model, respectively, to obtain the predicted location information of the target in the sample image and the predicted category information of the target in the sample segmentation.

[0016] The parameter adjustment unit is used to adjust the parameters of the detection and segmentation model using a loss function constructed based on the predicted location information and the predicted category information.

[0017] Optionally, the detection and segmentation model further includes: a first decoding network and a second decoding network; the first decoding network and the second decoding network have the same network structure but different network parameters;

[0018] Before inputting the multi-scale attention feature map into the target detection head network and the target segmentation head network of the detection and segmentation model, the second feature extraction unit is further configured to:

[0019] The first decoding feature map is obtained by decoding the multi-scale attention feature map through the first decoding network;

[0020] The second decoding network decodes the multi-scale attention feature map to obtain the second decoded feature map.

[0021] Optionally, the second feature extraction unit is specifically used for:

[0022] The first decoded feature map is input into the target detection head network, and the target detection head network is used to perform target detection on the sample image to obtain the predicted location information of the target detected in the sample.

[0023] The second decoded feature map is input into the target segmentation head network, and the target segmentation head network is used to segment the sample image to obtain the predicted category information of the target segmentation in the sample.

[0024] Optionally, each training sample in the training sample set further includes the true location information of the target detected in the corresponding sample image and the true category information of the target segmented in the sample; the parameter adjustment unit is specifically used for:

[0025] A detection loss function is constructed based on the difference between the predicted location information and the true location information; a segmentation loss function is constructed based on the difference between the predicted category information and the true category information.

[0026] The detection and segmentation model can be adjusted by at least one of the following methods:

[0027] The detection loss function and the segmentation loss function are used alternately to adjust the parameters of the detection and segmentation model;

[0028] The detection and segmentation model is adjusted by using a total loss function constructed based on the detection loss function and the segmentation loss function.

[0029] Optionally, the detection and segmentation model further includes: a first decoding network and a second decoding network for decoding the multi-scale attention feature map; the first decoding network and the second decoding network have the same network structure but different network parameters;

[0030] The parameter adjustment unit is specifically used for:

[0031] Based on the detection loss function, the parameters of the feature extraction network, the first decoding network, and the target detection head network in the detection segmentation model are updated.

[0032] The parameters of the feature extraction network, the second decoding network, and the target segmentation head network in the detection and segmentation model are updated based on the segmentation loss function.

[0033] Optionally, before constructing the detection loss function based on the difference between the predicted location information and the true location information, the parameter adjustment unit is further configured to:

[0034] Based on the target detection head network, a first probability distribution map corresponding to the sample image is obtained. The pixel value of each pixel in the first probability distribution map represents the first predicted probability of the category to which the target of the sample belongs.

[0035] The step of constructing a detection loss function based on the difference between the predicted location information and the actual location information includes:

[0036] A classification loss function is constructed based on each first predicted probability;

[0037] Based on the difference between the predicted location information and the actual location information, a location loss function is constructed;

[0038] The detection loss function is constructed based on the classification loss function and the location loss function.

[0039] Optionally, before determining the detection loss function based on the classification loss function and the location loss function, the parameter adjustment unit is further configured to:

[0040] The second probability distribution map corresponding to the sample image is obtained based on the target detection head network. The pixel value of each pixel in the second probability distribution map represents the second predicted probability of the detected target.

[0041] The parameter adjustment unit is specifically used for:

[0042] Construct a target loss function based on each second prediction probability;

[0043] The detection loss function is obtained by weighted summing of the target loss function, the classification loss function, and the location loss function.

[0044] Optionally, the sample detection target and the sample segmentation target belong to different categories.

[0045] Optionally, the device further includes:

[0046] The model application unit is used to input the image to be processed into the trained detection and segmentation model;

[0047] Based on the feature extraction network of the detection and segmentation model, multi-scale feature fusion and attention feature extraction are performed on the image to be processed to obtain the target multi-scale attention feature map corresponding to the image to be processed.

[0048] The target multi-scale attention feature map is input into the target detection head network and the target segmentation head network in the detection and segmentation model, respectively, to obtain the target location information and target category information of the target to be segmented in the image to be processed; the target to be segmented and the target to be segmented belong to different categories.

[0049] An electronic device provided in this application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any of the above-described training methods for detection and segmentation models.

[0050] This application provides a computer-readable storage medium including a computer program. When the computer program is run on an electronic device, the computer program is used to cause the electronic device to perform the steps of any of the above-described training methods for detection and segmentation models.

[0051] This application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. When the processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the above-described training methods for detection and segmentation models.

[0052] The beneficial effects of this application are as follows:

[0053] This application provides a training method, apparatus, electronic device, and storage medium for a detection and segmentation model. The newly proposed detection and segmentation model in this application designs a target detection head and a target segmentation head based on a general deep image feature extraction network, performing multi-task learning without using different network results, effectively reducing computational resource consumption and improving real-time performance. Furthermore, the feature extraction network is a common network for both target detection and target segmentation tasks. After extracting multi-scale attention feature maps of sample images based on this network, different tasks are further performed based on these multi-scale attention feature maps. This effectively ensures the correlation between the target to be detected and the target to be segmented at the visual semantic feature level, efficiently and accurately achieving the detection and segmentation of different targets in the image.

[0054] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0055] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0056] Figure 1AThis is a schematic diagram of a target detection result in a related technology;

[0057] Figure 1B This is a schematic diagram of a target segmentation result in a related technology;

[0058] Figure 2 This is an optional schematic diagram of an application scenario in an embodiment of this application;

[0059] Figure 3 This is a flowchart illustrating a training method for a detection and segmentation model according to an embodiment of this application.

[0060] Figure 4 This is a schematic diagram of the structure of a multi-task joint detection and segmentation model in an embodiment of this application;

[0061] Figure 5 This is a schematic diagram of the structure of another multi-task joint detection and segmentation model in the embodiments of this application;

[0062] Figure 6 This is a flowchart of a deep learning-based multi-task joint detection and segmentation model in an embodiment of this application.

[0063] Figure 7 A flowchart illustrating the implementation of a training method for a detection and segmentation model provided in this application embodiment;

[0064] Figure 8 This is a schematic diagram of target detection box decoding in an embodiment of this application;

[0065] Figure 9 This is a schematic diagram illustrating an example of the results of applying this application in a high-speed scenario.

[0066] Figure 10 This is a schematic diagram illustrating the results of applying this application in an intersection scenario.

[0067] Figure 11 This is a schematic diagram illustrating a model training and application process in an embodiment of this application.

[0068] Figure 12 This is a schematic diagram of an image to be processed in an embodiment of this application.

[0069] Figure 13 This is a schematic diagram of the composition structure of a detection and segmentation model training device according to an embodiment of this application;

[0070] Figure 14 This is a schematic diagram of the hardware structure of an electronic device using an embodiment of this application;

[0071] Figure 15This is a schematic diagram of the hardware structure of another electronic device using an embodiment of this application. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0073] The following describes some of the concepts involved in the embodiments of this application.

[0074] Object detection: refers to the process of detecting a specified category of target (object) in a given image using detection boxes, determining their position and size, such as detecting the specific positions of human bodies, cats, dogs, and cars in an image, and marking them with rectangles of different colors.

[0075] Target segmentation refers to the process of separating a given image into targets of a specified category using a mask. This process involves analyzing which target each pixel in the image belongs to, essentially labeling each pixel to ensure that pixels with the same label share certain visual characteristics. The result of target segmentation is either a set of sub-regions in the image or a set of contour lines extracted from the image (e.g., edge detection). Within a sub-region, each pixel is similar under certain metric or calculated characteristics, such as color, brightness, and texture. Adjacent regions, however, can differ significantly under certain metric.

[0076] Multi-task learning refers to using the same deep learning network architecture to simultaneously learn multiple different tasks. It is an inductive transfer mechanism whose main goal is to improve generalization ability by utilizing domain-specific information implicit in the training signals of multiple related tasks. Multi-task learning achieves this goal by training multiple tasks in parallel using shared representations. In short, multi-task learning can learn about other related problems while learning about one problem, by using shared representations. The detection and segmentation model in this application is a multi-task learning model that jointly learns object detection and object segmentation tasks.

[0077] Multi-scale attention feature map: refers to the feature map obtained by performing multi-scale feature fusion and attention feature extraction on an image. Specifically, the process of multi-scale feature fusion on an image requires representing the image as feature maps of different scales, and then fusing them to obtain a feature map containing multi-scale feature information. Then, an attention weight is learned for each feature in the fused feature map through an attention mechanism, and the learning result is the multi-scale attention feature map.

[0078] Location information: Represents the position of the target to be detected in the image. In the embodiments of this application, for a sample detection target, its corresponding true location information refers to the location information of the sample detection target in the sample image that is pre-annotated according to the actual situation; the predicted location information refers to the location information of the detection box predicted by the model after the target detection is performed on the input sample image by the detection and segmentation model during the model training process.

[0079] Category information: Indicates the category to which the target to be classified belongs. In the embodiments of this application, for a sample segmentation target, its corresponding true category information refers to the category of the sample segmentation target that is pre-labeled according to the actual situation; the predicted category information refers to the category of the sample segmentation target predicted by the model when the detection segmentation model performs target segmentation on the input sample image during the model training process.

[0080] Anchor boxes: These are pre-defined rectangular boxes of different sizes and aspect ratios on an image. In other words, anchor boxes help define the width and height of common targets. During prediction, these pre-defined widths and heights are used to aid in the prediction process. For example, when detecting vehicles, the pre-defined anchor boxes can predict the vehicle's position. By traversing each anchor box in the image, classification and fine-tuning are performed to complete the target detection.

[0081] The embodiments of this application relate to artificial intelligence (AI) and machine learning (ML) technologies, and are designed based on computer vision technology and machine learning in artificial intelligence.

[0082] Artificial intelligence (AI) technology mainly includes computer vision, natural language processing, machine learning / deep learning, autonomous driving, and intelligent transportation. With the research and advancement of AI technology, it is being researched and applied in various fields, such as smart homes, intelligent customer service, virtual assistants, smart speakers, intelligent marketing, autonomous driving, robotics, and smart healthcare. It is believed that with technological development, AI will be applied in more fields and play an increasingly important role. The training method of the detection and segmentation model in this application embodiment can be applied to autonomous driving, medical image recognition, security systems, image processing, video processing, and other fields. Combining AI with target detection and segmentation in these fields can efficiently and accurately detect and segment different targets in images.

[0083] Furthermore, the detection and segmentation model in this embodiment is trained using machine learning or deep learning techniques. After training the detection and segmentation model using these techniques, it can be applied to simultaneously detect and segment different targets in an image.

[0084] The design concept of the embodiments of this application is briefly introduced below:

[0085] Supervised deep learning-based object detection and segmentation methods have seen successful implementations in various fields in recent years, such as object detection and segmentation in autonomous driving scenarios. However, these methods typically use different network architectures to implement object detection and segmentation tasks. For example, Yolov5 networks are used for object detection, while UNet networks are used for object segmentation.

[0086] like Figure 1A The image shown is an example of a Yolov5-based target detection technique. In this example, different targets are marked with different rectangles; for example, solid rectangles mark cars, and dashed rectangles mark buses. Another example... Figure 1B As shown, it is an example diagram of a UNet-based target segmentation technique in related technologies.

[0087] However, in real-world industrial applications, the computational resources of hardware devices such as Graphics Processing Units (GPUs) and Central Processing Units (CPUs) are limited. Using different network structures for image object detection and segmentation would consume significant computational resources, failing to meet the system's high real-time performance requirements. Furthermore, the objects to be detected and segmented often exhibit a certain correlation at the visual semantic feature level. Treating object detection and segmentation as two independent sub-tasks would fail to capture this correlation.

[0088] In view of this, embodiments of this application propose a training method, apparatus, electronic device, and storage medium for a detection and segmentation model. The newly proposed detection and segmentation model in this application designs a target detection head and a target segmentation head based on a general deep image feature extraction network, performing multi-task learning without using different network results, effectively reducing computational resource consumption and improving real-time performance. Furthermore, the feature extraction network is a common network for both target detection and target segmentation tasks. After extracting multi-scale attention feature maps of sample images based on this network, different tasks are further performed based on these multi-scale attention feature maps. This effectively ensures the correlation between the target to be detected and the target to be segmented at the visual semantic feature level, efficiently and accurately achieving the detection and segmentation of different targets in the image.

[0089] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0090] like Figure 2 The diagram shown is an application scenario illustration of an embodiment of this application. The application scenario diagram includes two terminal devices 210 and one server 220.

[0091] In this embodiment, the terminal device 210 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, smart voice interaction devices, smart home appliances, and in-vehicle terminals. The terminal device may have an image processing-related client installed, which can be software (such as a browser, image processing software, etc.), or a webpage, mini-program, etc. The server 220 is a backend server corresponding to the software, webpage, mini-program, etc., or a server specifically used for image processing; this application does not impose specific limitations. The server 220 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0092] It should be noted that the training method of the detection and segmentation model in each embodiment of this application can be executed by an electronic device, which can be a terminal device 210 or a server 220. That is, the method can be executed by the terminal device 210 or the server 220 alone, or by both the terminal device 210 and the server 220. For example, when executed by both the terminal device 210 and the server 220, during the model training phase, a training sample set is constructed through the server 220, and the detection and segmentation model to be trained is iteratively trained multiple times based on the training sample set to obtain the detection and segmentation model, which is then deployed on the server 220.

[0093] In the model application phase, the terminal device 210 acquires the image to be processed and sends it to the server 22, or sends the image's Uniform Resource Locator (URL) to the server 220 for automatic download, and so on. Then, the feature extraction network of the detection and segmentation model deployed on the server 220 performs multi-scale feature fusion and attention feature extraction on the image to be processed, obtaining a target multi-scale attention feature map corresponding to the image to be processed. Next, the multi-scale attention feature map is decoded by the first decoding network in the detection and segmentation model to obtain a first decoded feature map; the multi-scale attention feature map is also decoded by the second decoding network in the detection and segmentation model to obtain a second decoded feature map; the first decoded feature map is input into the target detection head network in the detection and segmentation model, which performs target detection on the image to be processed to obtain the target location information of the target to be detected; the second decoded feature map is input into the target segmentation head network in the detection and segmentation model, which performs target segmentation on the image to be processed to obtain the target category information of the target to be segmented. Finally, the server 220 feeds back the output of the detection and segmentation model (target location information of the target to be detected, target category information of the target to be segmented, etc.) to the terminal device 210, which then displays it to the user.

[0094] It should be noted that the above-described process of the terminal device 210 and server 220 jointly executing the training method of the detection and segmentation model is only one feasible implementation method. In fact, any process of the terminal device 210 and server 220 executing the method is applicable to the embodiments of this application, and will not be described in detail here.

[0095] In one alternative implementation, the terminal device 210 and the server 220 can communicate via a communication network.

[0096] In one alternative implementation, the communication network is a wired network or a wireless network.

[0097] It should be noted that, Figure 2 The examples shown are merely illustrative; in reality, the number of terminal devices and servers is unlimited and is not specifically limited in the embodiments of this application.

[0098] In this embodiment of the application, when there are multiple servers, the multiple servers can form a blockchain, and the server is a node on the blockchain; as in the training method of the detection and segmentation model disclosed in this embodiment of the application, the training sample data, image-related feature information, model output results, etc. involved can all be stored on the blockchain.

[0099] Furthermore, the embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, and autonomous driving.

[0100] The following describes the training method of the detection and segmentation model provided by the exemplary embodiments of this application, in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0101] See Figure 3 The diagram shows an implementation flowchart of a training method for a detection and segmentation model provided in this application. Taking the server as the execution entity as an example, the specific implementation flow of this method is as follows (S31-S32):

[0102] S31: Obtain the training sample set.

[0103] In this embodiment of the application, each training sample in the training sample set includes at least one sample image, and may further include annotation information related to target detection and target segmentation, such as the true location information of the target detected in the sample image and the true category information of the target segmented in the sample.

[0104] Object detection refers to detecting a specified category of object in a given image and labeling the specific location of the specified category of object in the image. For example, using bounding boxes for labeling, the location information of the detected object refers to the coordinates of the center point and the length and width of the bounding box. In the embodiments of this application, for a sample detected object, its corresponding true location information refers to the location information of the sample detected object in the sample image that is pre-labeled according to the actual situation; the predicted location information refers to the location information of the detection box predicted by the model after the detection and segmentation model performs object detection on the input sample image during the model training process, and the detection box is used to label the sample detected object.

[0105] In this context, target segmentation refers to separating the portion corresponding to a target of a specified category from a given image. Accordingly, for a target segmentation sample, its true category information refers to the category of the target sample pre-labeled based on the actual situation; the predicted category information refers to the category of the target sample predicted by the model when the detection and segmentation model performs target segmentation on the input sample image during model training.

[0106] The specific category of the target to be detected or segmented depends on the actual needs. For example, when this model is applied to autonomous driving scenarios, the target categories include, but are not limited to, some or all of the following:

[0107] Pedestrians, vehicles, road surfaces, buildings, traffic signs, and traffic lights.

[0108] It should be noted that the above examples of target categories are only one feasible implementation method. In actual applications, different target categories can be set for different scenarios, and this article does not make any specific limitations.

[0109] In this embodiment of the application, the detection and segmentation model to be trained is subjected to multiple rounds of iterative training based on the training sample set to obtain the target detection and segmentation model; wherein, in each round of iterative training, the following operations can be performed (S32-S34):

[0110] S32: Select training samples from the training sample set and input them into the feature extraction network of the detection and segmentation model to be trained. Based on the feature extraction network, perform multi-scale feature fusion and attention feature extraction on the sample images in the training samples to obtain multi-scale attention feature maps of the sample images.

[0111] In this embodiment, the detection and segmentation model is a multi-task joint learning model. By jointly learning the target detection task and the target segmentation task, it can simultaneously detect and segment different targets in an image. The feature extraction network is a common network for both the target detection task and the target segmentation task. After extracting multi-scale attention feature maps of the sample images based on this network, different tasks are further performed based on these multi-scale attention feature maps. In this way, the correlation between the target to be detected and the target to be segmented at the visual semantic feature level can be effectively guaranteed.

[0112] Furthermore, using a unified, public network for object detection and segmentation of images can effectively reduce the consumption of computing resources and improve real-time performance compared to using different network structures for object detection and segmentation.

[0113] In S32, feature extraction is specifically performed on the sample image to represent it as a multi-scale attention feature map. This multi-scale attention feature map refers to the feature map obtained by fusing feature information from multiple scales of the image and learning the importance of each feature through an attention mechanism.

[0114] Optionally, the feature extraction network includes a convolutional neural subnetwork, a feature pyramid subnetwork, and an attention mechanism subnetwork; based on this structure, S32 can be further divided into the following sub-steps (S321-S323, Figure 3 (Not shown):

[0115] S321: Extract features from sample images based on convolutional subnetworks to obtain image features of sample images.

[0116] In this embodiment, a residual convolutional neural network is used as an example of the convolutional sub-network for detailed explanation.

[0117] See Figure 4 As shown, it is a structural schematic diagram of a multi-task joint detection and segmentation model (hereinafter referred to as the detection and segmentation model) in an embodiment of this application. Figure 4 As shown in S321, the input to this multi-task joint detection and segmentation model can be an image frame (a sample image) acquired by a camera sensor. The image frame is first input to an image feature extraction module (i.e., a convolutional sub-network, also known as a residual convolutional neural network) for preliminary feature extraction to obtain image features.

[0118] S322: Multi-scale feature extraction of image features is performed based on the feature pyramid sub-network, and the extracted image features at multiple different scales are fused to obtain multi-scale fused features;

[0119] Still with Figure 4 For example, in S322, the extracted image features are further input into the feature pyramid module (i.e., the feature pyramid sub-network). The feature pyramid module performs deep fusion of features at different levels extracted by the residual convolutional neural network to enhance the representation ability of the features and obtain multi-scale fused features.

[0120] S323: Based on the attention mechanism sub-network, attention weights are learned for multi-scale fusion features to obtain multi-scale attention feature maps.

[0121] Still with Figure 4 For example, in S323, the extracted multi-scale fusion features are then input into the attention mechanism module (i.e., the attention mechanism sub-network). The attention mechanism module learns an attention weight for each fused feature, thereby extracting a multi-scale attention feature map with important weights.

[0122] S33: Input the multi-scale attention feature map into the target detection head network and the target segmentation head network in the detection and segmentation model respectively to obtain the predicted location information of the target in the sample image and the predicted category information of the target in the sample segmentation.

[0123] In step S33, the relevant processing is performed through the head networks of the two tasks respectively to obtain their respective output results. The predicted location information of the target in the sample image is the output result of the target detection head network, and the predicted category information of the target in the sample segmentation is the output result of the target segmentation head network.

[0124] Considering that the aforementioned feature extraction network involves multi-scale features, in order to expand the receptive field, this application further inputs the multi-scale attention feature map into the feature decoding module for decoding before processing by the head network, such as... Figure 4As shown, the feature decoding module on the left is the first decoding network used to decode the multi-scale attention feature map, and the right side is the second decoding network used to decode the multi-scale attention feature map. The first and second decoding networks have the same network structure but different network parameters.

[0125] In this embodiment, the feature decoding module is described as being composed of a series of continuous deconvolutional neural networks. After the feature decoding module decodes the feature map, the multi-scale attention feature map is upsampled by a certain ratio to increase its size, thereby obtaining the final decoded feature.

[0126] Optionally, before step S33, the multi-scale attention feature map needs to be decoded by the first decoding network to obtain the first decoded feature map; and the multi-scale attention feature map needs to be decoded by the second decoding network to obtain the second decoded feature map, such as... Figure 4 As shown. Then, the first decoded feature map is input into the target detection head network, which performs target detection on the sample image to obtain the predicted location information of the detected target; the second decoded feature map is input into the target segmentation head network, which performs target segmentation on the sample image to obtain the predicted category information of the segmented target.

[0127] After obtaining these prediction results, the corresponding loss function can be constructed, and the model parameters can be adjusted, as follows:

[0128] S34: The parameters of the detection and segmentation model are adjusted using a loss function constructed based on predicted location information and predicted category information.

[0129] In implementing step S34, this application embodiment proposes several different model training methods:

[0130] Optionally, each training sample in the training sample set also includes the true location information of the target detected in the corresponding sample image and the true category information of the target segmented in the sample; in each iteration of training, the following two training methods are used:

[0131] Training Method 1: Construct a detection loss function based on the difference between predicted location information and true location information; construct a segmentation loss function based on the difference between predicted category information and true category information; alternately use the detection loss function and the segmentation loss function to adjust the parameters of the detection and segmentation model.

[0132] One of the training methods is an alternating training approach.

[0133] Specifically, since the detection and segmentation model in this application is a multi-task joint learning model, the training objectives of each task are inconsistent when jointly learning the target detection task and the target segmentation task. Therefore, different loss functions can be used alternately to adjust the different network parameters in the detection and segmentation model respectively.

[0134] In one alternative implementation, the process of alternatingly training the model using different loss functions is as follows:

[0135] Based on the detection loss function, the parameters of the feature extraction network, the first decoding network, and the target detection head network in the detection and segmentation model are updated; then, based on the segmentation loss function, the parameters of the feature extraction network, the second decoding network, and the target segmentation head network in the detection and segmentation model are updated.

[0136] It should be noted that the above description of the alternating training process only refers to the loss function. In fact, alternating training can also be used directly before step S34. For example, alternating training is also used in steps S32 and S33.

[0137] Specifically, in step S32, when selecting training samples from the training sample set, the same training samples can be selected, which contain both sample detection target a and sample segmentation target b. In this way, there is no need for alternating training in steps S32 and S33.

[0138] Alternatively, different training samples can be selected, such as training sample 1 containing sample detection target a, and training sample 2 containing sample segmentation target b. In this way, in step S32, training can be performed alternately: selecting training sample 1 and inputting it into the feature extraction network of the detection and segmentation model; based on the feature extraction network, multi-scale feature fusion and attention feature extraction are performed on the sample image in training sample 1 to obtain a multi-scale attention feature map 1 of the sample image; then, step S33 is executed: inputting the multi-scale attention feature map 1 into the target detection head network of the detection and segmentation model to obtain the predicted location information of sample detection target a in the sample image. Then, step S34 is executed: based on the difference between the predicted location information and the corresponding real location information, a detection loss function is constructed; based on the detection loss function, the parameters of the feature extraction network, the first decoding network, and the target detection head network in the detection and segmentation model are updated.

[0139] Then, based on training sample 2, the above steps S32-S34 are repeated. The specific process is as follows: In step S32, training sample 2 is selected and input into the feature extraction network of the detection and segmentation model. Based on the feature extraction network, multi-scale feature fusion and attention feature extraction are performed on the sample images in training sample 2 to obtain the multi-scale attention features of the sample images. Figure 2Then, step S33 is executed: multi-scale attention features are applied. Figure 2 The target segmentation head network in the detection and segmentation model is input to obtain the predicted category information of the target b in the sample image. Then, step S34 is executed: based on the difference between the predicted category information and the corresponding true category information, a segmentation loss function is constructed; based on the segmentation loss function, the parameters of the feature extraction network, the second decoding network and the target segmentation head network in the detection and segmentation model are updated.

[0140] In other words, alternating training refers to the process of iteratively training the network. First, the network is propagated forward using object detection data, then the gradient is backpropagated to update the parameters of the feature extraction network, the first decoding network, and the object detection head network. Then, the network is propagated forward using object segmentation data, and the gradient is backpropagated to update the parameters of the feature extraction network, the second decoding network, and the object segmentation head network. This alternating training process is repeated until the network converges.

[0141] When adjusting model parameters, if the detection loss function is Ldet and the segmentation loss function is Lseg, then during alternating training, the model will be based on Ldet... Figure 5 The network parameters of the middle Sa part are adjusted, that is, the parameters of the feature extraction network, the first decoding network, and the object detection head network are adjusted; based on Lseg, Figure 5 The network parameters of the Sb part are adjusted, specifically the parameters of the feature extraction network, the second decoding network, and the target segmentation head network. Figure 5 This is a schematic diagram of the structure of another multi-task joint detection and segmentation model (hereinafter referred to as the detection and segmentation model) in the embodiments of this application.

[0142] Training Method 2: Construct a detection loss function based on the difference between predicted and true location information; construct a segmentation loss function based on the difference between predicted and true category information; construct a total loss function based on the detection and segmentation loss functions; and use the total loss function to adjust the parameters of the detection and segmentation model.

[0143] Among them, training method two is an end-to-end training method. That is, this training method does not need to consider the different loss functions corresponding to different training objectives of different tasks, but rather adjusts the parameters of the model as a whole based on a total loss function.

[0144] For example, if the detection loss function is Ldet and the segmentation loss function is Lseg, then the total loss function Lall = γ1Ldet + γ2Lseg, which is also the multi-task loss function. Here, γ1 and γ2 are the weight ratio coefficients of the detection loss function and the segmentation loss function, which can be flexibly set according to the actual situation, such as setting γ1 to 0.5 and γ2 to 0.5.

[0145] It should be noted that the two training methods listed above can also be combined. For example, when training the detection and segmentation model to be trained for 10 rounds, training method one can be used in each round; training method two can be used in each round; or training method one can be used for some rounds and training method two for some rounds, such as using training method one in the first 1-6 rounds and training method two in the last 4 rounds, and so on. Any combination method is applicable to the embodiments of this application, and will not be described in detail here.

[0146] The specific calculation process of the detection loss function is explained below:

[0147] In this embodiment, the object detection head network can output not only the predicted location information of the detected target in the sample, but also a predicted probability distribution map of the same size as the original image (i.e., the sample image). For example, if the input sample image is 512*512, the network will also output a predicted probability distribution map of size 512*512*the number of detection categories. The number of channels in this probability distribution map is the same as the number of detection categories. The meaning of each pixel in the probability distribution map indicates the probability that the corresponding pixel position in the original input sample image belongs to each detection category, and one channel corresponds to one detection category.

[0148] For example, if there are three preset detection categories: pedestrians, electric vehicles, and cars, the target detection head network can output a 512*512*3 predicted probability distribution map corresponding to the sample image. The "*3" indicates that it corresponds to the three detection categories respectively. Therefore, it contains a 512*512*1 probability distribution map corresponding to the target detection category (i.e., the actual category to which the sample detection target belongs), which is the first probability distribution map. The pixel value of each pixel in the first probability distribution map represents the first predicted probability of the category to which the sample detection target belongs.

[0149] One possible implementation involves constructing a classification loss function based on each first predicted probability in the first probability distribution map; constructing a location loss function based on the difference between predicted location information and true location information; and constructing a detection loss function based on the classification loss function and the location loss function.

[0150] In this embodiment of the application, the object detection task also needs to classify and identify each object in the image. The classification loss function can be represented by Lclass, which can characterize the accuracy of classification and identification in the task.

[0151] Specifically, Lclass can be calculated using focal loss. One possible formula for calculating Lclass is as follows:

[0152] Lclass=-(1-Pt1)γ a *log(Pt1);

[0153] Where, γ a Pt1 is a constant, Pt1 is the first predicted probability, and log is the logarithmic function.

[0154] Furthermore, the location loss function can be represented by an Lbox, which characterizes the accuracy of the location prediction of the classified target in the task.

[0155] Specifically, the Lbox can use L1 loss, and one possible formula for calculating the Lbox is as follows:

[0156] Lbox = ∑|Yi-Yp|;

[0157] Where Yi and Yp represent the true coordinates (i.e., true location information) and predicted coordinates (i.e., predicted location information) of the target detected in the sample, respectively.

[0158] When constructing a detection loss function based on Lclass and Lbox, a weighted summation method can be used, as described below:

[0159] Ldet = α1Lclass + α2Lbox;

[0160] Where α1 and α2 are different weight ratio coefficients in the Ldet loss function, which can be determined according to the actual situation, such as setting α1 to 0.5 and α2 to 0.5.

[0161] In another optional implementation, for the object detection head network, if the input sample image size is 512*512, the network can also output a 512*512*(number of detection categories + 1) prediction probability distribution map, where the number of channels of the probability distribution map is the number of detection categories + 1. For example, if the preset detection categories are still the 3 listed above, the object detection head network can output a 512*512*4 prediction probability distribution map corresponding to the sample image, where "*4" means "*(3+1)". The first 3 channels are as described above, corresponding to the 3 detection categories respectively, while the remaining channel represents a second probability distribution map. The pixel value of each pixel in the second probability distribution map represents the second predicted probability that the pixel is a detection target (belonging to any preset detection category represents that the pixel is a detection target).

[0162] Optionally, before calculating the detection loss function, a target loss function can be further constructed based on each second prediction probability in the second probability distribution map.

[0163] In the embodiments of this application, the target loss function can be represented by Lobj, which can characterize the accuracy of target detection in the task.

[0164] Specifically, Lobj can be calculated using focal loss. One possible formula for calculating Lobj is as follows:

[0165] Lobj=-(1-Pt2)γ b *log(Pt2);

[0166] Where, γ b Pt2 is a constant, Pt2 is the second prediction probability, and log is the logarithmic function.

[0167] Therefore, when constructing the detection loss function, the target loss function, classification loss function, and location loss function can still be obtained by weighted summation, as described below:

[0168] Ldet=β1Lclass++β2Lobj+β3Lbox;

[0169] Wherein, β1, β2, and β3 are different weight ratio coefficients in the Ldet loss function, which can be determined according to the actual situation, such as setting β1 to 0.35, β2 to 0.3, and β to 0.35.

[0170] It should be noted that the calculation of the detection loss function listed above is only an example. Any method of constructing the detection loss function based on the difference between the predicted location information and the actual location information of the sample is applicable to the embodiments of this application, and this document does not make any specific limitations.

[0171] Optionally, the segmentation loss function Lseg can be cross-entropy loss. One possible formula for calculating Lseg is as follows:

[0172] Lseg = -Yic * log(Pic);

[0173] Where Yic is the true class c of the sample segmentation target i, Pic is the probability that the sample segmentation target i belongs to class c, and log is the logarithmic function.

[0174] It should be noted that the calculation of the segmentation loss function listed above is only an example. Any method of constructing a segmentation loss function based on the difference between the predicted category information and the true category information of the sample is applicable to the embodiments of this application, and this document does not impose any specific limitations.

[0175] Optionally, the target of sample detection and the target of sample segmentation belong to different categories.

[0176] In this embodiment, the detection and segmentation model does not require the target categories of the detection task and the segmentation task to be strictly consistent. Therefore, it is not limited to detecting and segmenting targets of the same category. For example, it can detect and segment pedestrians and vehicles at the same time. This application can realize the detection and segmentation of targets of different categories. For example, it can detect pedestrians and vehicles, segment road surfaces and buildings, detect traffic signs and traffic lights in images, and segment vehicles, pedestrians, road surfaces and other elements in images.

[0177] In summary, please refer to Figure 6 As shown, it is a flowchart of a deep learning-based multi-task joint detection and segmentation model in an embodiment of this application. Taking the image frame acquired by the camera as the model input as an example, by combining the alternating training strategy and multi-task learning loss function listed above, this application can solve the problem of performing object detection and segmentation tasks simultaneously in any domain scenario, and can effectively complete the object detection and object segmentation tasks for different categories of elements in the image.

[0178] The training method of the detection and segmentation model in this application embodiment is further described below from the perspective of model application:

[0179] See Figure 7 The diagram shown is an implementation flowchart of a training method for a detection and segmentation model provided in this application. Taking the server as the execution entity as an example, the specific implementation flow of this method is as follows:

[0180] S71: Input the image to be processed into the trained detection and segmentation model.

[0181] The detection and segmentation model is trained using any of the detection and segmentation model training methods described in this application, and will not be repeated here.

[0182] S72: A feature extraction network based on a detection and segmentation model performs multi-scale feature fusion and attention feature extraction on the image to be processed, and obtains the target multi-scale attention feature map corresponding to the image to be processed.

[0183] Optionally, step S72, similar to S32 above, can also be divided into the following two sub-steps when obtaining the target multi-scale attention feature map:

[0184] S721: Based on the convolutional sub-network in the detection and segmentation model, feature extraction is performed on the image to be processed to obtain the target image features of the image to be processed;

[0185] S722: Based on the feature pyramid sub-network in the detection and segmentation model, multi-scale feature extraction of target image features is performed, and the extracted target image features at multiple different scales are fused to obtain target multi-scale fused features.

[0186] S723: Based on the attention mechanism sub-network in the detection and segmentation model, attention weights are learned for the multi-scale fusion features of the target to obtain the multi-scale attention feature map of the target.

[0187] It should be noted that the specific implementation methods of S721-S723 are the same as those of S321-S323 above, and the repeated parts will not be described again.

[0188] S73: Input the target multi-scale attention feature map into the target detection head network and the target segmentation head network in the detection and segmentation model respectively to obtain the target location information and target category information of the target to be segmented in the image to be processed.

[0189] Similarly, before executing step S73, the target multi-scale attention feature map can also be decoded based on the decoding feature module. Based on this, the specific process is as follows ( Figure 7 (Not shown):

[0190] S731a: The first decoded feature map of the target is obtained by decoding the multi-scale attention feature map of the target through the first decoding network in the detection segmentation model;

[0191] S731b: The target's multi-scale attention feature map is decoded by the second decoding network in the detection segmentation model to obtain the target's second decoded feature map;

[0192] S732a: Input the first decoded feature map of the target into the target detection head network in the detection and segmentation model, and use the target detection head network to perform target detection on the image to be processed to obtain the target location information of the target to be detected;

[0193] S732b: Input the second decoded feature map of the target into the target segmentation head network in the detection and segmentation model. The target segmentation head network performs target segmentation on the image to be processed and obtains the target category information of the target to be segmented.

[0194] In this method, the target to be segmented and the target to be segmented belong to different categories, so as to achieve simultaneous detection and segmentation of targets of different categories in the image.

[0195] It should be noted that the specific implementation methods of the above steps can also refer to the relevant processing of sample images, and the repeated parts will not be repeated.

[0196] The following describes the process of obtaining the target location information of the target to be segmented in the image based on the target detection head network in the detection and segmentation model:

[0197] In this embodiment of the application, after the model is trained, the trained detection and segmentation model is used to predict the image to be processed. The multi-task joint object detection head network outputs prediction results (predicted boxes) based on prior boxes. Figure 8 As shown, this is a schematic diagram of target detection box decoding in an embodiment of this application. The solid-lined boxes are the predicted boxes output by the multi-task joint target detection head network. bx, by, bh, and bw represent the center coordinates and dimensions (length and width) of the target predicted by the target detection head network in the image, respectively. cx and cy are the distances between the grid occupied by the center of the target and the top-left corner of the grid. The dashed-lined boxes are the prior boxes, and ph and pw are the length and width of the prior boxes. δ represents the sigmoid activation function. The center coordinates and dimensions (length and width) of the target can be calculated using the following formula:

[0198] bx = 2δ(tx) – 0.5 + cx;

[0199] by=2δ(ty)–0.5+cy;

[0200] bw=pw(2δ(tw)) 2 ;

[0201] bh=ph(2δ(th)) 2 ;

[0202] Where δ(x)=1 / (1+e -x ).

[0203] In this embodiment, the prior box is a set or several sets of fixed reference boxes with different scales and aspect ratios. Each reference box is responsible for detecting targets with an intersection-union ratio (IU) greater than a threshold (preset values ​​of 0.5 or 0.7). Using the prior box mechanism can significantly improve the network's recall rate.

[0204] Furthermore, since the prediction results output by the object detection head network are based on prior boxes, there will be a large number of redundant and overlapping prediction boxes. Therefore, non-maximum suppression methods are needed to filter all the prediction result boxes in the output.

[0205] Specifically, all predicted bounding boxes are grouped into set B, and the remaining predicted bounding boxes are grouped into set S. Each time, the predicted bounding box with the highest confidence in set B is added to set S. Then, the intersection-union ratio (IUR) of this predicted bounding box is calculated with all the remaining boxes in B. If the IUR is greater than a first threshold (0.45 in this application), the corresponding predicted bounding box is filtered out. The IUR calculation formula is as follows:

[0206] IOU = (A∩B) / (A∪B);

[0207] Here, A and B are two prediction boxes, A∩B is used to calculate the intersection area of ​​A and B, and A∪B is used to calculate the union area of ​​A and B.

[0208] Repeat the above process until set B is empty or the confidence of all remaining prediction boxes is less than the second threshold (0.25 in this application).

[0209] For example, the initial set B has four prediction boxes, denoted as candidate prediction boxes 1, 2, 3, and 4. The confidence level of candidate prediction box 1 is 0.8, that of candidate prediction box 2 is 0.7, that of candidate prediction box 3 is 0.6, and that of candidate prediction box 4 is 0.3. In the first iteration, candidate prediction box 1 has the highest confidence level, so it is added to set S. At this point, set B contains candidate prediction boxes 2-4. The intersection-union ratios (IURs) of candidate prediction boxes 1 and 2-4 are calculated. The IUR 1 for candidate prediction box 2 is 0.9, that for candidate prediction box 3 is 0.1, and that for candidate prediction box 4 is 0.3. If the cross-union ratio (CUN) is 0.2 and the first threshold is 0.45, then candidate predicted box 2 corresponding to the cross-union ratio 1 is deleted, and the next iteration begins. The second iteration filters: the candidate predicted boxes in the current set B are candidate predicted box 3 and candidate predicted box 4. Candidate predicted box 3 has the highest confidence, so it is added to set S. The CUN of candidate predicted box 3 and candidate predicted box 4 is calculated to be 0.6, which is greater than the first threshold, so candidate predicted box 4 is deleted. At this point, the number of candidate predicted boxes in set B is zero, the iteration stops, and candidate predicted box 1 and candidate predicted box 3 in set S are taken as the final predicted boxes. Each of these two predicted boxes can correspond to a detection target.

[0210] The following describes the process of obtaining target category information of the target to be segmented in the image based on the target segmentation head network in the detection and segmentation model:

[0211] In this embodiment, after the model is trained, the trained detection and segmentation model is used to predict the image to be processed. The multi-task joint object segmentation head network outputs a prediction probability distribution map of the same size as the image to be processed. For example, if the image to be processed is 512*512, the network will also output a prediction probability distribution map of size 512*512*number of segmentation categories. In this probability distribution map, each pixel represents the probability that the corresponding pixel position in the original input image to be processed belongs to each segmentation category.

[0212] Specifically, the segmentation prediction category for each pixel in the input 512*512 image can be obtained using the following formula:

[0213] Pcls = argmax(P f (axis);

[0214] Where Pcls represents the predicted segmentation category for each pixel in the image to be processed, argmax is the index function that takes the maximum value, and P... f This is the predicted probability distribution map output by the target segmentation head network, where `axis` is the dimensional index of the segmentation category in the predicted probability distribution map. The formula calculates the segmentation category corresponding to the maximum predicted probability value for each pixel along the dimension of the number of segmentation categories.

[0215] After decoding the output of the target detection and segmentation head network through the above process, the results can be merged using OpenCV development tools to visualize the detection and segmentation results. For example... Figure 9 The diagram shown is an example of the results of applying this application in a high-speed scenario. Wherein, Figure 9 This example uses the target to be detected and the target to be segmented as belonging to the same category. The shading represents the segmentation result, while the bounding box represents the detection result. For different types of vehicles, different forms of bounding boxes can be used for marking, such as different colors or different lines. Figure 9 This is illustrated using different lines as examples.

[0216] For example Figure 10 The diagram shown is an example of the results of applying this application in an intersection scenario. Figure 9 similar, Figure 10 This example uses the target to be detected and the target to be segmented as being of the same category. The shading represents the segmentation result, while the bounding box represents the detection result. For different types of vehicles, different forms of bounding boxes can be used for marking, such as different colors or different lines. Figure 10 This is also illustrated using different lines as examples.

[0217] The following is a brief explanation of the overall process of model training and model application:

[0218] See Figure 11 As shown, this is a flowchart illustrating a model training and application process in an embodiment of this application. Taking the server as the execution entity as an example, the specific implementation process of this method is as follows:

[0219] Step S1101: The server obtains the training sample set;

[0220] Step S1102: The server selects training samples from the training sample set, inputs the target detection-related data from the selected training samples into the detection and segmentation model, and obtains the predicted location information of the target in the corresponding sample image.

[0221] Step S1103: The server constructs a detection loss function based on the difference between the predicted location information and the actual location information, and updates the parameters of the feature extraction network, the first decoding network and the target detection head network in the detection segmentation model based on the detection loss function;

[0222] Step S1104: The server selects training samples from the training sample set, inputs the target segmentation related data from the selected training samples into the detection and segmentation model, and obtains the predicted category information of the target segmentation in the corresponding sample image.

[0223] Step S1105: The server constructs a segmentation loss function based on the difference between the predicted category information and the true category information, and updates the parameters of the feature extraction network, the second decoding network and the target segmentation head network in the detection and segmentation model based on the segmentation loss function.

[0224] Step S1106: The server determines whether the iteration stopping condition has been met. If yes, it executes step S1107; otherwise, it returns to step S1102.

[0225] The iteration stopping conditions include, but are not limited to, some or all of the following: the number of iterations reaches a preset upper limit, and the model converges.

[0226] Step S1107: The server uses the current detection and segmentation model as the detection and segmentation model;

[0227] Step S1108: The server inputs the image to be processed into the trained detection and segmentation model;

[0228] Step S1109: The server extracts features from the image to be processed based on the convolutional sub-network in the detection and segmentation model to obtain the image features of the image to be processed;

[0229] Step S1110: The server performs multi-scale feature extraction on image features based on the feature pyramid sub-network in the detection and segmentation model, and fuses the extracted image features at multiple different scales to obtain multi-scale fused features;

[0230] Step S1111: The server learns attention weights for the multi-scale fusion features based on the attention mechanism sub-network in the detection and segmentation model to obtain a multi-scale attention feature map;

[0231] Step S1112a: The server decodes the multi-scale attention feature map by detecting the first decoding network in the segmentation model to obtain the first decoded feature map;

[0232] Step S1112b: The server decodes the multi-scale attention feature map by detecting the second decoding network in the segmentation model to obtain the second decoded feature map;

[0233] Step S1113a: The server inputs the first decoded feature map into the target detection head network in the detection and segmentation model, and performs target detection on the image to be processed through the target detection head network to obtain the target location information of the target to be detected;

[0234] Step S1113b: The server inputs the second decoded feature map into the target segmentation head network in the detection and segmentation model. The target segmentation head network performs target segmentation on the image to be processed to obtain the target category information of the target to be segmented.

[0235] The following example uses the case where the target to be detected and the target to be segmented belong to different categories. (See below) Figure 12 As shown, it is a schematic diagram of an image to be processed in an embodiment of this application.

[0236] and Figure 9 and Figure 10 The difference is, Figure 12 In the image to be processed shown, the detection type of the target to be detected is trees, and the segmentation type of the target to be segmented is vehicles. Obviously, the target to be detected is a static object, and the target to be segmented is a dynamic object. These two types of targets are completely different from each other. Unlike the difference between bicycles and electric vehicles, or buses and cars, bicycles, electric vehicles, buses, and cars all belong to vehicles. In addition, it is also different from the difference between vehicles and pedestrians, as vehicles and pedestrians are both dynamic objects.

[0237] Clearly, the method in the embodiments of this application can be used to detect and segment different targets in an image.

[0238] Based on the same inventive concept, embodiments of this application also provide a detection and segmentation model training device. For example... Figure 13 As shown, this is a schematic diagram of the structure of the detection and segmentation model training device 1300, which may include:

[0239] The sample acquisition unit 1301 is used to acquire the training sample set;

[0240] The first feature extraction unit 1302 is used to select training samples in the training sample set, input them into the feature extraction network of the detection and segmentation model to be trained, and perform multi-scale feature fusion and attention feature extraction on the sample images in the training samples based on the feature extraction network to obtain multi-scale attention feature maps of the sample images.

[0241] The second feature extraction unit 1303 is used to input the multi-scale attention feature map into the target detection head network and the target segmentation head network in the detection and segmentation model respectively, to obtain the predicted location information of the target in the sample image and the predicted category information of the target in the sample segmentation.

[0242] The parameter adjustment unit 1304 is used to adjust the parameters of the detection and segmentation model using a loss function constructed based on predicted location information and predicted category information.

[0243] Optionally, the feature extraction network includes a convolutional neural subnetwork, a feature pyramid subnetwork, and an attention mechanism subnetwork;

[0244] The first feature extraction unit 1302 is specifically used for:

[0245] Feature extraction of sample images is performed based on convolutional subnetworks to obtain the image features of the sample images;

[0246] Multi-scale feature extraction of image features is performed based on the feature pyramid sub-network, and the extracted image features at multiple different scales are fused to obtain multi-scale fused features;

[0247] Attention weights are learned for multi-scale fusion features based on an attention mechanism sub-network to obtain multi-scale attention feature maps.

[0248] Optionally, the detection and segmentation model may further include: a first decoding network and a second decoding network; the first decoding network and the second decoding network have the same network structure but different network parameters;

[0249] Before inputting the multi-scale attention feature map into the object detection head network and object segmentation head network of the detection and segmentation model, the second feature extraction unit 1303 is also used for:

[0250] The first decoded feature map is obtained by decoding the multi-scale attention feature map through the first decoding network;

[0251] The second decoding feature map is obtained by decoding the multi-scale attention feature map through a second decoding network.

[0252] Optionally, the second feature extraction unit 1303 is specifically used for:

[0253] The first decoded feature map is input into the target detection head network, and the target detection head network performs target detection on the sample image to obtain the predicted location information of the target in the sample.

[0254] The second decoded feature map is input into the target segmentation head network, which then performs target segmentation on the sample image to obtain the predicted category information of the target segmentation.

[0255] Optionally, each training sample in the training sample set also includes the true location information of the target detected in the corresponding sample image and the true category information of the target segmented in the sample; the parameter adjustment unit 1304 is specifically used for:

[0256] A detection loss function is constructed based on the difference between predicted location information and true location information; a segmentation loss function is constructed based on the difference between predicted category information and true category information.

[0257] The parameters of the detection and segmentation model can be adjusted using at least one of the following methods:

[0258] Alternately use the detection loss function and the segmentation loss function to adjust the parameters of the detection and segmentation model;

[0259] The parameters of the detection and segmentation model are adjusted by using a total loss function constructed based on the detection loss function and the segmentation loss function.

[0260] Optionally, the detection and segmentation model further includes: a first decoding network and a second decoding network for decoding the multi-scale attention feature map; the first decoding network and the second decoding network have the same network structure but different network parameters;

[0261] The parameter adjustment unit 1304 is specifically used for:

[0262] Based on the detection loss function, the parameters of the feature extraction network, the first decoding network, and the object detection head network in the detection and segmentation model are updated.

[0263] The parameters of the feature extraction network, the second decoding network, and the target segmentation head network in the detection and segmentation model are updated based on the segmentation loss function.

[0264] Optionally, before constructing the detection loss function based on the difference between the predicted location information and the true location information, the parameter adjustment unit 1304 is also used for:

[0265] The first probability distribution map corresponding to the sample image is obtained based on the object detection head network. The pixel value of each pixel in the first probability distribution map represents the first predicted probability of the category to which the sample detection target belongs.

[0266] Based on the difference between predicted and actual location information, a detection loss function is constructed, including:

[0267] A classification loss function is constructed based on each first predicted probability;

[0268] Based on the difference between predicted and actual location information, a location loss function is constructed.

[0269] A detection loss function is constructed based on the classification loss function and the location loss function.

[0270] Optionally, before determining the detection loss function based on the classification loss function and the location loss function, the parameter adjustment unit 1304 is also used for:

[0271] The second probability distribution map corresponding to the sample image is obtained based on the object detection head network. The pixel value of each pixel in the second probability distribution map represents the second predicted probability of the pixel as the detected object.

[0272] The parameter adjustment unit 1304 is specifically used for:

[0273] Construct a target loss function based on each second prediction probability;

[0274] The detection loss function is obtained by weighted summation of the target loss function, classification loss function, and location loss function.

[0275] Optionally, the target of sample detection and the target of sample segmentation belong to different categories.

[0276] Optionally, the device also includes:

[0277] The model application unit 1305 is used to input the image to be processed into the trained detection and segmentation model;

[0278] Based on the feature extraction network of the detection and segmentation model, multi-scale feature fusion and attention feature extraction are performed on the image to be processed to obtain the target multi-scale attention feature map corresponding to the image to be processed.

[0279] The target multi-scale attention feature map is input into the target detection head network and the target segmentation head network in the detection and segmentation model, respectively, to obtain the target location information and target category information of the target to be segmented in the image to be processed; the target to be segmented and the target to be segmented belong to different categories.

[0280] The newly proposed detection and segmentation model in this application is based on a general deep image feature extraction network, with a target detection head and a target segmentation head designed for multi-task learning. This eliminates the need for different network results, effectively reducing computational resource consumption and improving real-time performance. Furthermore, the feature extraction network is a common network for both target detection and target segmentation tasks. After extracting multi-scale attention feature maps of sample images based on this network, different tasks are further performed based on these multi-scale attention feature maps. This effectively ensures the correlation between the target to be detected and the target to be segmented at the visual semantic feature level, achieving efficient and accurate detection and segmentation of different targets in the image.

[0281] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0282] Having introduced the training method and apparatus for the detection and segmentation model according to exemplary embodiments of this application, we will now introduce an electronic device according to another exemplary embodiment of this application.

[0283] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0284] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. In one embodiment, the electronic device may be a server, such as... Figure 2 The server 220 is shown. In this embodiment, the structure of the electronic device can be as follows: Figure 14 As shown, it includes a memory 1401, a communication module 1403, and one or more processors 1402.

[0285] The memory 1401 is used to store computer programs executed by the processor 1402. The memory 1401 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0286] Memory 1401 may be volatile memory, such as random-access memory (RAM); memory 1401 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1401 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1401 may be a combination of the above-described memories.

[0287] The processor 1402 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1402 is used to implement the training method of the detection and segmentation model described above when it calls the computer program stored in the memory 1401.

[0288] The communication module 1403 is used to communicate with terminal devices and other servers.

[0289] This application embodiment does not limit the specific connection medium between the memory 1401, communication module 1403, and processor 1402. This application embodiment... Figure 14 The memory 1401 and the processor 1402 are connected via a bus 1404, and the bus 1404 is in Figure 14 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 1404 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 14 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0290] The memory 1401 stores a computer storage medium, which stores computer-executable instructions. These instructions are used to implement the training method of the detection and segmentation model according to embodiments of this application. The processor 1402 is used to execute the aforementioned training method of the detection and segmentation model, such as... Figure 3 or Figure 7 As shown.

[0291] In another embodiment, the electronic device may also be other electronic devices, such as... Figure 2 The terminal device 210 is shown. In this embodiment, the electronic device can be structured as follows: Figure 15 As shown, it includes components such as: communication component 1510, memory 1520, display unit 1530, camera 1540, sensor 1550, audio circuit 1560, Bluetooth module 1570, processor 1580, etc.

[0292] The communication component 1510 is used to communicate with the server. In some embodiments, it may include a Circuit-Based Wireless Fidelity (WiFi) module, which is a short-range wireless transmission technology. Electronic devices can use the WiFi module to help users send and receive information.

[0293] The memory 1520 can be used to store software programs and data. The processor 1580 executes various functions of the terminal device 210 and performs data processing by running the software programs or data stored in the memory 1520. The memory 1520 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The memory 1520 stores an operating system that enables the terminal device 210 to run. In this application, the memory 1520 may store the operating system and various application programs, and may also store a computer program that executes the training method of the detection segmentation model of the embodiments of this application.

[0294] The display unit 1530 can also be used to display information input by the user or information provided to the user, as well as various menus of the terminal device 210, forming a graphical user interface (GUI). Specifically, the display unit 1530 may include a display screen 1532 disposed on the front of the terminal device 210. The display screen 1532 may be configured as a liquid crystal display, a light-emitting diode, or the like. The display unit 1530 can be used to display image interfaces, etc., as described in the embodiments of this application.

[0295] The display unit 1530 can also be used to receive input digital or character information and generate signal inputs related to user settings and function control of the terminal device 210. Specifically, the display unit 1530 may include a touch screen 1531 disposed on the front of the terminal device 210, which can collect touch operations of the user on or near it, such as clicking buttons, dragging scroll boxes, etc.

[0296] The touchscreen 1531 can be placed on top of the display screen 1532, or the touchscreen 1531 and the display screen 1532 can be integrated to realize the input and output functions of the terminal device 210. After integration, it can be referred to as a touch display screen. In this application, the display unit 1530 can display the application and the corresponding operation steps.

[0297] Camera 1540 can be used to capture still images, which users can then share via an application. There can be one or multiple cameras 1540. An object is projected onto a photosensitive element through a lens, generating an optical image. This photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the processor 1580 for conversion into a digital image signal.

[0298] The terminal device may also include at least one sensor 1550, such as an accelerometer 1551, a proximity sensor 1552, a fingerprint sensor 1553, and a temperature sensor 1554. The terminal device may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, and motion sensor.

[0299] Audio circuitry 1560, speaker 1561, and microphone 1562 provide an audio interface between the user and terminal device 210. Audio circuitry 1560 converts received audio data into electrical signals, which are then transmitted to speaker 1561, where they are converted into sound signals for output. Terminal device 210 may also be equipped with volume buttons for adjusting the volume of the sound signal. Conversely, microphone 1562 converts collected sound signals into electrical signals, which are then received by audio circuitry 1560, converted back into audio data, and output to communication component 1510 for transmission to, for example, another terminal device 210, or to memory 1520 for further processing.

[0300] The Bluetooth module 1570 is used to interact with other Bluetooth devices that also have a Bluetooth module via the Bluetooth protocol. For example, a terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smartwatch) that also has a Bluetooth module through the Bluetooth module 1570, thereby exchanging data.

[0301] The processor 1580 is the control center of the terminal device, connecting various parts of the terminal through various interfaces and lines. It executes various functions and processes data by running or executing software programs stored in the memory 1520 and calling data stored in the memory 1520. In some embodiments, the processor 1580 may include one or more processing units; the processor 1580 may also integrate an application processor and a baseband processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the baseband processor mainly handles wireless communication. It is understood that the baseband processor may not be integrated into the processor 1580. In this application, the processor 1580 can run the operating system, applications, user interface display and touch response, and the training method of the detection and segmentation model in the embodiments of this application. Furthermore, the processor 1580 is coupled to the display unit 1530.

[0302] In some possible implementations, various aspects of the training method for the detection and segmentation model provided in this application can also be implemented as a program product, comprising a computer program. When the program product is run on an electronic device, the computer program causes the electronic device to perform the steps in the training method for the detection and segmentation model according to various exemplary embodiments of this application described above. For example, the electronic device can perform actions such as... Figure 3 or Figure 7 The steps are shown in the figure.

[0303] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0304] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0305] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0306] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0307] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer program can execute entirely on the user's electronic device, partially on the user's electronic device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0308] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0309] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0310] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing a computer-usable computer program.

[0311] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0312] These computer program commands may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the commands stored in the computer-readable storage medium produce an article of manufacture including command means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0313] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0314] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A training method for a detection segmentation model, characterized in that, The method includes: Obtain a training sample set; each training sample in the training sample set includes at least one sample image, and each sample image includes a sample detection target and a sample segmentation target, wherein the target category to which the sample detection target and the target category to which the sample segmentation target belong are different. Based on the training sample set, the detection and segmentation model to be trained is subjected to multiple rounds of iterative training to obtain a trained detection and segmentation model. In each round of iterative training, the parameters of the detection and segmentation model are adjusted by alternating training. The following operations are performed in each round of iterative training: Select training samples from the training sample set and input them into the feature extraction network of the detection and segmentation model to be trained. Based on the feature extraction network, perform multi-scale feature fusion and attention feature extraction on the sample images in the training samples to obtain the multi-scale attention feature map of the sample images. The multi-scale attention feature map is input into the first decoding network and the second decoding network in the detection and segmentation model for decoding to obtain the first decoding feature map and the second decoding feature map. The first decoded feature map is input into the target detection head network in the detection and segmentation model to perform the target detection task, and the second decoded feature map is input into the target segmentation head network in the detection and segmentation model to perform the target segmentation task, so as to obtain the predicted location information of the target in the sample image and the predicted category information of the target in the sample segmentation. A detection loss function is constructed based on the predicted location information, and a segmentation loss function is constructed based on the predicted category information. The detection loss function and the segmentation loss function are used alternately to adjust the parameters of the detection and segmentation model.

2. The method as described in claim 1, characterized in that, The feature extraction network includes a convolutional neural subnetwork, a feature pyramid subnetwork, and an attention mechanism subnetwork; The step of performing multi-scale feature fusion and attention feature extraction on the sample images in the training samples based on the feature extraction network to obtain a multi-scale attention feature map of the sample images includes: Based on the convolutional subnetwork, feature extraction is performed on the sample image to obtain the image features of the sample image; Based on the feature pyramid subnetwork, multi-scale feature extraction is performed on the image features, and the extracted image features at multiple different scales are fused to obtain multi-scale fused features; Based on the attention mechanism subnetwork, attention weights are learned for the multi-scale fusion features to obtain the multi-scale attention feature map.

3. The method as described in claim 1, characterized in that, The first decoding network and the second decoding network have the same network structure, but different network parameters.

4. The method as described in claim 3, characterized in that, The step of inputting the first decoded feature map into the target detection head network of the detection and segmentation model, and inputting the second decoded feature map into the target segmentation head network of the detection and segmentation model, to obtain the predicted location information of the target in the sample image and the predicted category information of the target in the sample segmentation, includes: The first decoded feature map is input into the target detection head network, and the target detection head network is used to perform target detection on the sample image to obtain the predicted location information of the target detected in the sample. The second decoded feature map is input into the target segmentation head network, and the target segmentation head network is used to segment the sample image to obtain the predicted category information of the target segmentation in the sample.

5. The method according to any one of claims 1 to 4, characterized in that, Each training sample in the training sample set also includes the true location information of the target detected in the corresponding sample image and the true category information of the target segmented in the sample. The step of constructing a detection loss function based on the predicted location information and a segmentation loss function based on the predicted category information includes: Based on the difference between the predicted location information and the actual location information, a detection loss function is constructed; A segmentation loss function is constructed based on the difference between the predicted category information and the true category information.

6. The method as described in claim 1, characterized in that, The alternating use of the detection loss function and the segmentation loss function to adjust the parameters of the detection and segmentation model includes: Based on the detection loss function, the parameters of the feature extraction network, the first decoding network, and the target detection head network in the detection and segmentation model are updated. The parameters of the feature extraction network, the second decoding network, and the target segmentation head network in the detection and segmentation model are updated based on the segmentation loss function.

7. The method as described in claim 5, characterized in that, Before constructing the detection loss function based on the difference between the predicted location information and the actual location information, the method further includes: Based on the target detection head network, a first probability distribution map corresponding to the sample image is obtained. The pixel value of each pixel in the first probability distribution map represents the first predicted probability of the category to which the target of the sample belongs. The step of constructing a detection loss function based on the difference between the predicted location information and the actual location information includes: A classification loss function is constructed based on each first predicted probability; Based on the difference between the predicted location information and the actual location information, a location loss function is constructed; The detection loss function is constructed based on the classification loss function and the location loss function.

8. The method as described in claim 7, characterized in that, Before determining the detection loss function based on the classification loss function and the location loss function, the method further includes: The second probability distribution map corresponding to the sample image is obtained based on the target detection head network. The pixel value of each pixel in the second probability distribution map represents the second predicted probability of the detected target. The step of determining the detection loss function based on the classification loss function and the location loss function includes: Construct a target loss function based on each second prediction probability; The detection loss function is obtained by weighted summing of the target loss function, the classification loss function, and the location loss function.

9. The method as described in claim 1, characterized in that, The method further includes: Input the image to be processed into the trained detection and segmentation model; Based on the feature extraction network of the detection and segmentation model, multi-scale feature fusion and attention feature extraction are performed on the image to be processed to obtain the target multi-scale attention feature map corresponding to the image to be processed. The target multi-scale attention feature map is input into the target detection head network and the target segmentation head network in the detection and segmentation model, respectively, to obtain the target location information and target category information of the target to be segmented in the image to be processed; the target to be segmented and the target to be segmented belong to different categories.

10. A detection and segmentation model training device, characterized in that, include: The sample acquisition unit is used to acquire the training sample set; Each training sample in the training sample set includes at least one sample image, and each sample image includes a sample detection target and a sample segmentation target, wherein the sample detection target and the sample segmentation target belong to different target categories; Based on the training sample set, the detection and segmentation model to be trained is subjected to multiple rounds of iterative training to obtain a trained detection and segmentation model. In each round of iterative training, the parameters of the detection and segmentation model are adjusted by alternating training. The following operations are performed in each round of iterative training: The first feature extraction unit is used to select training samples from the training sample set, input them into the feature extraction network of the detection and segmentation model to be trained, and perform multi-scale feature fusion and attention feature extraction on the sample images in the training samples based on the feature extraction network to obtain the multi-scale attention feature map of the sample images. The second feature extraction unit is used to input the multi-scale attention feature map into the first decoding network and the second decoding network in the detection and segmentation model for decoding to obtain the first decoded feature map and the second decoded feature map; input the first decoded feature map into the target detection head network in the detection and segmentation model to perform the target detection task, and input the second decoded feature map into the target segmentation head network in the detection and segmentation model to perform the target segmentation task, so as to obtain the predicted location information of the target in the sample image and the predicted category information of the target in the segmentation. The parameter adjustment unit is used to construct a detection loss function based on the predicted location information and a segmentation loss function based on the predicted category information, and alternately use the detection loss function and the segmentation loss function to adjust the parameters of the detection and segmentation model.

11. The apparatus as claimed in claim 10, characterized in that, The feature extraction network includes a convolutional neural network, a feature pyramid subnetwork, and an attention mechanism subnetwork; the first feature extraction unit is specifically used for: Based on the convolutional subnetwork, feature extraction is performed on the sample image to obtain the image features of the sample image; Based on the feature pyramid subnetwork, multi-scale feature extraction is performed on the image features, and the extracted image features at multiple different scales are fused to obtain multi-scale fused features; Based on the attention mechanism subnetwork, attention weights are learned for the multi-scale fusion features to obtain the multi-scale attention feature map.

12. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any of the methods described in claims 1 to 9.

13. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the methods described in claims 1 to 9.

14. A computer program product, characterized in that, The method includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any one of claims 1 to 9.

Images