Multi-task multi-classification detection method, device, equipment, product and vehicle

Abstract

The invention provides a multi-task multi-classification detection method, device and equipment, a product and a vehicle. The method comprises the following steps: acquiring a to-be-detected image; calling a pre-trained target detection model, and performing multi-task multi-classification target detection on the to-be-detected image to obtain a target detection result of the to-be-detected image; wherein the target detection model is obtained by training based on first configuration information, and the first configuration information comprises a plurality of target detection tasks, at least one classification subtask of each target detection task, a category label of each classification subtask and a prediction unit corresponding to each category label; the target detection result comprises a classification prediction result of each classification subtask in each target detection task. According to the method, the unified model capable of simultaneously processing multiple different types of targets and fine-grained classification subtasks thereof is constructed and trained, multiple target detection tasks and classification subtasks thereof can be simultaneously processed, the number of models can be reduced, and the system cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a multi-task and multi-classification detection method, device, equipment, product and vehicle. Background Art

[0002] With the continuous development of technology, the demand for multi-task and multi-classification target detection has become increasingly prominent in many fields. It requires the system to be able to simultaneously identify multiple different categories of targets in the image and perform detailed classification processing on these targets. For example, in an intelligent driving system, it is necessary to simultaneously determine the status of a traffic light group (such as the color, shape, countdown, etc. of the traffic light) and the specific meaning of traffic signs (such as speed limit signs, no parking signs, direction signs, etc.).

[0003] However, most of the current multi-task and multi-classification target detection technologies adopt a segmentation processing method: splitting the complex target detection task of multi-task and multi-classification into multiple independent detection tasks (such as traffic light group detection, traffic sign recognition, etc.), and building and training models for each detection task. Although this processing method solves the problem to a certain extent, it also brings disadvantages that cannot be ignored. For example, each independent task requires the design, training and optimization of a dedicated model, which significantly increases the development and maintenance costs of the model; for example, in the deployment stage, each model needs to configure resources and set the reasoning process separately, resulting in repeated investment of computing resources and low efficiency. In resource-constrained situations, such as in vehicle systems or mobile devices, running multiple models at the same time may also overload the device, thereby damaging the rapid response capability and real-time performance of the detection. Summary of the invention

[0004] In view of this, the embodiments of the present application are committed to providing a multi-task and multi-classification detection method, device, equipment, product and vehicle. By constructing and training a unified model that can simultaneously handle multiple different categories of targets and their fine-grained classification subtasks, it is possible to simultaneously handle multiple target detection tasks and their classification subtasks, thereby reducing the number of models and reducing system costs.

[0005] According to a first aspect of an embodiment of the present application, a multi-task multi-classification detection method is provided, comprising:

[0006] Acquire the image to be detected;

[0007] Calling a pre-trained target detection model to perform multi-task and multi-classification target detection on the image to be detected, and obtaining a target detection result of the image to be detected;

[0008] Among them, the target detection model is trained based on first configuration information, the first configuration information includes multiple target detection tasks, at least one classification subtask of each target detection task, the category label of each classification subtask and the prediction unit corresponding to each category label; the target detection result includes the classification prediction result of each classification subtask in each target detection task.

[0009] Optionally, calling a pre-trained target detection model to perform multi-task multi-classification target detection on the image to be detected to obtain a target detection result of the image to be detected includes:

[0010] Call the pre-trained object detection model to perform the following processing:

[0011] According to the image features of the image to be detected and the first configuration information, each prediction unit is used to predict the corresponding category label to obtain the prediction confidence of each category label, and the category label prediction confidences of different classification subtasks are independent of each other;

[0012] According to the preset decision rules and the prediction confidence of each category label of each classification subtask, the optimal category label of the classification subtask is determined from each category label of each classification subtask, and the optimal category label is determined as the classification prediction result of the classification subtask.

[0013] Optionally, the target detection model is obtained based on improving a first model, wherein the first model includes a focus module;

[0014] The object detection model is modified based on the first model and includes:

[0015] The slice splicing mechanism in the focus module is modified into a multi-layer convolution feature extraction mechanism, which is used to downsample and adjust channels of the input image through multi-layer convolution, and the multi-layer convolution feature extraction mechanism has the same processing effect on the input image as the slice splicing mechanism, and the processing effect includes the downsampling ratio and the change in the number of channels.

[0016] Optionally, the training process of the target detection model includes:

[0017] Acquire a sample image to be detected and label information of the sample image to be detected, wherein the label information includes a classification labeling result of each classification subtask in each target detection task;

[0018] Inputting the sample image to be detected into the target detection model to obtain a target detection result corresponding to the sample image to be detected, wherein the target detection result includes a classification prediction result of each classification subtask in each target detection task;

[0019] Determine the total classification loss according to the classification prediction result of each classification subtask in each target detection task corresponding to the sample image to be detected and the classification labeling result of each classification subtask in each target detection task corresponding to the sample image to be detected;

[0020] Parameters of the object detection model are optimized based at least on the total classification loss.

[0021] Optionally, determining the total classification loss according to the classification prediction result of each classification subtask in each target detection task corresponding to the sample image to be detected and the classification labeling result of each classification subtask in each target detection task corresponding to the sample image to be detected includes:

[0022] Determine the first classification loss of each classification subtask based on the classification prediction result of each classification subtask and its corresponding classification labeling result;

[0023] According to the first classification loss of each classification subtask in each target detection task, determine the second classification loss of each target detection task, wherein for each target detection task, perform the following processing: according to the first weight corresponding to each classification subtask in the i-th target detection task, perform weighted sum calculation on the first classification loss of each classification subtask in the i-th target detection task to obtain the second classification loss of the i-th target detection task, where i is a positive integer;

[0024] According to the second classification loss of each object detection task, the total classification loss is determined.

[0025] Optionally, the training process of the target detection model further includes:

[0026] Before the training starts, the first weight is assigned to each classification subtask in the i-th object detection task according to the number of first sample detection images corresponding to each classification subtask in the i-th object detection task, and the first sample detection image corresponding to each classification subtask includes an image element representing a classification labeling result of the classification subtask;

[0027] and / or,

[0028] After each training stage, the first weight of each classification subtask in the first object detection task is adjusted according to the prediction accuracy of each classification subtask in the i-th object detection task in the current training stage.

[0029] Optionally, the annotation information of the sample image to be detected includes annotation box information, and the annotation box information includes the classification annotation result of each classification subtask of each target detection task in the annotation box;

[0030] The step of inputting the sample image to be detected into the target detection model to obtain the target detection result of the sample image to be detected includes:

[0031] Inputting the sample image to be detected into the target detection model to obtain prediction box information, wherein the prediction box information includes the classification prediction result of each classification subtask of each target detection task in the prediction box;

[0032] According to the prediction box information, the annotation box information and the preset allocation strategy, the prediction box and the annotation box are matched to determine the positive sample prediction box in each prediction box; wherein the preset allocation strategy includes a strategy for label allocation based on the comprehensive cost between the prediction box and the annotation box, the comprehensive cost is determined at least based on the total classification cost, the total classification cost is determined based on the classification cost of each target detection task, and the classification cost of each target detection task is determined by weighted summing the classification cost of each classification subtask in the target detection task and each corresponding second weight of each classification subtask;

[0033] The determining of the total classification loss according to the classification prediction result of each classification subtask in each target detection task and the classification labeling result of each classification subtask in each target detection task includes:

[0034] The total classification loss is determined according to the prediction box information of the positive sample prediction box and the labeling box information.

[0035] Optionally, the training process of the target detection model further includes:

[0036] Before the training starts, according to the number of first sample detection images corresponding to each classification subtask in the j-th target detection task, the second weight is assigned to each classification subtask in the j-th target detection task, the first sample detection image corresponding to each classification subtask includes an image element representing a classification labeling result of the classification subtask, and j is a positive integer;

[0037] and / or,

[0038] After each training stage, the second weight of each classification subtask in the j-th target detection task is adjusted according to the classification loss of each classification subtask in the j-th target detection task in the current training stage, where j is a positive integer.

[0039] According to a second aspect of an embodiment of the present application, a multi-task multi-classification detection device is provided, comprising:

[0040] The first unit is used to obtain an image to be detected;

[0041] The second unit is used to call a pre-trained target detection model to perform multi-task multi-classification target detection on the image to be detected, and obtain a target detection result of the image to be detected;

[0042] Among them, the target detection model is trained based on first configuration information, the first configuration information includes multiple target detection tasks, at least one classification subtask of each target detection task, the category label of each classification subtask and the prediction unit corresponding to each category label, and the target detection result includes the classification prediction result of each classification subtask in each target detection task.

[0043] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including a memory and a processor;

[0044] The memory is connected to the processor and is used to store programs;

[0045] The processor is used to implement the multi-task multi-classification detection method as described in any one of the first aspects of the embodiments of the present application by running the program in the memory.

[0046] According to a fourth aspect of an embodiment of the present application, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, enable the processor to implement the multi-task multi-classification detection method as described in any one of the first aspects of the embodiments of the present application.

[0047] According to a fifth aspect of the embodiment of the present application, a vehicle is provided, comprising at least one image acquisition device mounted on a vehicle body and the electronic device according to the third aspect of the embodiment of the present application, wherein:

[0048] The image acquisition device is communicatively connected with the electronic device;

[0049] The image acquisition device is used to collect a vehicle driving image and send the vehicle driving image to the electronic device, and the image to be detected is determined based on the vehicle driving image.

[0050] The multi-task and multi-classification detection method provided in the embodiment of the present application can simultaneously handle multiple target detection tasks and their classification subtasks by constructing a unified model that can simultaneously handle multiple different categories of targets and their fine-grained classification subtasks. On the one hand, only a comprehensive model needs to be designed and trained, which reduces the number of models and reduces development costs; on the other hand, only a single model needs to be adjusted and optimized, which greatly simplifies maintenance work and reduces long-term operating costs; on the other hand, there is no need to configure hardware resources and reasoning processes for each task separately during deployment, avoiding repeated investment of resources, and is particularly suitable for resource-constrained environments such as vehicle-mounted systems or mobile devices, ensuring the system's rapid response capabilities and real-time performance; by sharing the underlying feature extraction layer, the method provided in the embodiment of the present application also enhances the generalization ability and accuracy of the model, uses the correlation between tasks to improve learning effects, reduces overfitting, and improves the performance of the model on unseen data and the classification accuracy of each task. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0052] Figure 1 A schematic diagram of the structure of a multi-task and multi-classification detection system provided in an embodiment of the present application.

[0053] Figure 2 A flowchart of a multi-task and multi-classification detection method provided in an embodiment of the present application.

[0054] Figure 3 A schematic diagram of the training process of a target detection model provided in an embodiment of the present application.

[0055] Figure 4 A schematic diagram of the structure of a multi-task and multi-classification detection device provided in an embodiment of the present application.

[0056] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solution of the embodiment of the present application is applicable to every scenario that requires scheduling of simulation tasks, and is applicable to every field such as manufacturing, scientific research, autonomous driving and intelligent transportation, aerospace, etc. In the application scenario, the user has submitted the simulation task to the simulation system, and the simulation system needs to assign the corresponding candidate simulation computing node to the simulation task submitted by the user to execute the simulation task. The technical solution of the embodiment of the present application enables the simulation task to be assigned to the simulation computing node that is suitable for its execution requirements, improves the resource utilization of the simulation computing cluster, and improves the overall execution efficiency of the simulation task.

[0058] The technical solution provided in the embodiments of the present application can be exemplarily applied to hardware devices such as processors, electronic devices, servers (including cloud servers), or packaged into software programs to be run. When the hardware device executes the processing of the technical solution of the embodiments of the present application, or the above-mentioned software program is run, the automatic splitting of the target task and the automatic calling of the application program interface required for the task can be achieved to complete the purpose of the target task. The embodiments of the present application only exemplarily introduce the specific processing of the technical solution of the present application, and do not limit the specific implementation form of the technical solution of the present application. Any technical implementation form that can execute the processing of the technical solution of the present application can be adopted by the embodiments of the present application.

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] Before introducing this application solution, the relevant technologies are first introduced:

[0061] With the continuous development of information technology, each type of software, application and system is equipped with a complex menu structure so that users can access a variety of functions and information. However, with the increasing complexity and hierarchy of menu structures, users often face difficulties in navigation, especially in applications with a large number of menu nodes and deep hierarchical structures. Users often need to switch repeatedly between multiple levels to find the required functions or information, which greatly reduces the user experience.

[0062] Existing menu navigation methods often have limitations when dealing with complex menu structures and multi-level menu navigation. For example, some menu navigation methods rely on predefined keywords or fixed command patterns. When the menu structure is complex, users need to remember a large number of specific commands to operate, which increases the user's learning cost and difficulty of use; some menu navigation methods rely on intent recognition. When the menu structure is complex and there are many menu nodes, a large number of menu nodes need to be considered when performing intent recognition, resulting in huge computational complexity, time-consuming and inefficient. These problems seriously affect the user experience and reduce the efficiency and accuracy of menu navigation.

[0063] In view of this, the embodiments of the present application provide a menu navigation method, apparatus, device, storage medium and product, which not only support the user's natural language input, but also reduce the computational complexity and time consumption, and improve the efficiency and accuracy of menu navigation, which are described in detail one by one in the following embodiments.

[0064] Exemplary Systems

[0065] For ease of understanding, firstly, an exemplary introduction to the implementation environment of the multi-task multi-classification detection method provided in the embodiment of the present application is given. Figure 1 , Figure 1 This is a schematic diagram of the structure of a multi-task multi-classification detection system provided in an embodiment of the present application. The multi-task multi-classification detection method provided in the present application can be exemplarily applied to the system.

[0066] like Figure 1 As shown, the multi-task multi-classification detection system includes an image acquisition device 110 and a processing module 120 , wherein the image acquisition module 110 is communicatively connected with the processing module 120 .

[0067] The image acquisition device 110 is used to capture image data in an actual scene and send the acquired image data to the processing module 120 via a communication network.

[0068] Optionally, the image acquisition module 110 can be a camera (for example, an RGB camera, an infrared camera, a 3D camera, etc.), or other types of imaging devices (for example, a thermal imager, an X-ray machine, etc.). In some embodiments, the image acquisition device 110 can also be integrated with a pre-processing unit, which can perform preliminary processing on the original image data, such as denoising, enhancing contrast, adjusting brightness, etc., to improve the accuracy of subsequent target detection. In addition, the image acquisition device 110 can also be equipped with a wide-angle lens or a fisheye lens to expand the field of view and adapt to a wider range of application scenarios. This application is not limited to this, as long as image data that meets the subsequent processing requirements can be acquired.

[0069] The processing module 120 is used to execute the multi-task and multi-classification detection method provided in the embodiment of the present application, which specifically includes: obtaining an image to be detected; calling a pre-trained target detection model to perform multi-task and multi-classification target detection on the image to be detected based on first configuration information, and obtaining a target detection result of the image to be detected, wherein the first configuration information includes multiple target detection tasks, at least one classification subtask of each target detection task, a category label of each classification subtask, and a prediction unit corresponding to each category label, different target detection tasks correspond to different target object categories, and the target detection result includes a classification prediction result of each classification subtask in each target detection task.

[0070] As for the specific implementation of the multi-task and multi-classification detection method, it will be explained in subsequent content and will not be described in detail here.

[0071] Optionally, the processing module 120 can be any computing device or electronic device, such as a computer server, a workstation, a personal computer, a virtual machine, a cloud server, etc., or it can be a software program or script installed on any of the above hardware devices, which is not limited in this application.

[0072] Optionally, the multi-task multi-classification detection system may further include a storage module for storing the original image data collected by the image acquisition module, the target detection results processed by the processing module 120, or the target detection model parameters, configuration information, etc. The storage module may be a local storage device (such as a hard disk, a solid-state drive, etc.) or a remote storage service (such as cloud storage, etc.), which is not limited in this application.

[0073] Optionally, the multi-task multi-classification detection system may further include a display module for displaying the target detection results obtained by the processing module 120 in a visual manner, such as framing the detected target objects and marking specific category labels, so that users can intuitively understand the detection results. The display module 140 may be a display device such as a display or a projector.

[0074] Optionally, the multi-task multi-classification detection system may further include a post-processing module for performing further analysis and processing based on the target detection results obtained by the processing module 120 for use in downstream tasks.

[0075] Exemplarily, the multi-task and multi-classification detection system can be integrated in a vehicle, capable of identifying multiple targets in the driving environment, such as signal light groups and traffic sign lights, and can perform fine-grained classification of these targets, such as distinguishing different types of road signs (such as speed limits, no entry, direction indications, etc.) and different states of signal lights (such as the color, shape, and display numbers of each signal light in the signal light group), which helps smart driving cars or drivers make more accurate and timely driving decisions and improve driving safety and efficiency.

[0076] In this example, the image acquisition module 110 may be at least one camera installed on the vehicle body, used to capture the environment image during the vehicle driving process in real time, and may also include an infrared camera with night vision function to ensure normal operation under low light conditions. The processing module 120 may be an electronic device including a memory and a processor, supporting real-time multi-task and multi-classification target detection of the vehicle driving environment image acquired by the image acquisition device. The detection results can be used for the vehicle's automatic driving or assisted driving functions, such as path planning, automatic braking, etc., and can also be used to provide auxiliary information for the driver's driving decision.

[0077] Exemplary Methods

[0078] Figure 2 A flowchart of a multi-task multi-classification detection method provided in an embodiment of the present application. Figure 2 As shown, the multi-task multi-classification detection method can be applied to Figure 1 The multi-task multi-classification detection system is specifically executed by a processing module in the multi-task multi-classification detection system. The method includes steps S201-S202:

[0079] S201, obtaining an image to be detected.

[0080] The image to be detected is an image that needs to be subjected to multi-task multi-classification target detection. Optionally, the image to be detected contains target objects of multiple different categories, and the system needs to simultaneously identify these target objects and perform further fine-grained classification.

[0081] The images to be detected can come from a variety of channels, such as video frames captured by a camera in real time, static image files in a storage device, image data transmitted over a network, etc. The method of obtaining the images to be detected can be determined according to the specific application scenario and actual needs, and this application does not limit this. In the scenario of an intelligent driving system, these images usually come from a vehicle-mounted camera that captures the road and surrounding environment in real time.

[0082] Before formally performing target detection, it may be necessary to perform certain preprocessing operations on the acquired images to improve the accuracy and efficiency of detection. Preprocessing operations may include image scaling, cropping, grayscale, denoising, contrast enhancement, etc., depending on the needs and input requirements of the subsequent target detection model.

[0083] S202. Call a pre-trained target detection model to perform multi-task and multi-classification target detection on the image to be detected, and obtain the target detection result of the image to be detected; wherein the target detection model is trained based on first configuration information, and the first configuration information includes multiple target detection tasks, at least one classification subtask of each target detection task, a category label of each classification subtask, and a prediction unit corresponding to each category label; the target detection result includes a classification prediction result of each classification subtask in each target detection task.

[0084] The target detection task can be understood as a group of target recognition tasks that need to be performed simultaneously. Each target detection task corresponds to the detection of a specific type of target object in the image, where the target object is pre-defined according to the application scenario. For example, in the intelligent driving scenario, the target detection tasks include traffic light detection and traffic sign detection.

[0085] The classification subtask can be understood as a further subdivision of each target detection task, which is used to classify the identified target object in more detail. A target detection task includes at least one classification subtask. For example, the traffic light detection task includes 10 classification subtasks: the color classification subtask of the first signal light...the color classification subtask of the fourth signal light, the shape classification subtask of the first signal light...the shape classification subtask of the fourth signal light, the countdown single digit classification subtask, and the countdown tens digit classification subtask.

[0086] The category label is an identifier set for a specific category in each classification subtask, which is used to distinguish different categories during model training and prediction. The category label is pre-defined and corresponds to the annotation information in the training data set. A classification subtask includes at least one classification label. For example, in the first traffic light color classification subtask, the category labels may include four labels: "red light", "green light", "yellow light", and "default value". In the countdown single digit classification subtask, the category labels may include ten labels: "0", "1", "2"... "9".

[0087] Before executing step S202, a target detection model has been trained. After being fully trained, the target detection model can perform multi-task multi-classification target detection, not only can it identify multiple different types of target objects in the same image, but it can also perform further multi-classification tasks for each target object. As for the training process of the target detection model, it will be explained in subsequent embodiments and will not be elaborated in detail here.

[0088] The target detection model is taken as an example of a vehicle driving scenario. The pre-trained target detection model is a traffic light and traffic sign detection model. The trained traffic light and traffic sign detection model can handle traffic light detection tasks and traffic sign detection tasks at the same time. In this way, the traffic light and traffic sign detection model can provide comprehensive road information for the intelligent driving system or the driver, helping them to make more accurate driving decisions.

[0089] The target detection model can adopt deep learning architectures such as YOLOX network, FsaterRCNN network, YOLOv4 network, SSD network, etc., or a network architecture improved from any of the above networks. This application does not limit this, that is, it does not limit the specific network structure selection, as long as the function of multi-task and multi-classification detection can be realized.

[0090] The target detection model is trained based on the first configuration information. The first configuration information can be understood as parameter settings for the model, including but not limited to multiple target detection tasks, at least one classification subtask of each target detection task, the category label of each classification subtask, and the prediction unit corresponding to each category label. Among them, multiple target detection tasks define the target detection tasks that the model needs to process simultaneously, and these tasks are predetermined according to the application scenario and requirements; at least one classification subtask of each target detection task defines a more detailed classification work for each detection task; the category label of each classification subtask defines a specific category identifier for each classification subtask, telling the model which parts of the input image should be classified into which category; the prediction unit corresponding to each category label refers to the part of the model responsible for outputting the prediction result of a specific category, and each prediction unit determines whether a certain area belongs to a certain category label based on its learned features, and gives the corresponding confidence. Exemplarily, when the target detection model is implemented based on the YOLOX network architecture, the prediction unit can be understood as a neuron in the Cls. branch of the decoupling head, each neuron corresponds to a category label, and the output value of the neuron represents the probability or confidence of predicting the category label.

[0091] The first configuration information is configurable, allowing the object detection task, classification subtask, category label and its corresponding prediction unit to be adjusted according to actual needs.

[0092] Optionally, the first configuration information may also include structural parameters of the model (such as the number of layers, the number of neurons, the size of the convolution kernel, etc.), training parameters (such as learning rate, batch size, number of training rounds, etc.), data enhancement strategies (such as rotation, flipping, cropping, etc.) and other setting information related to model training and prediction.

[0093] Specifically, after obtaining the image to be detected, the system will call the pre-trained target detection model and pass the obtained image to be detected as input to the target detection model. The target detection model can perform multi-task and multi-classification target detection on the image to be detected, make predictions for each classification subtask of each target detection task, and output the final target detection result, which contains all the identified target detection objects in the image and their fine-grained classification information.

[0094] The target prediction result includes the classification prediction result of each classification subtask in each target detection task, that is, for each detection task, the model will give its prediction results on different classification subtasks. The classification prediction result usually includes the optimal category label corresponding to each classification subtask. Optionally, the classification prediction result also includes the prediction probability or confidence score of the optimal analog label. The target detection result can be in various forms, such as a list containing information such as target location, category and confidence, or a visualization result that directly marks the target and its category on the image. The specific form depends on the design and application requirements of the model.

[0095] As an optional implementation, step S202 "calling a pre-trained target detection model to perform multi-task multi-classification target detection on the image to be detected based on the first configuration information to obtain a target detection result of the image to be detected" includes:

[0096] Call the pre-trained target detection model to perform the following processing: according to the image features of the image to be detected and the first configuration information, use each prediction unit to predict the corresponding category label respectively to obtain the prediction confidence of each category label, and the category label prediction confidences of different classification subtasks are independent of each other; according to the preset decision rules and the prediction confidence of each category label of each classification subtask, determine the optimal category label of each classification subtask from each category label of each classification subtask, and determine the optimal category label as the classification prediction result of the classification subtask.

[0097] First, the object detection model extracts high-level features of the image to be detected through its convolutional layers. These features can capture the key attributes of different objects in the image, such as shape, color, texture, etc.

[0098] Then, the target detection model will use each prediction unit to predict the corresponding category label based on the extracted image features and the first configuration information (including multiple target detection tasks, classification subtasks of each task, category labels and corresponding prediction units). Each prediction unit is responsible for predicting a specific category label, and it will output a prediction confidence, which indicates the probability that the model believes that a certain area belongs to the category label. The category label prediction confidences of different classification subtasks are independent of each other.

[0099] Secondly, in order to select the most appropriate label from multiple category label predictions for each classification subtask, the model uses a preset decision rule. The decision rule represents a rule for determining the optimal category label for the classification subtask based on the prediction confidence of each category label of the classification subtask. For example, for each classification subtask, the category label with the highest prediction confidence is selected as the optimal category label; for example, a confidence threshold is set, and only when the prediction confidence of a category label exceeds this threshold will it be regarded as a valid prediction result; for example, the optimal category label is determined in combination with contextual information (such as the color change order of traffic lights) or historical data (such as the detection result of the previous frame image).

[0100] Finally, the model uses the first configuration information to integrate the optimal category labels of each classification subtask to form a complete classification prediction result, including all identified target objects in the image to be detected and their optimal category labels for each classification subtask.

[0101] The multi-task and multi-classification detection method provided in the embodiment of the present application can simultaneously handle multiple target detection tasks and their classification subtasks by constructing a unified model that can simultaneously handle multiple different categories of targets and their fine-grained classification subtasks. On the one hand, only a comprehensive model needs to be designed and trained, which reduces the number of models and reduces development costs; on the other hand, only a single model needs to be adjusted and optimized, which greatly simplifies maintenance work and reduces long-term operating costs; on the other hand, there is no need to configure hardware resources and reasoning processes for each task separately during deployment, avoiding repeated investment of resources, and is particularly suitable for resource-constrained environments such as vehicle-mounted systems or mobile devices, ensuring the system's rapid response capabilities and real-time performance; by sharing the underlying feature extraction layer, the method provided in the embodiment of the present application also enhances the generalization ability and accuracy of the model, uses the correlation between tasks to improve learning effects, reduces overfitting, and improves the performance of the model on unseen data and the classification accuracy of each task.

[0102] Next, we will take the target detection model using the YOLOX network architecture as an example to explain the training process of the target detection model. Figure 3 As shown, the training process of the target detection model provided in the embodiment of the present application includes steps S301-S304:

[0103] S301 , obtaining a sample image to be detected and label information of the sample image to be detected, wherein the label information includes a classification annotation result of each classification subtask in each object detection task.

[0104] The sample images to be detected are images in the data set used to train the model, and usually contain target objects that need to be detected and classified. The images in the training set should contain target objects in each scene, lighting condition, and weather condition that match the first configuration information to ensure that the model can learn features in different situations. In the example of a vehicle driving scene, the multiple sample images to be detected in the training data set may include traffic lights and traffic sign images at different times (daytime, night, dusk, etc.), different weather (sunny, rainy, foggy, etc.), and different road environments (urban roads, rural roads, highways, etc.).

[0105] Each sample image to be detected is annotated with the label information of the sample image to be detected. These label information include the classification labeling results of each classification subtask in each target detection task in the sample image to be detected. For example, in the traffic light detection task, it is necessary to mark the color (red light, green light, yellow light, etc.), shape (circle, left turn arrow, right turn arrow, straight arrow, etc.) and possible countdown numbers of each traffic light; in the traffic sign detection task, it is necessary to mark the type of sign (speed limit, no turn, parking, etc.) and other relevant information. These label information will be used for subsequent calculation of classification loss and model training.

[0106] S302: Input the sample image to be detected into the target detection model to obtain a target detection result corresponding to the sample image to be detected, wherein the target detection result includes a classification prediction result of each classification subtask in each target detection task.

[0107] During the training process, the sample image to be detected is sent to the target detection model based on the YOLOX architecture. The feature map containing the high-level semantic information of the target object in the image is extracted from the image through multiple convolutional neural layers in the backbone network; then multi-level fusion processing is performed through the feature pyramid network or path aggregation network to capture target objects of different sizes; finally, the fused data is input into the decoupled detection head to classify, locate and predict the target of each scale feature map, where the classification branch (Cls. branch) is responsible for predicting the classification prediction results of each classification subtask in each target detection task, the localization branch (Reg. branch) is responsible for predicting the location of the target object (bounding box coordinates), and the target branch (Obj. branch) is responsible for judging whether a certain area contains the target object. Finally, the model outputs the target detection result, which includes the classification prediction results of each classification subtask in each target detection task.

[0108] S303, determining a total classification loss according to the classification prediction result of each classification subtask in each target detection task corresponding to the sample image to be detected and the classification labeling result of each classification subtask in each target detection task corresponding to the sample image to be detected.

[0109] For each sample image to be tested, its classification prediction result needs to be compared with the classification annotation result, and the total classification loss of the sample test image is determined based on the difference between the two.

[0110] Optionally, step S303 includes steps A1-A3:

[0111] A1. Determine the first classification loss of each classification subtask based on the classification prediction result of each classification subtask and its corresponding classification labeling result.

[0112] A2. Determine the second classification loss of each target detection task according to the first classification loss of each classification subtask in each target detection task, wherein for each target detection task, perform the following processing: according to the first weight corresponding to each classification subtask in the i-th target detection task, perform weighted sum calculation on the first classification loss of each classification subtask in the i-th target detection task to obtain the second classification loss of the i-th target detection task, where i is a positive integer.

[0113] Among them, the setting of the first weight corresponding to each classification subtask can be adjusted according to factors such as the importance of each classification subtask or the category distribution in the data set, and is determined based on actual conditions, and this application does not limit this.

[0114] As an optional implementation, in order to better balance the importance of different classification subtasks and improve the overall performance of the model, the first weight of each classification subtask can be dynamically adjusted. In this implementation, the method for dynamically determining the first weight includes steps B1 and / or C1:

[0115] B1. Before training begins, the first weight is assigned to each classification subtask in the i-th target detection task according to the number of first sample detection images corresponding to each classification subtask in the i-th target detection task, and the first sample detection image corresponding to each classification subtask includes image elements that characterize the classification labeling results of the classification subtask.

[0116] In actual data sets, different classification subtasks may correspond to different numbers of sample detection images, which contain image elements that represent the classification and annotation results of the classification subtask. For example, in a traffic light detection task, the color of a traffic light may appear more frequently than the countdown of a traffic light. Therefore, the classification subtask corresponding to the traffic light color may correspond to more sample detection images.

[0117] Before model training, we will first traverse the number of first sample detection images corresponding to each classification subtask in the training set data, and assign a lower first weight value to the classification subtask with a larger number, because the model will be exposed to more such samples during the training process, so there is no need to overemphasize them; the classification subtask with a smaller number will be assigned a higher first weight value for balance, so as to encourage the model to pay more attention to such samples during training.

[0118] In this way, for the same object detection task, even if the number of samples of different classification subtasks is unbalanced, by adjusting the first weight, it can ensure that the model pays proper attention to all classification subtasks during training.

[0119] C1. After each training stage, the first weight of each classification subtask in the first target detection task is adjusted according to the prediction accuracy of each classification subtask in the i-th target detection task in the current training stage.

[0120] The prediction accuracy can be understood as the proportion of samples in a certain classification subtask that the target detection model correctly predicts.

[0121] After each training stage (such as an epoch or a certain number of iterations), evaluate the prediction accuracy of the model for each classification subtask on the validation set or training set. For the same object detection task, analyze the difference in prediction accuracy between different classification subtasks. For classification subtasks with higher prediction accuracy, appropriately reduce their first weights, because the object detection model may have learned these tasks relatively well; for classification subtasks with lower prediction accuracy, give them higher first weights, because the model may perform poorly on these tasks, and it is necessary to encourage the model to pay more attention to these tasks in subsequent training by increasing the weights, thereby improving the overall performance.

[0122] In this way, the target detection model will focus more on improving the performance of classification subtasks with lower prediction accuracy during training.

[0123] A3. Determine the total classification loss based on the second classification loss of each target detection task.

[0124] Optionally, the second classification loss of each target detection task is summed to obtain a total classification loss. The total classification loss reflects the overall performance of the target detection model on all classification subtasks. In the YOLOX network architecture, the total classification loss is the classification loss corresponding to the Cls. branch in the decoupling head, which measures the accuracy of the target detection model on the classification task.

[0125] S304: Optimize parameters of the target detection model at least based on the total classification loss.

[0126] When the target detection model adopts the YOLOX network architecture, since the decoupled detection head of the YOLOX network architecture has Cls. branch, Reg. branch and Obj. branch, the total loss of the model is determined by the total classification loss corresponding to the Cls. branch, the positioning loss corresponding to the Reg. branch, and the obj loss corresponding to the Obj. branch.

[0127] The parameter optimization of the target detection model based at least on the total classification loss includes: calculating the total model loss based on the total classification loss, the positioning loss and the obj loss, and optimizing the parameters of the target detection model based on the total model loss. As for the determination method of the positioning loss and the obj loss, it can be implemented with reference to the relevant prior art, and this application will not be described in detail here.

[0128] The above steps S301-S304 constitute a complete training iteration. During the entire training process, the target detection model will repeatedly go through the cycle of S301-S304 and continuously adjust its own parameters until the preset stop condition is reached, such as the number of training rounds (Epochs) reaches the upper limit, the loss no longer decreases significantly, or the performance on the validation set no longer improves.

[0129] After the above training process, the target detection model gradually converges and can show good performance on the training set and the validation set. Finally, the trained target detection model can perform multi-task and multi-classification target detection on the input image to be detected, and output the target detection result of the image to be detected.

[0130] As an optional implementation, the annotation information of the sample to-be-detected image includes annotation box information, and the annotation box information includes the classification annotation result of each classification subtask of each target detection task in the annotation box;

[0131] The step of inputting the sample image to be detected into the target detection model to obtain the target detection result of the sample image to be detected includes:

[0132] Inputting the sample image to be detected into the target detection model to obtain prediction box information, wherein the prediction box information includes the classification prediction result of each classification subtask of each target detection task in the prediction box;

[0133] According to the prediction box information, the annotation box information and the preset allocation strategy, the prediction box and the annotation box are matched to determine the positive sample prediction box in each prediction box; wherein the preset allocation strategy includes a strategy for label allocation based on the comprehensive cost between the prediction box and the annotation box, the comprehensive cost is determined at least based on the total classification cost, the total classification cost is determined based on the classification cost of each target detection task, and the classification cost of each target detection task is determined by weighted summing the classification cost of each classification subtask in the target detection task and each corresponding second weight of each classification subtask;

[0134] The determining of the total classification loss according to the classification prediction result of each classification subtask in each target detection task and the classification labeling result of each classification subtask in each target detection task includes:

[0135] The total classification loss is determined according to the prediction box information of the positive sample prediction box and the labeling box information.

[0136] In this embodiment, in order to calculate the classification loss, it is necessary to match the predicted box with the labeled box to determine which predicted boxes are positive samples (ie, boxes that match the true target). The matching process is determined based on a preset allocation strategy.

[0137] The comprehensive cost is used to measure the similarity between the predicted box and the labeled box, and determines whether the two match. The comprehensive cost is determined based on at least the total classification cost, which is in turn calculated based on the classification cost of each object detection task.

[0138] For each object detection task, the classification cost reflects the difference between the classification prediction result in the prediction box and the classification annotation result in the annotation box. The classification cost can be calculated by methods such as cross entropy loss.

[0139] The classification cost of each target detection task is determined by weighted summing the classification costs of each classification subtask in the task according to the second weight corresponding to each classification subtask. The second weight can be adjusted according to factors such as the importance of the classification subtask or the category distribution in the data set.

[0140] In order to better balance the importance of different classification subtasks and improve the overall performance of the model, the second weight of each classification subtask can be dynamically adjusted. As an optional implementation, the training process of the target detection model also includes steps D1 and / or E1:

[0141] D1. Before training begins, the second weight is assigned to each classification subtask in the j-th target detection task according to the number of first sample detection images corresponding to each classification subtask in the j-th target detection task, the first sample detection image corresponding to each classification subtask includes image elements representing the classification and labeling results of the classification subtask, and j is a positive integer.

[0142] Before model training, the number of first sample detection images corresponding to each classification subtask in the training set data will be traversed first, and the classification subtasks with a larger proportion in the same target detection task will be assigned a higher second weight value to prevent the model from overfitting these common categories; the classification subtasks with a smaller proportion will be assigned a lower second weight value for balance, so as to increase their proportion in the classification cost, so that the model pays more attention to these categories.

[0143] E1. After each training stage, the second weight of each classification subtask in the j-th target detection task is adjusted according to the classification loss of each classification subtask in the j-th target detection task in the current training stage, where j is a positive integer.

[0144] After each training phase (such as an epoch or a certain number of iterations), for the same object detection task, analyze the difference in classification loss between different classification subtasks. For classification subtasks with higher classification loss, appropriately reduce their second weight; for classification subtasks with lower classification loss, give them a higher second weight.

[0145] In this implementation, on the one hand, the single-classification confidence positive and negative sample allocation strategy is expanded to multi-task multi-classification confidence for positive and negative sample allocation, and the screening of foreground targets is bound to multi-task targets, realizing the function of multi-task multi-classification foreground target mining.

[0146] On the other hand, the original category fixed weight method has been changed and optimized to dynamically adjust the label assignment weight according to the performance of the detection model in various tasks and categories, enhancing the generalization of the model on different data sets. The label assignment weight can be adaptively adjusted according to the indicators of each task and category to screen positive samples, simplifying the model optimization process, and can quickly find the extreme value of model performance. It can maintain the degree of fit between the model and the objective function when the model changes in environmental conditions, improve the adaptability and generalization of the model, and realize automatic optimization and rapid deployment of the model without increasing manual intervention, thereby improving work efficiency.

[0147] On the other hand, after the image to be detected is subjected to multi-layer feature extraction by the convolutional neural network of the model backbone network, the size of the feature map will be downsampled to a very small value. At this time, the receptive field corresponding to each feature point is large, but the detail information is insufficient. If it is a low-quality image (low-resolution image), the pixel value corresponding to each feature point is even smaller, which will result in a low detection recall rate for small-sized targets or distant targets. However, through the calculation of the multi-task multi-classification loss function and the dynamic multi-classification label allocation strategy in this implementation method, even if the pixel value occupied by the target is small, its loss or cost can be significantly different from that of the background target through dynamic weighted smoothing, making it easy to correctly classify small targets as positive samples instead of filtering them out as background negative samples. This makes the performance on low-quality images better, thereby reducing the demand for high-quality images.

[0148] As an optional implementation, the target detection model is implemented based on the YOLOX architecture. In the preset allocation strategy, the comprehensive cost between the prediction box and the annotation box is determined based on the total classification cost, the regression cost and the object (obj) cost.

[0149] The comprehensive cost is calculated based on the following formula:

[0150]

[0151] Among them, C ij It refers to the comprehensive cost between the predicted box at the feature map ij and the labeled box. It refers to the total classification cost between the predicted box at the feature map ij and the labeled box. It refers to the obj cost between the predicted box at the feature map ij and the labeled box. It refers to the regression cost between the predicted box at the feature map ij and the labeled box. It refers to the output feature value of the Obj. branch at the feature map ij, and λ is the weight parameter.

[0152] It is the sum of the classification costs of each target detection task. The classification cost of each target detection task is the dynamic weighted sum of the classification costs of each classification subtask in the target detection task. For obvious non-foreground targets (such as background objects, targets that do not need to be detected, etc.), the classification is the weighted sum cost of the classification of None Very low, if only through It is difficult to distinguish the foreground from the background. After that, the total cost C of the foreground target and background can be ij There is a clear difference, which is beneficial for the allocation strategy to distinguish foreground targets from background. For potential foreground targets, The difference will not be very large, and for inconspicuous foreground targets (such as targets that are similar to traffic lights or traffic signs, but are not actually traffic lights or traffic signs), the The value will be larger, which can effectively distinguish between probabilistically blurred foreground targets and obvious foreground targets. In the experimental results, this improvement also increased the average accuracy of detection and classification by more than 5%.

[0153] Further, as an optional implementation, the target detection model is obtained based on improving the first model, and the first model includes a focus module. Optionally, the first model is a model based on the YOLOX network architecture.

[0154] In this implementation, the object detection model is modified based on the first model to include:

[0155] The slice splicing mechanism in the focus module is modified into a multi-layer convolution feature extraction mechanism, which is used to downsample and adjust channels of the input image through multi-layer convolution, and the multi-layer convolution feature extraction mechanism has the same processing effect on the input image as the slice splicing mechanism, and the processing effect includes the downsampling ratio and the change in the number of channels.

[0156] In the original network architecture of the first model, the focus module is used to preliminarily process the input image and extract features through a specific mechanism. Specifically, the focus module cuts the input image into multiple small blocks (patches), and then rearranges or splices these blocks together in a specific way to form a new, smaller feature map. This operation not only reduces the spatial size of the image (i.e., downsampling), but also increases the number of channels of the output feature map, providing richer information for subsequent convolutional layers. The new feature map is then fed into the subsequent part of the network for further processing.

[0157] Since image slicing inference takes a long time on some deployment devices with limited resources, it affects real-time performance. For example, the first model uses the A1000 chip when deployed on the vehicle side. If the chip slices the image for each frame of inference, it will consume a lot of computing power. Therefore, this application replaces the image patch slicing and splicing operation of the original focus module with multi-layer convolution feature extraction, while keeping the feature size and number of channels unchanged.

[0158] In this application, the original focus module is improved. The core of the improvement is to replace the original slicing and splicing mechanism with a multi-layer convolutional feature extraction mechanism. The multi-layer convolutional feature extraction mechanism can directly downsample and adjust the channels of the input image through a series of convolutional layers without the need for explicit image slicing and splicing. The improvement maintains the same processing effect as the original slicing and splicing mechanism, that is, the downsampling ratio and the change in the number of channels remain unchanged. This means that the improved model is compatible with the original model at the feature level and can be seamlessly integrated into the original network architecture.

[0159] The information of each convolution layer in the multi-layer convolution is determined based on the preset processing effect in the actual application scenario, and this application does not limit this. For example, the original method of dividing the input image into 4 patches and then performing channel fusion is modified to 3 convolution layers, and the information of the first convolution layer is "Conv&ReLU&Norm, convolution kernel size K=3, step size S=2", the information of the second convolution layer is "Conv&ReLU&Norm, convolution kernel size K=3, step size S=1", and the information of the third convolution layer is "Conv&ReLU&Norm, convolution kernel size K=3, step size S=1".

[0160] In this implementation, on the one hand, each layer of convolution can be regarded as a filter, which can slide on the input image and extract different features. By stacking multiple such convolution layers, the spatial resolution of the image can be gradually reduced (downsampling) while increasing or adjusting the number of channels. On the other hand, since the position distribution of static elements (such as road signs, vehicles, etc.) on the image has a certain regularity, the use of multi-layer convolution can directly capture these regularities, thereby extracting features more effectively. Compared with image slicing and splicing, multi-layer convolution operations are more efficient because they avoid complex image processing steps, making the reasoning speed of the entire model faster. In the experimental results, this improvement reduces the inference time of the model when it is deployed by 5% and increases the recall rate by 0.02.

[0161] This implementation not only maintains the original downsampling and channel adjustment effects, but also significantly improves the model's reasoning efficiency and improves detection performance to a certain extent. This improvement is particularly suitable for deployment environments with limited resources and helps achieve faster and more accurate target detection.

[0162] Exemplary Devices

[0163] Corresponding to the above-mentioned multi-task multi-classification detection method, an embodiment of the present application also provides a multi-task multi-classification detection device. Figure 4 is a schematic diagram of the structure of a multi-task multi-classification detection device provided in an embodiment of the present application, such as Figure 4 As shown, the multi-task multi-classification detection device provided in the embodiment of the present application includes:

[0164] The first unit 401 is used to obtain an image to be detected;

[0165] The second unit 402 is used to call a pre-trained target detection model to perform multi-task multi-classification target detection on the image to be detected, and obtain a target detection result of the image to be detected;

[0166] Among them, the target detection model is trained based on first configuration information, the first configuration information includes multiple target detection tasks, at least one classification subtask of each target detection task, the category label of each classification subtask and the prediction unit corresponding to each category label; the target detection result includes the classification prediction result of each classification subtask in each target detection task.

[0167] The multi-task and multi-classification detection device provided in the embodiment of the present application can simultaneously handle multiple target detection tasks and their classification subtasks by constructing a unified model that can simultaneously handle multiple different categories of targets and their fine-grained classification subtasks. On the one hand, only a comprehensive model needs to be designed and trained, which reduces the number of models and reduces development costs; on the other hand, only a single model needs to be adjusted and optimized, which greatly simplifies maintenance work and reduces long-term operating costs; on the other hand, there is no need to configure hardware resources and reasoning processes for each task separately during deployment, avoiding repeated investment of resources, and is particularly suitable for resource-constrained environments such as vehicle-mounted systems or mobile devices, ensuring the system's rapid response capabilities and real-time performance; by sharing the underlying feature extraction layer, the method provided in the embodiment of the present application also enhances the generalization ability and accuracy of the model, uses the correlation between tasks to improve learning effects, reduces overfitting, and improves the performance of the model on unseen data and the classification accuracy of each task.

[0168] Optionally, the second unit 402 may be specifically used for:

[0169] Call the pre-trained object detection model to perform the following processing:

[0170] According to the image features of the image to be detected and the first configuration information, each prediction unit is used to predict the corresponding category label to obtain the prediction confidence of each category label, and the category label prediction confidences of different classification subtasks are independent of each other;

[0171] According to the preset decision rules and the prediction confidence of each category label of each classification subtask, the optimal category label of the classification subtask is determined from each category label of each classification subtask, and the optimal category label is determined as the classification prediction result of the classification subtask.

[0172] Optionally, the target detection model is obtained based on improving a first model, wherein the first model includes a focus module;

[0173] The object detection model is modified based on the first model and includes:

[0174] The slice splicing mechanism in the focus module is modified into a multi-layer convolution feature extraction mechanism, which is used to downsample and adjust channels of the input image through multi-layer convolution, and the multi-layer convolution feature extraction mechanism has the same processing effect on the input image as the slice splicing mechanism, and the processing effect includes the downsampling ratio and the change in the number of channels.

[0175] Optionally, the training process of the target detection model includes:

[0176] Acquire a sample image to be detected and label information of the sample image to be detected, wherein the label information includes a classification labeling result of each classification subtask in each target detection task;

[0177] Inputting the sample image to be detected into the target detection model to obtain a target detection result corresponding to the sample image to be detected, wherein the target detection result includes a classification prediction result of each classification subtask in each target detection task;

[0178] Determine the total classification loss according to the classification prediction result of each classification subtask in each target detection task corresponding to the sample image to be detected and the classification labeling result of each classification subtask in each target detection task corresponding to the sample image to be detected;

[0179] Parameters of the object detection model are optimized based at least on the total classification loss.

[0180] Optionally, determining the total classification loss according to the classification prediction result of each classification subtask in each target detection task corresponding to the sample image to be detected and the classification labeling result of each classification subtask in each target detection task corresponding to the sample image to be detected includes:

[0181] Determine the first classification loss of each classification subtask based on the classification prediction result of each classification subtask and its corresponding classification labeling result;

[0182] According to the first classification loss of each classification subtask in each target detection task, determine the second classification loss of each target detection task, wherein for each target detection task, perform the following processing: according to the first weight corresponding to each classification subtask in the i-th target detection task, perform weighted sum calculation on the first classification loss of each classification subtask in the i-th target detection task to obtain the second classification loss of the i-th target detection task, where i is a positive integer;

[0183] According to the second classification loss of each object detection task, the total classification loss is determined.

[0184] Optionally, the training process of the target detection model further includes:

[0185] Before the training starts, the first weight is assigned to each classification subtask in the i-th object detection task according to the number of first sample detection images corresponding to each classification subtask in the i-th object detection task, and the first sample detection image corresponding to each classification subtask includes an image element representing a classification labeling result of the classification subtask;

[0186] and / or,

[0187] After each training stage, the first weight of each classification subtask in the first object detection task is adjusted according to the prediction accuracy of each classification subtask in the i-th object detection task in the current training stage.

[0188] Optionally, the annotation information of the sample image to be detected includes annotation box information, and the annotation box information includes the classification annotation result of each classification subtask of each target detection task in the annotation box;

[0189] The step of inputting the sample image to be detected into the target detection model to obtain the target detection result of the sample image to be detected includes:

[0190] Inputting the sample image to be detected into the target detection model to obtain prediction box information, wherein the prediction box information includes the classification prediction result of each classification subtask of each target detection task in the prediction box;

[0191] According to the prediction box information, the annotation box information and the preset allocation strategy, the prediction box and the annotation box are matched to determine the positive sample prediction box in each prediction box; wherein the preset allocation strategy includes a strategy for label allocation based on the comprehensive cost between the prediction box and the annotation box, the comprehensive cost is determined at least based on the total classification cost, the total classification cost is determined based on the classification cost of each target detection task, and the classification cost of each target detection task is determined by weighted summing the classification cost of each classification subtask in the target detection task and each corresponding second weight of each classification subtask;

[0192] The determining of the total classification loss according to the classification prediction result of each classification subtask in each target detection task and the classification labeling result of each classification subtask in each target detection task includes:

[0193] The total classification loss is determined according to the prediction box information of the positive sample prediction box and the labeling box information.

[0194] Optionally, the training process of the target detection model further includes:

[0195] Before the training starts, according to the number of first sample detection images corresponding to each classification subtask in the j-th target detection task, the second weight is assigned to each classification subtask in the j-th target detection task, the first sample detection image corresponding to each classification subtask includes an image element representing a classification labeling result of the classification subtask, and j is a positive integer;

[0196] and / or,

[0197] After each training stage, the second weight of each classification subtask in the j-th target detection task is adjusted according to the classification loss of each classification subtask in the j-th target detection task in the current training stage, where j is a positive integer.

[0198] The multi-task multi-classification detection device provided in this embodiment belongs to the same application concept as the multi-task multi-classification detection method provided in the above embodiments of this application, and can execute the multi-task multi-classification detection method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the multi-task multi-classification detection method. For technical details not fully described in this embodiment, please refer to the specific processing content of the multi-task multi-classification detection method provided in the above embodiments of this application, which will not be repeated here.

[0199] The functions implemented by the above first unit 401 and the second unit 402 may be implemented by the same or different processors, respectively, and the embodiment of the present application is not limited thereto.

[0200] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0201] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0202] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0203] In addition, each unit in the above device can be fully or partially integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0204] Exemplary Electronic Devices

[0205] Another embodiment of the present application also provides an electronic device, see Figure 5 As shown, the device includes:

[0206] Memory 200 and processor 210;

[0207] The memory 200 is connected to the processor 210 and is used to store programs;

[0208] The processor 210 is configured to implement the multi-task multi-classification detection method disclosed in any of the above embodiments by running the program stored in the memory 200 .

[0209] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0210] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0211] A bus may include a pathway that transfers information between each component of a computer system.

[0212] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0213] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0214] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0215] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0216] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0217] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0218] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any multi-task multi-classification detection method provided in the above embodiments of the present application.

[0219] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the multi-task and multi-classification detection method introduced in any of the above embodiments. The specific processing process and its beneficial effects can be found in the above-mentioned embodiment introduction of the multi-task and multi-classification detection method.

[0220] Exemplary computer program products and storage media

[0221] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the multi-task and multi-classification detection method according to each embodiment of the present application described in any of the above embodiments of this specification.

[0222] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0223] In addition, the embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the multi-task multi-classification detection method according to each embodiment of the present application described in any of the above embodiments of this specification, and specifically the following steps may be implemented:

[0224] S201, obtaining an image to be detected;

[0225] S202, calling a pre-trained target detection model to perform multi-task multi-classification target detection on the image to be detected, and obtaining a target detection result of the image to be detected;

[0226] Among them, the target detection model is trained based on first configuration information, the first configuration information includes multiple target detection tasks, at least one classification subtask of each target detection task, the category label of each classification subtask and the prediction unit corresponding to each category label; the target detection result includes the classification prediction result of each classification subtask in each target detection task.

[0227] For each of the above method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0228] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between each embodiment can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0229] The steps in each embodiment of the method of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0230] The modules and sub-modules in the device and the terminal in each embodiment of the present application can be combined, divided and deleted according to actual needs.

[0231] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0232] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0233] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0234] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-task multi-classification detection method, characterized in that: include: Acquire the image to be detected; Calling a pre-trained target detection model to perform multi-task and multi-classification target detection on the image to be detected, and obtaining a target detection result of the image to be detected; Among them, the target detection model is trained based on first configuration information, the first configuration information includes multiple target detection tasks, at least one classification subtask of each target detection task, the category label of each classification subtask and the prediction unit corresponding to each category label, and the target detection result includes the classification prediction result of each classification subtask in each target detection task.

2. The method according to claim 1, characterized in that The calling of the pre-trained target detection model to perform multi-task multi-classification target detection on the image to be detected to obtain the target detection result of the image to be detected includes: Call the pre-trained object detection model to perform the following processing: According to the image features of the image to be detected and the first configuration information, each prediction unit is used to predict the corresponding category label to obtain the prediction confidence of each category label, and the category label prediction confidences of different classification subtasks are independent of each other; According to the preset decision rules and the prediction confidence of each category label of each classification subtask, the optimal category label of the classification subtask is determined from each category label of each classification subtask, and the optimal category label is determined as the classification prediction result of the classification subtask.

3. The method according to claim 1, characterized in that: The target detection model is obtained based on improving the first model, wherein the first model includes a focus module; The object detection model is modified based on the first model and includes: The slice splicing mechanism in the focus module is modified into a multi-layer convolution feature extraction mechanism, which is used to downsample and adjust channels of the input image through multi-layer convolution, and the multi-layer convolution feature extraction mechanism has the same processing effect on the input image as the slice splicing mechanism, and the processing effect includes the downsampling ratio and the change in the number of channels.

4. The method according to any one of claims 1 to 3, characterized in that: The training process of the target detection model includes: Acquire a sample image to be detected and label information of the sample image to be detected, wherein the label information includes a classification labeling result of each classification subtask in each target detection task; Inputting the sample image to be detected into the target detection model to obtain a target detection result corresponding to the sample image to be detected, wherein the target detection result includes a classification prediction result of each classification subtask in each target detection task; Determine the total classification loss according to the classification prediction result of each classification subtask in each target detection task corresponding to the sample image to be detected and the classification labeling result of each classification subtask in each target detection task corresponding to the sample image to be detected; Parameters of the object detection model are optimized based at least on the total classification loss.

5. The method according to claim 4, characterized in that The determining of the total classification loss according to the classification prediction result of each classification subtask in each target detection task corresponding to the sample image to be detected and the classification labeling result of each classification subtask in each target detection task corresponding to the sample image to be detected includes: Determine the first classification loss of each classification subtask based on the classification prediction result of each classification subtask and its corresponding classification labeling result; According to the first classification loss of each classification subtask in each target detection task, determine the second classification loss of each target detection task, wherein for each target detection task, perform the following processing: according to the first weight corresponding to each classification subtask in the i-th target detection task, perform weighted sum calculation on the first classification loss of each classification subtask in the i-th target detection task to obtain the second classification loss of the i-th target detection task, where i is a positive integer; According to the second classification loss of each object detection task, the total classification loss is determined.

6. The method according to claim 5, characterized in that The training process of the target detection model also includes: Before the training starts, the first weight is assigned to each classification subtask in the i-th object detection task according to the number of first sample detection images corresponding to each classification subtask in the i-th object detection task, and the first sample detection image corresponding to each classification subtask includes an image element representing a classification labeling result of the classification subtask; and / or, After each training stage, the first weight of each classification subtask in the first object detection task is adjusted according to the prediction accuracy of each classification subtask in the i-th object detection task in the current training stage.

7. The method according to claim 4, characterized in that The annotation information of the sample to-be-detected image includes annotation box information, and the annotation box information includes the classification annotation result of each classification subtask of each target detection task in the annotation box; The step of inputting the sample image to be detected into the target detection model to obtain the target detection result of the sample image to be detected includes: Inputting the sample image to be detected into the target detection model to obtain prediction box information, wherein the prediction box information includes the classification prediction result of each classification subtask of each target detection task in the prediction box; According to the prediction box information, the annotation box information and the preset allocation strategy, the prediction box and the annotation box are matched to determine the positive sample prediction box in each prediction box; wherein the preset allocation strategy includes a strategy for label allocation based on the comprehensive cost between the prediction box and the annotation box, the comprehensive cost is determined at least based on the total classification cost, the total classification cost is determined based on the classification cost of each target detection task, and the classification cost of each target detection task is determined by weighted summing the classification cost of each classification subtask in the target detection task and each corresponding second weight of each classification subtask; The determining of the total classification loss according to the classification prediction result of each classification subtask in each target detection task and the classification labeling result of each classification subtask in each target detection task includes: The total classification loss is determined according to the prediction box information of the positive sample prediction box and the labeling box information.

8. The method according to claim 7, characterized in that The training process of the target detection model also includes: Before the training starts, according to the number of first sample detection images corresponding to each classification subtask in the j-th target detection task, the second weight is assigned to each classification subtask in the j-th target detection task, the first sample detection image corresponding to each classification subtask includes an image element representing a classification labeling result of the classification subtask, and j is a positive integer; and / or, After each training stage, the second weight of each classification subtask in the j-th target detection task is adjusted according to the classification loss of each classification subtask in the j-th target detection task in the current training stage, where j is a positive integer.

9. A multi-task multi-classification detection device, characterized in that: include: The first unit is used to obtain an image to be detected; The second unit is used to call a pre-trained target detection model to perform multi-task multi-classification target detection on the image to be detected, and obtain a target detection result of the image to be detected; Among them, the target detection model is trained based on first configuration information, the first configuration information includes multiple target detection tasks, at least one classification subtask of each target detection task, the category label of each classification subtask and the prediction unit corresponding to each category label, and the target detection result includes the classification prediction result of each classification subtask in each target detection task.

10. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the multi-task multi-classification detection method as described in any one of claims 1-8 by running the program in the memory.

11. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to implement the multi-task multi-classification detection method according to any one of claims 1 to 8.

12. A vehicle, characterized in that: comprising at least one image acquisition device mounted on a vehicle body and the electronic device as claimed in claim 10, wherein: The image acquisition device is communicatively connected with the electronic device; The image acquisition device is used to collect a vehicle driving image and send the vehicle driving image to the electronic device, and the image to be detected is determined based on the vehicle driving image.

Images