Target detection method, device, computer equipment and computer-readable storage medium

Abstract

Embodiments of the present invention disclose a target detection method, apparatus, computer device, and computer-readable storage medium. The method includes: obtaining multiple images and multiple label information; the first label information is the category information of the object to be detected included in the first image, and the first image is any image in the multiple images; inputting the first image into a first model to obtain a feature map of a candidate region and a corresponding confidence level; inputting the feature map of the candidate region into a second model to obtain a detection score matrix and a classification score matrix, wherein the first model and the second model are initial detection models; determining a mean square error loss value and a cross entropy loss value based on the first label information, the confidence level, the detection score matrix, and the classification score matrix; optimizing the parameters of the initial detection model based on the mean square error loss value and the cross entropy loss value to obtain a trained detection model; and using the detection model to detect the object to be detected in the image. Embodiments of the present invention can improve sample annotation efficiency.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a target detection method, device, computer equipment, and computer-readable storage medium. Background Art

[0002] Object detection is a core problem in computer vision. It involves finding all objects of interest in an image and determining their categories and locations. Applications include unmanned smart supermarkets, autonomous driving, intelligent surveillance, and helmet / seatbelt detection.

[0003] Currently, object detection algorithms are primarily based on supervised learning, relying on a large amount of manually annotated, complete ground truth (GT) data. Training with this large amount of GT data yields a well-performing object detection model. For example, for a training image, it is necessary to know the category and specific location of the object within the image. The location is typically annotated with a rectangular box or (x, y, length, width), where (x, y) represents the coordinates of the center of the object, and length and width represent the distance from the center to the object's sides, respectively. Manually producing GT data is time- and costly, and the quality of annotation varies between different people, making it difficult to guarantee overall quality. Furthermore, if a large number of images are required for training, image annotation efficiency is low. Summary of the Invention

[0004] The embodiments of the present invention disclose a target detection method, apparatus, computer device, and computer-readable storage medium that can improve the efficiency of sample (i.e., image data) annotation and enhance the effectiveness of target detection. Furthermore, when the model's capabilities reach a certain level (e.g., detection and classification accuracy greater than 95%), image-level labels can be automatically generated by the algorithm (i.e., a trained detection model), enabling "self-annotation" of collected data and "online learning" of the detection model.

[0005] A first aspect discloses a target detection method, which can be applied to a computer device or a module (e.g., a chip) in a computer device. The following description uses the application of the computer device as an example. The method may include:

[0006] Acquire multiple images and multiple label information; the multiple images correspond one-to-one to the multiple label information, the first label information is the category information of the object to be detected included in the first image, and the first image is any image in the multiple images;

[0007] Inputting the first image into the first model to obtain a feature map of the candidate region and a corresponding confidence score;

[0008] Inputting the feature map of the candidate region into the second model to obtain a detection score matrix and a classification score matrix, wherein the first model and the second model belong to an initial detection model;

[0009] Determining a mean square error loss value and a cross entropy loss value according to the first label information, the confidence, the detection score matrix, and the classification score matrix;

[0010] Optimizing the parameters of the initial detection model according to the mean square error loss value and the cross entropy loss value to obtain a trained detection model; the detection model is used to detect the object to be detected in the image.

[0011] In an embodiment of the present invention, a computer device may first obtain multiple images and label information corresponding to the multiple images (i.e., training data), wherein the first label information is the category information of the object to be detected included in the first image, and the first image is any image among the multiple images. Afterwards, the computer device may input the first image into the first model to obtain the feature map of the candidate region (i.e., extract features through the first model) and the confidence level. Then, the computer device may input the feature map of the candidate region into the second model to obtain the detection score matrix and the classification score matrix. The first model and the second model are two parts of the initial detection model. Afterwards, the computer device may determine the mean square error loss value and the cross entropy loss value based on the first label information, the confidence level, the detection score matrix, and the classification score matrix. Thereafter, the computer device may optimize the parameters of the initial detection model based on the mean square error loss value and the cross entropy loss value to obtain a trained detection model. It can be seen that in an embodiment of the present invention, only the category information of the object to be detected included in the first image needs to be known, which belongs to the image-level label information. Therefore, the labeling efficiency of the sample (i.e., the training image) can be improved. At the same time, when optimizing the parameters of the initial detection model, the computer equipment comprehensively considers the mean square error loss value and the cross entropy loss value. Therefore, after optimizing the model parameters (such as the gradient descent algorithm), the classification loss value and the detection loss value of the initial detection model can be balanced. The trained detection model has a relatively even performance in the detection accuracy and classification accuracy of the target, and does not focus on any one aspect. A detection model with high detection accuracy and classification accuracy can be obtained. Moreover, since it is a detection model trained based on weak supervision labels, the detection model has good robustness.

[0012] Among them, the samples fed into the initial detection model can be data collected by the machine during operation. These data are naturally random, diverse, and massive. For example, the different data collected by the vehicle during operation (such as image data) can be generated by manual annotation in the early stage, and can be generated by the model itself after the model accuracy is high in the later stage. For example, if the detection model (including the first model and the second model) trained by multiple images already has excellent detection capabilities, and the detection accuracy and classification accuracy are high (such as the detection accuracy and classification accuracy have reached a specific threshold (such as 95%)), then the image-level labels of the data can all be generated by the second model itself (that is, it can be targeted at the image), so as to achieve "self-labeling" of the data and "self-training" of the model, and realize "online learning".

[0013] As a possible implementation, the first model is a strongly supervised learning target detection model trained with true value data.

[0014] In this embodiment of the present invention, the first model can be a strongly supervised learning object detection model trained with ground truth data. That is, it can be a pre-trained model. Therefore, the first model can produce better feature maps of candidate regions (i.e., better extracted features) and more accurate confidence scores for the candidate regions. Furthermore, the total loss value can be used to fine-tune the parameters of the first model, thereby further improving its performance.

[0015] As a possible implementation, the second model includes a detection branch and a classification branch; the feature map of the candidate region is input into the second model to obtain a detection score matrix and a classification score matrix, including:

[0016] Inputting the feature map of the candidate region into the detection branch to obtain the detection score matrix, the detection branch includes a global average pooling layer and a first fully connected layer;

[0017] The feature map of the candidate region is input into the classification branch to obtain the classification score matrix, where the classification branch includes a global average pooling layer and a second fully connected layer, and the parameters of the first fully connected layer and the second fully connected layer are different.

[0018] As a possible implementation, determining the mean square error loss value and the cross entropy loss value according to the first label information, the confidence, the detection score matrix, and the classification score matrix includes:

[0019] Determining prediction confidence based on the detection score matrix;

[0020] Determining a predicted label based on the detection score matrix and the classification score matrix;

[0021] Determining a mean square error loss value according to the confidence level and the prediction confidence level;

[0022] A cross entropy loss value is determined according to the first label information and the predicted label.

[0023] As a possible implementation, the second model further includes a normalization layer, and determining the prediction confidence according to the detection score matrix includes:

[0024] Input the detection score matrix into the normalization layer to obtain a normalized matrix;

[0025] The maximum value of each row in the normalized matrix is ​​determined as the prediction confidence.

[0026] As a possible implementation, the second model further includes a normalization layer, a first softmax layer, and a second softmax layer, and determining the predicted label according to the detection score matrix and the classification score matrix includes:

[0027] Input the detection score matrix into the normalization layer to obtain a normalized matrix;

[0028] Inputting the normalized matrix into the first softmax layer to obtain a first matrix;

[0029] Inputting the classification score matrix into the second softmax layer to obtain a second matrix;

[0030] Performing a Hadamard product matrix transformation on the first matrix and the second matrix to obtain a third matrix;

[0031] The predicted label is obtained by performing a sum operation on each column of the third matrix.

[0032] As a possible implementation, the step of optimizing the parameters of the initial detection model according to the mean square error loss value and the cross entropy loss value includes:

[0033] The cross entropy loss value is multiplied by the weight R / C to obtain the weighted cross entropy loss value; C is the number of categories of objects to be detected, and R is the number of candidate regions;

[0034] The sum of the weighted cross entropy loss value and the mean square error loss value is determined as the total loss value;

[0035] The parameters of the initial detection model are optimized according to the total loss value.

[0036] In an embodiment of the present invention, the computer device can multiply the cross entropy loss value (i.e., the classification loss value) by a weight (R / C), and then add it to the mean square error loss value (i.e., the detection loss value) to obtain a total loss value, so that the classification loss value and the detection loss value can make the same contribution when optimizing the initial detection model. During the loss back propagation process, after the model parameters are corrected by the gradient descent algorithm, the classification loss value and the detection loss value of the initial detection model can be balanced. This avoids focusing only on reducing the classification loss value or the detection loss value during model optimization, and the trained detection model performs more evenly in detection accuracy and classification accuracy, and will not focus on any one aspect. A detection model with high detection accuracy and classification accuracy can be obtained.

[0037] A second aspect discloses an object detection device, which may be a computer device or a module (e.g., a chip) in a computer device. The device may include:

[0038] an acquisition unit, configured to acquire a plurality of images and a plurality of label information; the plurality of images correspond one-to-one to the plurality of label information, wherein the first label information is category information of the object to be detected included in the first image, and the first image is any image in the plurality of images;

[0039] A first processing unit is configured to input the first image into a first model to obtain a feature map of the candidate region and a corresponding confidence score;

[0040] A second processing unit is configured to input the feature map of the candidate region into a second model to obtain a detection score matrix and a classification score matrix, wherein the first model and the second model belong to an initial detection model;

[0041] a determining unit, configured to determine a mean square error loss value and a cross entropy loss value according to the first label information, the confidence level, the detection score matrix, and the classification score matrix;

[0042] An optimization unit is used to optimize the parameters of the initial detection model according to the mean square error loss value and the cross entropy loss value to obtain a trained detection model; the detection model is used to detect the object to be detected in the image.

[0043] As a possible implementation, the first model is a strongly supervised learning target detection model trained with true value data.

[0044] As a possible implementation, the second model includes a detection branch and a classification branch; the second processing unit is specifically configured to:

[0045] Inputting the feature map of the candidate region into the detection branch to obtain the detection score matrix, the detection branch includes a global average pooling layer and a first fully connected layer;

[0046] The feature map of the candidate region is input into the classification branch to obtain the classification score matrix, where the classification branch includes a global average pooling layer and a second fully connected layer, and the parameters of the first fully connected layer and the second fully connected layer are different.

[0047] As a possible implementation, the determining unit determines the mean square error loss value and the cross entropy loss value according to the first label information, the confidence, the detection score matrix, and the classification score matrix, including:

[0048] Determining prediction confidence based on the detection score matrix;

[0049] Determining a predicted label based on the detection score matrix and the classification score matrix;

[0050] Determining a mean square error loss value according to the confidence level and the prediction confidence level;

[0051] A cross entropy loss value is determined according to the first label information and the predicted label.

[0052] As a possible implementation, the second model further includes a normalization layer, and the determination unit determines the prediction confidence according to the detection score matrix including:

[0053] Input the detection score matrix into the normalization layer to obtain a normalized matrix;

[0054] The maximum value of each row in the normalized matrix is ​​determined as the prediction confidence.

[0055] As a possible implementation, the second model further includes a normalization layer, a first softmax layer, and a second softmax layer, and the determination unit determines the predicted label according to the detection score matrix and the classification score matrix, including:

[0056] Input the detection score matrix into the normalization layer to obtain a normalized matrix;

[0057] Inputting the normalized matrix into the first softmax layer to obtain a first matrix;

[0058] Inputting the classification score matrix into the second softmax layer to obtain a second matrix;

[0059] Performing a Hadamard product matrix transformation on the first matrix and the second matrix to obtain a third matrix;

[0060] The predicted label is obtained by performing a sum operation on each column of the third matrix.

[0061] As a possible implementation, the optimization unit is specifically configured to:

[0062] The cross entropy loss value is multiplied by the weight R / C to obtain the weighted cross entropy loss value; C is the number of categories of objects to be detected, and R is the number of candidate regions;

[0063] The sum of the weighted cross entropy loss value and the mean square error loss value is determined as the total loss value;

[0064] The parameters of the initial detection model are optimized according to the total loss value.

[0065] A third aspect discloses a computer device comprising a processor and a memory. The memory is configured to store a computer program, and the processor is configured to invoke the computer program. When the processor executes the computer program stored in the memory, the processor performs the object detection method disclosed in the first aspect or any embodiment of the first aspect.

[0066] A fourth aspect discloses a computer-readable storage medium having a computer program or computer instructions stored thereon. When the computer program or computer instructions are executed, the target detection method disclosed in the above aspects is implemented.

[0067] A fifth aspect discloses a chip, comprising a processor for executing a program stored in a memory. When the program is executed, the chip executes the above method.

[0068] As a possible implementation, the memory is located outside the chip.

[0069] A sixth aspect discloses a computer program product, which includes a computer program code. When the computer program code is run, the above-mentioned target detection method is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a schematic diagram of a target detection scenario disclosed in an embodiment of the present invention;

[0071] Figure 2 It is a logical flow diagram of a target detection method disclosed in an embodiment of the present invention;

[0072] Figure 3 is a flow chart of a target detection method disclosed in an embodiment of the present invention;

[0073] Figure 4 is a schematic diagram of a geometric image of a sigmoid function disclosed in an embodiment of the present invention;

[0074] Figure 5 is a schematic structural diagram of a target detection model disclosed in an embodiment of the present invention;

[0075] Figure 6is a flow chart of another target detection method disclosed in an embodiment of the present invention;

[0076] Figure 7 is a schematic diagram of a detection result disclosed in an embodiment of the present invention;

[0077] Figure 8 is a schematic structural diagram of a target detection device disclosed in an embodiment of the present invention;

[0078] Figure 9 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The embodiments of the present invention disclose a target detection method, apparatus, computer equipment and computer-readable storage medium, which can improve the labeling efficiency of samples (i.e., image data) and improve the effect of target detection. At the same time, when the capabilities of the model are improved to a certain stage (such as detection accuracy and classification accuracy greater than 95%), image-level labels can be automatically generated by the algorithm (i.e., the trained detection model), which can realize the "self-labeling" of the collected data and the "online learning" of the detection model. The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0080] Obviously, the described embodiments are only some of the embodiments of this application, and not all of them. Reference to "embodiments" herein means that the specific features, structures, or characteristics described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it represent independent or alternative embodiments that are mutually exclusive with other embodiments. It is understood, both explicitly and implicitly, that the embodiments described herein can be combined with other embodiments. All other embodiments derived by one of ordinary skill in the art without inventive effort based on the embodiments in this application are within the scope of protection of this application. In the specification, claims, and accompanying drawings of this application, the terms "first," "second," "third," and so on are used to distinguish between different objects, not to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a list of steps or elements may be included, or alternatively, steps or elements not listed may be included, or alternatively, other steps or elements inherent to such process, method, product, or device may be included.

[0081] Only part relevant to the present application is shown in the accompanying drawings, not all of it. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processing or methods depicted as flow charts. Although flow charts describe various operations (or steps) as sequential processing, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. When its operation is completed, the processing can be terminated, but can also have additional steps not included in the accompanying drawings. The processing can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0082] As used in this specification, the terms "component," "module," "system," "unit," and the like are used to refer to computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a unit can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or distributed between two or more computers. In addition, these units can be executed from various computer-readable media having various data structures stored thereon. Units can communicate, for example, through local and / or remote processes based on signals having one or more data packets (e.g., data from a second unit interacting with another unit in a local system, a distributed system, and / or a network. For example, the Internet interacts with other systems via signals).

[0083] In order to better understand the embodiments of the present invention, some terms and related technologies of the embodiments of the present invention are described below.

[0084] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0085] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0086] Computer vision (CV) is the science of making machines "see." Specifically, it refers to the use of cameras and computers to replace the human eye in identifying, tracking, and measuring objects. This involves further processing the images, transforming them into images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0087] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to imbue computers with intelligence. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, weakly supervised learning, and strongly supervised learning.

[0088] Transfer learning involves migrating the parameters of a trained model (pre-trained model) to another untrained model to improve its training efficiency. Since most data or tasks are related, transfer learning allows you to share the learned model parameters (also known as the knowledge learned by the model) with other untrained models in some way, thereby accelerating and optimizing the model's learning efficiency without having to learn a model from scratch.

[0089] For object detection, traditional strongly supervised learning object detection algorithms rely on large amounts of manually annotated complete ground truth (GT) data to train object detection / segmentation models. Complete GT data, for each image, requires annotation of all objects and their corresponding categories. This means knowing how many objects are in the image, as well as the specific location and category of each object. Consequently, for traditional strongly supervised learning object detection algorithms, producing complete GT training data is time-consuming and expensive, and the quality of annotation varies between different people, making it difficult to guarantee the overall quality of the annotations.

[0090] To address the difficulty in obtaining complete GT training data, the industry has begun researching weakly supervised object detection algorithms. Weakly supervised object detection algorithms rely solely on simple image-level labeling information. Image-level labels are a form of weakly supervised labeling; they simply indicate which object categories are present in an image, without requiring knowledge of the specific locations of the objects or the number of objects in each category. This type of weakly supervised labeling can be hundreds of times more efficient than traditional data annotation, and in certain situations, it can even enable automatic image annotation. For example, the target categories to be detected may only include cars (sedans), bicycles, and electric motorcycles. Therefore, sample images collected on highways or overpasses can be assumed to contain only cars and can be directly labeled as [car, none, none], significantly improving labeling efficiency. This significantly reduces the time and cost of obtaining image-level labels, while also providing more accurate labeling results. However, developing a high-performance weakly supervised object detection model remains a major concern for researchers.

[0091] In order to better understand the embodiments of the present invention, the following first introduces the applicable scenarios of the embodiments of the present invention by way of example.

[0092] For example, the target detection method provided by the embodiment of the present invention can be used to obtain a weakly supervised target detection model with high detection accuracy and classification accuracy. The weakly supervised target detection model can detect the target to be detected from an image or video, that is, it can find all targets of interest (i.e., targets to be detected, or referred to as targets to be detected) from the image and determine their categories and specific locations.

[0093] For example, in an assisted driving scenario, the targets that need to be detected may include cars, traffic lights, and traffic signs. In order to obtain a target detection model that can identify the above three targets, it is first necessary to obtain training data. The training data can be images collected by a data acquisition vehicle and the corresponding labels. The label of an image can be (x1, x2, x3), where the values ​​of x1, x2, and x3 can be 0 or 1. x1 is 1, which means that the image includes a car, and x1 is 0, which means that the image does not include a car. Similarly, x2 is 1, which means that the image includes a traffic light, and x2 is 0, which means that the image does not include a traffic light. x3 is 1, which means that the image includes a traffic sign, and x2 is 0, which means that the image does not include a traffic sign. By training the initial target detection model with the training data, a target detection model can be obtained. The target detection model can be used to detect cars, traffic lights, and traffic signs included in the image.

[0094] The trained target detection model can be run on the vehicle system. The vehicle system can obtain the real-time environment image of the smart car through the onboard camera of the smart car. Then, the surrounding environment image can be input into the target detection model to realize the recognition of the target, so as to assist the driver in driving and improve driving safety. For example, see Figure 1 , Figure 1 This is a schematic diagram of a target detection scenario disclosed in an embodiment of the present invention. Figure 1 As shown in the figure, there is a car B to the right of car A. Car B is in the blind spot of the driver of car A, meaning that the driver cannot see car B through the rearview mirror. At this time, if car B is driving faster, car A will change lanes to the right and there is a high risk of a traffic accident. However, if car A's onboard camera can capture real-time images of the area behind car A and then use an object detection model to detect objects in this image, car B can be detected in advance. The driver can then be alerted to the car behind them through voice or other means, thus avoiding a traffic accident.

[0095] It should be understood that the target detection method provided by the embodiment of the present invention can also be applied to other target detection scenarios, such as unmanned smart supermarkets, intelligent monitoring, safety helmet / seat belt detection, etc., and is not limited to the assisted driving scenario introduced above.

[0096] The target detection method provided by the embodiment of the present invention can be executed by a computer device, and the computer device includes but is not limited to a terminal device or a server. The above-mentioned server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms. The above-mentioned terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, a smart TV, etc., but is not limited to this. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0097] In order to better understand the embodiments of the present invention, Figure 2 , the basic logical flow of a target detection method disclosed in an embodiment of the present invention is introduced.

[0098] like Figure 2As shown, for an original image, the original image (i.e., the sample image) is first input into a candidate box generator, and the candidate box generator generates information about the detection candidate box (region proposal box, RP_BOX) and the corresponding confidence S for the original image. The confidence can be understood as the probability that the corresponding detection candidate box has a target (i.e., the object to be detected), or it can be understood as the probability that the candidate region corresponding to the detection candidate box is a target. The confidence can also be called a confidence score. It should be understood that for an original image, the number of detection candidate boxes generated by the candidate box generator can be 0 or one or more. For example, when the original image does not include the target to be detected, the candidate box generator may not detect any target, so no detection candidate box will be generated, and the number of detection candidate boxes can be 0; when the original image includes one or more targets to be detected, the candidate box generator can detect one or more targets, so a detection candidate box will be generated, and the number of detection candidate boxes can be one or more.

[0099] The candidate box generator can be a strongly supervised learning object detection model trained with GT data. At the same time, transfer learning can be combined to further improve the capabilities of the object detection model (i.e., the candidate box generator).

[0100] After obtaining the candidate box information and confidence information of the original image, the candidate box information, confidence information and image-level label information of the original image can be input into the positive and negative bag and positive and negative example generator to generate positive and negative bags and positive and negative examples for each category target of the original image. For example, the number of categories of the detection task is 3, namely cars, bicycles, and traffic lights. For an image, if it includes 3 detection candidate boxes, and the categories corresponding to the three candidate areas are cars, bicycles, and traffic lights, respectively, then for the image candidate area including the car (i.e., the area corresponding to the detection candidate box including the car), it can be a positive bag of the car category, or a negative bag of the two categories of bicycles and traffic lights. For a part of the image candidate area, such as a pixel, it can be a positive example of the car category, or a negative example of the two categories of bicycles and traffic lights. It can be seen that a bag can be composed of multiple examples.

[0101] Afterwards, the positive and negative bags and positive and negative examples for each category of objects can be fed into the positive and negative example trainer to train a detection model that can distinguish the pixel types in each detection candidate box. Using the trained detection model, the multi-instance learning (MIL) interactor can detect objects in new input images.

[0102] See also Figure 3 , Figure 3 This is a flow chart of a target detection method disclosed in an embodiment of the present invention. Figure 3 As shown, the target detection method may include the following steps.

[0103] 301. Acquire multiple images and multiple label information, where the first label information is category information of the object to be detected included in the first image, and the first image is any image among the multiple images.

[0104] Specifically, in order to train the initial detection model, a trained detection model is obtained. The computer device first needs to obtain training data (i.e., multiple images and multiple label information) in order to train the initial detection model. Among them, multiple images correspond one to one with multiple label information, that is, one training image corresponds to one label information. The label information is the category information of the object to be detected included in the image, which is used to indicate which categories of targets (objects) the image includes. The first label information is the category information of the object to be detected included in the first image. The object to be detected can be understood as an object that needs to be detected, or it can be understood as an object of interest. For different detection tasks, the object to be detected can be different. Among them, the object can also be called a target. For example, for target detection of assisted driving, the object to be detected is generally a car, traffic lights, etc. For target detection of safety helmets / safety belts, the object to be detected is generally a safety belt, a safety helmet, etc.

[0105] For example, for a detection task, it is necessary to detect four categories of targets (objects), including cars, bicycles, traffic lights, and motorcycles. The category information of the object to be detected included in the image can be the name of the object's category. In the case where a training image includes objects of two categories, car and traffic light, the label information of the image can be [car, traffic light] or [car, traffic light]. Similarly, in the case where a training image includes a car, a bicycle, and a traffic light, the label information of the image can be [car, bicycle, traffic light]. The category information of the object to be detected included in the image can also be the identifier of the object's category. The identifier of the car can be 0, the identifier of the bicycle can be 1, the identifier of the traffic light can be 2, and the identifier of the motorcycle can be 3. In the case where a training image includes a car and a traffic light, the label information of the image can be [0, 2].

[0106] The category information of the objects to be detected included in the image can also be indicated by a one-dimensional array. The number of elements included in this array can be the same as the total number of categories to be detected, with each element in the array corresponding to a category. When detecting objects in four categories: cars, bicycles, traffic lights, and motorcycles, the array includes four elements. The first element can be used to indicate cars, the second element can be used to indicate bicycles, the third element can be used to indicate traffic lights, and the fourth element can be used to indicate motorcycles. An element value of 1 can indicate that an object of that category is included, while an element value of 0 can indicate that an object of that category is not included. If a training image includes a car and a traffic light, the label information for the image can be [1, 0, 1, 0], where the first element of 1 indicates that the image includes a car, the second element of 0 indicates that the image does not include a bicycle, the third element of 1 indicates that the image includes a traffic light, and the fourth element of 0 indicates that the image does not include a motorcycle. The above one-dimensional array can be used as a label during the model training process.

[0107] It should be understood that the label information corresponding to the image can be used to obtain the weakly supervised label of the image (ie, the true label), which can be used for model training.

[0108] It should be noted that the embodiments of the present invention do not limit the format of the aforementioned multiple images, and the images may be of any format, such as portable network graphics (PNG), bitmap images (BMP), and Joint Photographic Experts Group (JPEG) (one of the most commonly used image file formats). In addition, the aforementioned multiple images may be color images (RGB) having three color channels: red, green, and blue; or grayscale images having a luminance channel (Y component). The embodiments of the present application do not limit the image type.

[0109] 302. Input the first image into the first model to obtain a feature map of the candidate region and a corresponding confidence level.

[0110] After acquiring multiple images including the first image, the computer device can input the first image into the first model to obtain a feature map of the candidate area and a corresponding confidence level. The confidence level is used to indicate the probability that the corresponding candidate area is the object to be detected, that is, the probability that the corresponding candidate area is a target of any category that needs to be detected.

[0111] The first model can be a trained model. Specifically, in order to obtain a more accurate candidate frame for the first image (i.e., a frame that can more accurately frame the target to be detected in the first image), the computer device can use GT data to train a strongly supervised learning target detection model (supervised object detection, SOD), i.e., the first model. Such as fast_rcnn, faster_rcnn models, etc. Among them, the GT data used to train the first model is complete GT data. For a training image, it is necessary to mark all the targets that need to be detected included in the image, as well as the specific location of each target, etc. In order to obtain a model with a high detection accuracy, the GT data used should have a certain quantity and richness. At the same time, in order to match the current training task, the category of the GT data should include the categories to be detected included in the current training task. If the current training task is to train a detection model that can detect targets of two categories, cars and bicycles, then the GT data should include labeled data containing the two categories of cars and bicycles.

[0112] It should be understood that the first model can obtain the candidate area of ​​the input image (i.e., the area included in the candidate box) and the confidence corresponding to each candidate area. At the same time, the first model can output the feature map (feature_map) corresponding to each candidate area. Among them, the number of channels of the feature map of the candidate area (i.e., the number of matrix layers of the image) is related to the structure of the first model, such as the number of convolution kernels included in the first model. The feature map of the candidate area can be understood as the feature tensor of the candidate area, and the feature map can also be understood as a map that can represent the complete original image information after extraction by a convolutional neural network (CNN).

[0113] It should be noted that the confidence level can also be understood as the probability that the corresponding candidate region includes the object to be detected (ie, the probability of including any object to be detected).

[0114] 303. Input the feature map of the candidate region into the second model to obtain a detection score matrix and a classification score matrix.

[0115] Specifically, after the computer device obtains the feature map of the candidate area through the first model, it can input the feature map of the candidate area into the second model to obtain a detection score matrix and a classification score matrix. The first model and the second model are initial detection models.

[0116] The second model may include a detection branch and a classification branch. The detection branch may include a global average pooling layer and a first fully connected layer (FC), and the classification branch may include a global average pooling layer and a second fully connected layer. The parameters of the first fully connected layer and the second fully connected layer are different. The detection branch and the classification branch may share a global average pooling layer. In this case, the feature map of the candidate area is input into the detection branch to obtain a detection score matrix, and the feature map of the candidate area is input into the classification branch to obtain a classification score matrix.

[0117] Specifically, the computer device inputs the feature map of the candidate area into the detection branch, and first passes through a global average pooling layer to perform global average pooling on the feature maps of the R candidate areas to obtain a global feature map of the candidate area. The role of the global average pooling layer can be understood as extracting the global features of each candidate area, and can also reduce the number of parameters of the first fully connected layer, thereby reducing the risk of overfitting. Afterwards, the global feature map of the candidate area can be passed through a fully connected layer (i.e., the first fully connected layer) to obtain a detection score matrix. Among them, the dimension of the detection score matrix can be R*C, R is the number of candidate areas, and C is the number of categories of the detection task (i.e., the number of categories that need to be detected, corresponding to the categories included in the weak supervision labels of the image). For example, the categories that need to be detected can include cars, bicycles, traffic lights, motorcycles, pedestrians, and traffic signs, and the corresponding C is 6. Each row of the detection score matrix can correspond to a candidate area, and each column can correspond to a detection category. An element in the detection score matrix can be understood as the similarity score of a candidate region in a certain category (that is, the score of whether there is an object of this category), which can indicate the possibility that a candidate region includes an object of a certain category. The larger the value, the greater the possibility that the corresponding candidate region includes an object of the corresponding category, and the smaller the value, the smaller the possibility that the corresponding candidate region includes an object of the corresponding category.

[0118] The computer device inputs the feature map of the candidate area into the classification branch, and first passes through a global average pooling layer to perform global average pooling on the feature maps of the R candidate areas to obtain a global feature map of the candidate area. Afterwards, the computer device can pass the global feature map of the candidate area through a fully connected layer (i.e., the second fully connected layer) to obtain a classification score matrix. The dimension of the classification score matrix can be R*C. Each row of the classification score matrix can correspond to a candidate area, and each column can correspond to a target category. An element in the classification score matrix can indicate the possibility that a candidate area is an object of a certain category. The larger the value, the greater the possibility that the corresponding candidate area is an object of the corresponding category, and the smaller the value, the smaller the possibility that the corresponding candidate area is an object of the corresponding category.

[0119] It should be understood that the above-mentioned global average pooling layer can also be replaced by a maximum pooling layer (which can perform a maximum pooling operation) or other pooling layers.

[0120] It should be noted that since the detection branch and the classification branch have different functions, the fully connected layers of the detection branch and the classification branch cannot share parameters, and the corresponding fully connected layer parameters need to be trained separately.

[0121] 304. Determine a mean square error loss value and a cross entropy loss value based on the first label information, the confidence level, the detection score matrix, and the classification score matrix.

[0122] Specifically, the computer device may determine a prediction confidence based on the detection score matrix, and may also determine a predicted label based on the detection score matrix and the classification score matrix. Subsequently, the computer device may determine a mean square error loss value based on the confidence and the prediction confidence, and a cross entropy loss value based on the first label information and the predicted label.

[0123] Wherein, the second model may also include a normalization layer, and the computer device may determine the prediction confidence based on the detection score matrix, which may include: inputting the detection score matrix into the normalization layer to obtain a normalized matrix; determining the maximum value of each row in the normalized matrix as the prediction confidence (i.e., taking the maximum value of each row of the normalized matrix to obtain the prediction confidence), and the prediction confidence may be the prediction confidence corresponding to the candidate area. Specifically, the normalization layer may be a sigmod layer, and in the normalization layer, the computer device may perform a sigmod operation on each element of the detection score matrix to obtain a normalized matrix. Wherein, a value of the normalized matrix may be understood as the probability that a candidate area includes an object of a certain category to be detected. Taking the maximum value of each row of the normalized matrix may be understood as: taking the probability of a certain category of objects to be detected that is most likely to be included in a candidate area (i.e., the maximum value of each row) as the probability that the candidate area includes the object to be detected (i.e., any object of a category that needs to be detected). For example, the probability that a candidate region includes a car is 0.85, the probability that it includes a bicycle is 0.5, the probability that it includes a motorcycle is 0.6, and the probability that it includes a traffic light is 0.05. 0.85 can be used as the probability that the candidate region includes the object to be detected. The sigmoid function is shown in the following formula (1):

[0124]

[0125] Here, z can be an element in the detection score matrix, and f(z) can be the output of the element through the normalization layer.

[0126] See also Figure 4 , Figure 4 It is a schematic diagram of a geometric image of a sigmoid function disclosed in an embodiment of the present invention.

[0127] It should be understood that the computer device may also process the detection score matrix through other normalization functions, which is not limited in this embodiment of the present invention.

[0128] The computer device can determine the mean square error loss value based on the confidence level and the prediction confidence level. Specifically, in the embodiment of the present application, the computer device can use the mean square error loss (MSE) function to calculate the error between the confidence level and the prediction confidence level to obtain the mean square error loss value. The MSE function is shown in the following formula (2):

[0129]

[0130] Among them, J MSE can be the mean square error loss value, R can be the number of candidate regions, and y i Can be the prediction confidence of the i-th candidate region, It can be the confidence of the i-th candidate region.

[0131] The second model may further include a normalization layer, a first softmax layer, and a second softmax layer. The computer device may determine the predicted label according to the detection score matrix and the classification score matrix including the following steps.

[0132] First, the computer device can input the detection score matrix into the normalization layer to obtain a normalized matrix. The normalized matrix can be obtained through a sigmoid operation, and the specific description can refer to the above-mentioned related description. Then, the computer device can input the normalized matrix into the first softmax layer, perform a softmax operation on each column of the first matrix (that is, normalize all elements of each column so that the sum of all elements on each column is 1), calculate the category probability value of each row in the column, and at the same time, obtain the inter-class features of each candidate area to obtain the first matrix. The computer device can also first multiply the above-mentioned normalized matrix by a value β, and then perform a softmax operation, which can increase the separability of the classification task. β is a scalar greater than 1.

[0133] The computer device can input the classification score matrix into the second softmax layer, perform a softmax operation on each row of the classification score matrix (i.e., normalize all elements of each row so that the sum of all elements on each row is 1), and obtain the features of all candidate regions in a certain category to obtain a second matrix. Afterwards, the computer device can perform a Hadamard product matrix transformation on the first matrix and the second matrix (i.e., multiply the elements in the same position of the isotype matrix separately) to obtain a third matrix. The hadamard transformation can detect the total intensity of multiple signal combinations, reduce the mean square error of the signal and improve the signal-to-noise ratio. It is suitable for feature recognition and can extract features between multiple categories. Finally, the computer device can perform a summation operation on each column of the third matrix to obtain a predicted label. The predicted label can be a predicted label for the first image. Each element in the predicted label can represent the score of the first image including a certain category. The higher the score, the more likely the image is to include the object to be detected in that category. The softmax function is shown in the following formula (3):

[0134]

[0135] Among them, i can be an element in a row of the classification score matrix, ∑ j e j Can be the sum of the indices of all elements in the row, S i It can be the output of the softmax operation on this element.

[0136] The computer device can determine a cross-entropy loss value based on the first label information and the predicted label. Specifically, in an embodiment of the present application, the computer device can obtain a weakly supervised label (i.e., a true label) of the first image based on the first label information. Thereafter, the computer device can use a cross entropy loss (CE_loss) function to calculate the error between the weakly supervised label and the predicted label to obtain a cross entropy loss value. The cross entropy loss function can calculate the gap between the predicted probability distribution and the standard probability distribution.

[0137] 305. Optimize the parameters of the initial detection model according to the mean square error loss value and the cross entropy loss value to obtain a trained detection model.

[0138] Because R and C can be different, directly linearly fusing the classification loss and the cross-entropy loss results in an imbalanced contribution to the final loss. Therefore, the computer can multiply the cross-entropy loss by the weight R / C to obtain a weighted cross-entropy loss, where C is the number of categories of objects to be detected and R is the number of candidate regions. The computer can then determine the total loss as the sum of the weighted cross-entropy loss and the mean square error loss.

[0139] The computer device can optimize the parameters of the initial detection model based on the total loss value to obtain a trained detection model. This trained detection model can be used to detect objects to be detected in the image (i.e., objects of all categories corresponding to the true labels of the first image). Thus, the total loss value can be used to further fine-tune the parameters of the first model, thereby improving the performance of the first model.

[0140] Since the cross entropy loss value (i.e., the classification loss value) is multiplied by a weight (R / C) here and then added to the mean square error loss value (i.e., the detection loss value) to obtain the total loss value, the classification loss value and the detection loss value can be made to contribute the same when optimizing the initial detection model, so that in the process of loss reverse propagation, after the model parameters are corrected by the gradient descent algorithm, the classification loss value and the detection loss value of the initial detection model can be balanced. This avoids focusing only on reducing the classification loss value or the detection loss value during model optimization, and the target model obtained by training performs more evenly in the detection accuracy and classification accuracy of the target, without focusing on any one aspect. A detection model with high detection accuracy and classification accuracy can be obtained. Moreover, since it is a detection model obtained based on weakly supervised label training, the detection model has good robustness.

[0141] It is understandable that during the iterative training of the model, when the total loss value is less than the preset model error, the computer device can stop training to obtain a trained detection model. Optionally, the computer device can set a preset number of iterations for the initial detection model and record the number of training iterations of the initial detection model. When the number of training iterations is equal to the preset number of iterations, the training of the initial detection model can be stopped, and the model with the number of training iterations equal to the preset number of iterations can be determined as the final trained detection model. At the same time, during the model training process, the total loss values ​​of multiple different images can be obtained first, and then the total loss values ​​of the multiple images can be averaged to obtain an average total loss value. Afterwards, the model parameters can be optimized based on the average total loss value.

[0142] In this embodiment of the present invention, model training only requires simple image-level label information, and when calculating the loss value, the cross-entropy loss value can be multiplied by a weight (R / C) and then added to the mean square error loss value to obtain a total loss value. The trained detection model has good algorithmic effects, and its performance in indicators such as mean average precision (MAP), average precision (AP), recall rate, and intersection over union (IOU) of the detected targets is good, and the false detection rate is low.

[0143] Among them, the samples fed into the initial detection model can be data collected by the machine during operation. These data are naturally random, diverse, and massive. For example, the different data collected by the vehicle during operation (such as image data) can be generated by manual annotation in the early stage, and can be generated by the model itself after the model accuracy is high in the later stage. For example, if the detection model (including the first model and the second model) trained by multiple images already has excellent detection capabilities, and the detection accuracy and classification accuracy are high (such as the detection accuracy and classification accuracy have reached a specific threshold (such as 95%)), then the image-level labels of the data can all be generated by the second model itself (that is, it can be targeted at the image), so as to achieve "self-labeling" of the data and "self-training" of the model, and realize "online learning".

[0144] For example, for an unlabeled image, it can be input into a trained detection model to obtain the target detection frame and the category corresponding to each detection frame, and then the corresponding image-level label can be generated based on these categories. Specifically, assume that the trained detection model can detect four categories of targets (objects), including cars, bicycles, traffic lights, and motorcycles. Then, a captured image can be input into the detection model. Assuming that three detection frames can be obtained (i.e., three targets can be detected), one of the detection frames corresponds to the category of car, and the other two detection frames correspond to the category of traffic lights, then the image-level label of the image can be generated based on the obtained detection results (i.e., the detection frames and the corresponding categories). For example, the generated image-level label can be [1, 0, 1, 0], where the first element is 1, which means that the image includes a car, the second element is 0, which means that the image does not include a bicycle, the third element is 1, which means that the image includes a traffic light, and the fourth element is 0, which means that the image does not include a motorcycle.

[0145] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a target detection model disclosed in an embodiment of the present invention. Figure 5As shown, the computer device can first obtain multiple images, the first label information is the category information of the object to be detected included in the first image, and the first image is any image in the multiple images. For the first image, the candidate box generator can obtain R candidate regions of the first image and the confidence of each candidate region. In addition, the candidate box generator can output the feature map of each candidate region, so that R feature maps of size W*H*CH can be obtained, where CH can be the number of channels of the feature map, W can be the width of the feature map, and H can be the height of the feature map. Afterwards, the R feature maps can be input into the classification branch and the detection branch. In the detection branch, the R feature maps are first input into the global average pooling layer, and global average pooling processing is performed to obtain R feature maps of size CH*1*1. Then, after a fully connected layer (i.e., the first fully connected layer mentioned above), R feature maps of size C*1*1 can be obtained. Finally, after dimensionality reduction (i.e., view operation), a detection score matrix of size R*C can be obtained. Wherein, C is the number of categories of the detection task.

[0146] The computer can then input the detection score matrix into a normalization layer, such as a sigmoid layer, to generate a normalized matrix through a sigmoid operation. Finally, the maximum value in each row of the normalized matrix is ​​retained to obtain the prediction confidence. The prediction confidence is R*1, where each value corresponds to a candidate region. The mean square error loss (MSE) can then be calculated based on the prediction confidence and the confidence score.

[0147] In the classification branch, similar to the detection branch, the R feature maps are first fed into a global average pooling layer for global average pooling, resulting in R feature maps of size CH*1*1. These are then fed into a fully connected layer (the second fully connected layer mentioned above) to obtain R feature maps of size C*1*1. Finally, a dimensionality reduction (i.e., a view operation) is performed to obtain a classification score matrix of size R*C. The classification score matrix is ​​then fed into a softmax layer, where a softmax operation is performed on each row of the matrix, resulting in a second matrix of size R*C. Similarly, the normalized matrix is ​​first multiplied by a scalar β. The resulting matrix is ​​then fed into a softmax layer, where a softmax operation is performed on each column of the matrix, resulting in a first matrix. The Hadamard product matrix operation is then performed on the first and second matrices to obtain a third matrix. Finally, the predicted labels are summed across each row of the third matrix to obtain the predicted labels. The predicted labels are of size C*1, where each value corresponds to a class.

[0148] Afterwards, the cross-entropy loss can be calculated based on the true and predicted labels. This cross-entropy loss can then be multiplied by the weight (R / C) and added to the detection loss to obtain the total loss. Finally, the total loss can be propagated backwards to optimize the model parameters using stochastic gradient descent.

[0149] See also Figure 6 , Figure 6 FIG. 1 is a flow chart of another target detection method disclosed in an embodiment of the present invention. Figure 6 As shown, the target detection method may include the following steps.

[0150] 601. Acquire multiple images, where the second image is any image among the multiple images.

[0151] The labels of the acquired images are unknown (i.e., there are no rectangular box labels, category labels, etc.). For example, the images may be external images acquired in real time by smart car cameras or industrial cameras.

[0152] 602. Input the second image into the first model to obtain the candidate region and the feature map of the candidate region.

[0153] After acquiring multiple images including the second image, the computer device can input the first image into the first model, and can obtain the candidate area and the feature map of the candidate area. Here, the candidate boxes of all objects to be detected in the second image can be obtained, such as the specific position of each candidate box, which can be expressed as (x, y, w, h), where (x, y) represents the coordinate position of the center point of an object to be detected in the second image (the lower left corner of the second image can be used as the coordinate origin), w represents the distance to the left and right sides of the center point, and h represents the distance to the upper and lower sides of the center point. The first model can be obtained by Figure 3 The first model after fine-tuning in the embodiment shown. Figure 3 The relevant descriptions in the method embodiment shown are not repeated here.

[0154] 603. Input the feature map of the candidate area into the second model to obtain a detection score matrix and a classification score matrix.

[0155] Step 603 is similar to the above-mentioned step 303. Please refer to the relevant description of step 303 and will not be described in detail here.

[0156] 604. Obtain the category corresponding to the candidate region according to the detection score matrix and the classification score matrix.

[0157] Specifically, the computer device can obtain an image score matrix (i.e., the third matrix in step 304) based on the detection score matrix and the classification score matrix. The specific steps for obtaining the image score matrix are the same as the steps for obtaining the third matrix in step 304, and reference can be made to the above related description.

[0158] The computer device can obtain the category corresponding to the candidate area based on the image score matrix. Specifically, after the computer device obtains the image score matrix (R*C), since each column of the image score matrix can correspond to a category, each row can correspond to a candidate area. Therefore, for each row of the image score matrix (i.e., a candidate area), the maximum value in the row can be determined first, and the category corresponding to the column of the maximum value is used as the category corresponding to the candidate area. The candidate area (i.e., candidate box) obtained by step 602 and the category corresponding to the candidate area obtained by step 604 complete the target recognition of the second image.

[0159] See also Figure 7 , Figure 7 It is a schematic diagram of a detection result disclosed in an embodiment of the present invention. First, the computer device can obtain training data (multiple images and corresponding labels), and then train the above-mentioned initial detection model to obtain a trained model. For example, the real labels of the training data include four categories: cars, traffic lights, bicycles, and motorcycles. After the detection model is trained, the detection model can be used to identify the targets (objects) of the four categories of cars, traffic lights, bicycles, and motorcycles in an image. Figure 7 As shown in Figure 1, the detection model can output the detection box of the detected target and the specific category of the object in the target box.

[0160] See also Figure 8 , Figure 8 : is a schematic diagram of the structure of a target detection device disclosed in an embodiment of the present invention. The target detection device can be a computer device or a module in a computer device. Figure 8 As shown, the device may include:

[0161] An acquisition unit 801 is configured to acquire a plurality of images and a plurality of label information; the plurality of images correspond one-to-one to the plurality of label information, wherein the first label information is the category information of the object to be detected included in the first image, and the first image is any image in the plurality of images;

[0162] A first processing unit 802 is configured to input the first image into a first model to obtain a feature map of a candidate region and a corresponding confidence level;

[0163] A second processing unit 803 is configured to input the feature map of the candidate region into a second model to obtain a detection score matrix and a classification score matrix, wherein the first model and the second model belong to an initial detection model;

[0164] A determining unit 804 is configured to determine a mean square error loss value and a cross entropy loss value based on the first label information, the confidence level, the detection score matrix, and the classification score matrix;

[0165] The optimization unit 805 is used to optimize the parameters of the initial detection model according to the mean square error loss value and the cross entropy loss value to obtain a trained detection model; the detection model is used to detect the object to be detected in the image.

[0166] In one embodiment, the first model is a strongly supervised learning target detection model trained with real-valued data.

[0167] In one embodiment, the second model includes a detection branch and a classification branch; the second processing unit 803 is specifically configured to:

[0168] Inputting the feature map of the candidate region into the detection branch to obtain the detection score matrix, the detection branch includes a global average pooling layer and a first fully connected layer;

[0169] The feature map of the candidate region is input into the classification branch to obtain the classification score matrix, where the classification branch includes a global average pooling layer and a second fully connected layer, and the parameters of the first fully connected layer and the second fully connected layer are different.

[0170] In one embodiment, the determining unit 804 determines the mean square error loss value and the cross entropy loss value according to the first label information, the confidence, the detection score matrix, and the classification score matrix, including:

[0171] Determining prediction confidence based on the detection score matrix;

[0172] Determining a predicted label based on the detection score matrix and the classification score matrix;

[0173] Determining a mean square error loss value according to the confidence level and the prediction confidence level;

[0174] A cross entropy loss value is determined according to the first label information and the predicted label.

[0175] In one embodiment, the second model further includes a normalization layer, and the determining unit 804 determines the prediction confidence according to the detection score matrix including:

[0176] Input the detection score matrix into the normalization layer to obtain a normalized matrix;

[0177] The maximum value of each row in the normalized matrix is ​​determined as the prediction confidence.

[0178] In one embodiment, the second model further includes a normalization layer, a first softmax layer, and a second softmax layer, and the determination unit 804 determines the predicted label according to the detection score matrix and the classification score matrix, including:

[0179] Input the detection score matrix into the normalization layer to obtain a normalized matrix;

[0180] Inputting the normalized matrix into the first softmax layer to obtain a first matrix;

[0181] Inputting the classification score matrix into the second softmax layer to obtain a second matrix;

[0182] Performing a Hadamard product matrix transformation on the first matrix and the second matrix to obtain a third matrix;

[0183] The predicted label is obtained by performing a sum operation on each column of the third matrix.

[0184] In one embodiment, the optimization unit 805 is specifically configured to:

[0185] The cross entropy loss value is multiplied by the weight R / C to obtain the weighted cross entropy loss value; C is the number of categories of objects to be detected, and R is the number of candidate regions;

[0186] The sum of the weighted cross entropy loss value and the mean square error loss value is determined as the total loss value;

[0187] The parameters of the initial detection model are optimized according to the total loss value.

[0188] For a more detailed description of the acquisition unit 801, the first processing unit 802, the second processing unit 803, the determination unit 804 and the optimization unit 805, please refer to the above Figure 3 The relevant description in the illustrated method embodiment is directly available and will not be repeated here.

[0189] See Figure 9 , Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 9As shown, the computer device 900 may include: at least one processor 901, such as a CPU, at least one memory 905, and at least one communication bus 902. Optionally, the computer device 900 may also include at least one network interface 904 and a user interface 903. The communication bus 902 is used to achieve connection and communication between these components. The user interface 903 may include a display screen (display) and a keyboard (keyboard), and the network interface 904 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 905 may be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 905 may optionally also be at least one storage device located away from the aforementioned processor 901. As Figure 9 As shown, the memory 905 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application.

[0190] exist Figure 9 In the computer device 900 shown, the network interface 904 can provide network communication functions; the user interface 903 is mainly used to provide an interface for user input; and the processor 901 can be used to call the device control application stored in the memory 905 to achieve:

[0191] Acquire multiple images and multiple label information; the multiple images correspond one-to-one to the multiple label information, the first label information is the category information of the object to be detected included in the first image, and the first image is any image in the multiple images;

[0192] Inputting the first image into the first model to obtain a feature map of the candidate region and a corresponding confidence score;

[0193] Inputting the feature map of the candidate region into the second model to obtain a detection score matrix and a classification score matrix, wherein the first model and the second model belong to an initial detection model;

[0194] Determining a mean square error loss value and a cross entropy loss value according to the first label information, the confidence, the detection score matrix, and the classification score matrix;

[0195] Optimizing the parameters of the initial detection model according to the mean square error loss value and the cross entropy loss value to obtain a trained detection model; the detection model is used to detect the object to be detected in the image.

[0196] It should be understood that the computer device 900 described in the embodiment of the present application can be used to perform the above Figure 3 The method executed by the computer device in the method embodiment will not be described in detail here.

[0197] An embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed, executes the method in the above method embodiment.

[0198] An embodiment of the present invention further discloses a computer program product comprising instructions, which, when executed, performs the method in the above method embodiment.

[0199] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation methods of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of this application should be included in the scope of protection of this application.

Claims

1. A target detection method, characterized in that: include: Get multiple images and multiple label information; The multiple images correspond to the multiple label information in a one-to-one manner, the first label information is the category information of the object to be detected included in the first image, and the first image is any image among the multiple images; Inputting the first image into a first model to obtain a feature map of the candidate region and a corresponding confidence score; The first model is a strongly supervised learning target detection model trained with real-valued data; Inputting the feature map of the candidate region into the detection branch of the second model to obtain a detection score matrix; Inputting the feature map of the candidate region into the classification branch of the second model to obtain a classification score matrix; The first model and the second model are initial detection models; Determining a mean square error loss value and a cross entropy loss value according to the first label information, the confidence level, the detection score matrix, and the classification score matrix; Multiplying the cross entropy loss value by a weight R / C to obtain a weighted cross entropy loss value; C is the number of categories of the object to be detected, and R is the number of candidate regions; Determine the sum of the weighted cross entropy loss value and the mean square error loss value as the total loss value; Optimizing the parameters of the initial detection model according to the total loss value to obtain a trained detection model; the detection model is used to detect the object to be detected in the image.

2. The method according to claim 1, characterized in that The detection branch includes a global average pooling layer and a first fully connected layer; the classification branch includes the global average pooling layer and a second fully connected layer, and the parameters of the first fully connected layer and the second fully connected layer are different.

3. The method according to claim 2, characterized in that The determining the mean square error loss value and the cross entropy loss value according to the first label information, the confidence, the detection score matrix, and the classification score matrix includes: Determining prediction confidence based on the detection score matrix; Determining a prediction label based on the detection score matrix and the classification score matrix; Determining a mean square error loss value according to the confidence level and the prediction confidence level; A cross entropy loss value is determined based on the first label information and the predicted label.

4. The method according to claim 3, characterized in that The second model further includes a normalization layer, and determining the prediction confidence according to the detection score matrix includes: Inputting the detection score matrix into the normalization layer to obtain a normalized matrix; The maximum value of each row in the normalized matrix is ​​determined as the prediction confidence.

5. The method according to claim 3, characterized in that The second model further includes a normalization layer, a first softmax layer, and a second softmax layer, and determining the predicted label according to the detection score matrix and the classification score matrix includes: Inputting the detection score matrix into the normalization layer to obtain a normalized matrix; Inputting the normalized matrix into the first softmax layer to obtain a first matrix; Inputting the classification score matrix into the second softmax layer to obtain a second matrix; Performing a Hadamard product matrix transformation on the first matrix and the second matrix to obtain a third matrix; A sum operation is performed on each column of the third matrix to obtain the predicted label.

6. A target detection device, characterized in that: include: An acquisition unit, used for acquiring multiple images and multiple label information; The multiple images correspond to the multiple label information in a one-to-one manner, the first label information is the category information of the object to be detected included in the first image, and the first image is any image among the multiple images; a first processing unit, configured to input the first image into a first model to obtain a feature map of a candidate region and a corresponding confidence score; The first model is a strongly supervised learning target detection model trained with real-valued data; a second processing unit, configured to input the feature map of the candidate region into a detection branch of a second model to obtain a detection score matrix, and to input the feature map of the candidate region into a classification branch of the second model to obtain a classification score matrix, wherein the first model and the second model are initial detection models; a determining unit, configured to determine a mean square error loss value and a cross entropy loss value according to the first label information, the confidence level, the detection score matrix, and the classification score matrix; An optimization unit, configured to multiply the cross entropy loss value by a weight R / C to obtain a weighted cross entropy loss value, wherein C is the number of categories of the object to be detected, and R is the number of the candidate regions; The optimization unit is further configured to determine the sum of the weighted cross entropy loss value and the mean square error loss value as a total loss value; The optimization unit is further used to optimize the parameters of the initial detection model according to the total loss value to obtain a trained detection model; the detection model is used to detect the object to be detected in the image.

7. A computer device, characterized in that: include: a memory and a processor; wherein: The memory is used to store a computer program, wherein the computer program includes program instructions; The processor is configured to call the program instructions so that the computer device executes the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program or computer instructions, and when the computer program or computer instructions are executed, the method according to any one of claims 1 to 5 is implemented.

Images