Model training method, joint detection method, joint detection device and storage medium

Abstract

This application provides a model training method, a joint detection method, a joint detection device, and a computer-readable storage medium. The model training method includes: acquiring a training image and extracting target features from the training image; inputting the target features into a first detection head and a second detection head of the model to be trained, respectively, to obtain the overall detection box output by the first detection head and the component detection boxes output by the second detection head; obtaining the positional constraint loss value of the training image using the positional relationship between the overall detection box and the component detection box; obtaining the mutual supervision constraint loss value of the detection heads for the training image using the overall loss value of the overall detection box and the component loss value of the component detection box; and training the model to be trained based on the positional constraint loss value and the mutual supervision constraint loss value of the detection heads. Through the above method, the joint detection device improves model training efficiency and enhances model detection performance by adding key component detection and associating key components with the overall model during training.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a model training method, a joint detection method, a joint detection device, and a computer-readable storage medium. Background Technology

[0002] With the development of deep learning theory, computer vision-based detection algorithms have been applied in numerous fields such as security, transportation, and agriculture. Although object detection algorithms exhibit strong generalization capabilities, false positives and false negatives still exist in specific application scenarios. In recent years, fires caused by elevators entering residential buildings have resulted in significant property damage and casualties. Local governments have issued bans strictly prohibiting users from pushing electric vehicles into enclosed spaces such as residential buildings.

[0003] Currently, the solution to prevent users from pushing electric vehicles into elevators involves integrating object detection algorithms with the elevator to immediately trigger an alarm upon detecting an electric vehicle. However, in practical applications, it has been found that bicycles, children's toy cars, strollers, and other items are highly similar to standard electric vehicles, making it easy for ordinary algorithms to trigger false alarms and lead to user complaints. Furthermore, users deliberately obscuring electric vehicles can also cause the algorithm to miss detections of non-compliant vehicles. Summary of the Invention

[0004] This application provides a model training method, a joint detection method, a joint detection device, and a computer-readable storage medium.

[0005] This application provides a model training method, the model training method comprising:

[0006] Acquire the image to be trained and extract the target features from the image to be trained;

[0007] The target features are respectively input into the first detection head and the second detection head of the model to be trained to obtain the overall detection box output by the first detection head and the component detection box output by the second detection head.

[0008] The positional constraint loss value of the image to be trained is obtained by utilizing the positional relationship between the overall detection box and the component detection box;

[0009] Using the overall loss value of the overall detection box and the component loss value of the component detection box, the mutual supervision constraint loss value of the detection head of the image to be trained is obtained;

[0010] The model to be trained is trained based on the position constraint loss value and the mutual supervision constraint loss value of the detection heads.

[0011] The step of obtaining the mutual supervision constraint loss value of the detection heads in the image to be trained by using the overall loss value of the overall detection box and the component loss value of the component detection box includes:

[0012] Based on the attention levels of the first and second detection heads, obtain the mutual supervision weights;

[0013] The weighted sum of the overall loss value and the component loss value is obtained based on the mutual supervision weights to obtain the mutual supervision constraint loss value of the detection head.

[0014] The position constraint loss value includes the detection box inclusion constraint loss value and the center point offset constraint loss value;

[0015] The step of obtaining the positional constraint loss value of the image to be trained by utilizing the positional relationship between the overall detection box and the component detection box includes:

[0016] Obtain the overlap relationship between the overall detection box and the component detection box, and obtain the constraint loss value included in the detection box of the image to be trained;

[0017] Obtain the inclusion relationship between the center point of the component detection box and the overall detection box, and obtain the center point offset constraint loss value of the image to be trained.

[0018] The step of obtaining the overlap relationship between the overall detection box and the component detection box, and obtaining the constraint loss value of the detection box in the image to be trained, includes:

[0019] Obtain the intersection of the overall detection frame and the component detection frame;

[0020] Based on the ratio of the intersection to the component detection frame, it is determined that the detection frame contains constraint loss values.

[0021] The step of obtaining the inclusion relationship between the center point of the component detection box and the overall detection box, and obtaining the center point offset constraint loss value of the image to be trained, includes:

[0022] Obtain the first distance between the center point of the component detection frame and the center point of the overall detection frame;

[0023] Obtain the second distance between the center point and the corner point of the overall detection frame;

[0024] The center point offset constraint loss value is determined based on the difference between the first distance and half of the second distance.

[0025] The overall detection frame is the vehicle body detection frame, and the component detection frame is the front detection frame and / or the rear detection frame.

[0026] This application provides a joint detection method, the joint detection method comprising:

[0027] Acquire multiple consecutive frames of input image;

[0028] The component positions and overall positions of each frame of the input image are obtained using a pre-trained joint detection model.

[0029] Obtain the positional relationship between the component positions and the overall position of each frame of the input image;

[0030] Determine whether the joint detection conditions are met based on the positional relationship;

[0031] If so, then obtain the number of consecutive frames of the input image that satisfy the joint detection conditions;

[0032] When the number of consecutive frames is greater than or equal to a preset frame number threshold, information indicating the presence of a target is output.

[0033] The joint detection model is trained using the model training method described above.

[0034] The step of obtaining the positional relationship between the component positions and the overall position of each frame of the input image includes:

[0035] Obtain the third distance between the center point of the component's position and the center point of the overall position;

[0036] Obtain the dynamic distance between the center point and the corner points of the overall position;

[0037] When the third distance is less than the dynamic distance, the positional relationship between the component position and the overall position is determined to satisfy the joint detection condition.

[0038] The step of obtaining the positional relationship between the component positions and the overall position of each frame of the input image includes:

[0039] Obtain the component corner coordinates at the component's location, and the global corner coordinates at the overall location;

[0040] When the upper left corner coordinate of the component is greater than the upper left corner coordinate of the overall corner coordinate, and the upper right corner coordinate of the component is less than the upper right corner coordinate of the overall corner coordinate, the positional relationship between the component position and the overall position is determined to satisfy the joint detection condition.

[0041] The joint detection method, after obtaining the component corner coordinates of the component position and the overall corner coordinates of the overall position, further includes:

[0042] When the upper left corner coordinate of the component is less than the upper left corner coordinate of the overall corner coordinate, or the upper right corner coordinate of the component is greater than the upper right corner coordinate of the overall corner coordinate, the area intersection of the component position and the overall position is obtained.

[0043] When the area intersection is greater than or equal to a preset intersection, the positional relationship between the component position and the overall position is determined to satisfy the joint detection condition.

[0044] After obtaining the part positions and overall positions of each frame of the input image using a pre-trained joint detection model, the joint detection method further includes:

[0045] Obtain the component confidence score for the component location and the overall confidence score for the overall location;

[0046] When the confidence level of a component is greater than the confidence level of a component, and the confidence level of the whole is greater than the confidence level of the whole, the positional relationship between the component position and the overall position of each frame of the input image is obtained.

[0047] This application also provides a joint detection device, which includes a processor and a memory, wherein the memory stores program data, and the processor is used to execute the program data to implement the model training method and / or the joint detection method as described above.

[0048] This application also provides a computer-readable storage medium for storing program data, which, when executed by a processor, is used to implement the above-described model training method and / or joint detection method.

[0049] The beneficial effects of this application are as follows: A joint detection device acquires the image to be trained and extracts the target features from it; the target features are input into the first and second detection heads of the model to be trained, respectively, to obtain the overall detection box output by the first detection head and the component detection boxes output by the second detection head; the positional constraint loss value of the image to be trained is obtained using the positional relationship between the overall detection box and the component detection box; the mutual supervision constraint loss value of the detection heads for the image to be trained is obtained using the overall loss value of the overall detection box and the component loss value of the component detection box; the model to be trained is then trained based on the positional constraint loss value and the mutual supervision constraint loss value of the detection heads. Through the above method, the joint detection device improves model training efficiency and enhances model detection performance by adding key component detection and associating key components with the overall model during training. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0051] Figure 1 This is a flowchart illustrating an embodiment of the model training method provided in this application;

[0052] Figure 2 This is a schematic diagram of the overall process of the joint detection method for electric vehicles illegally entering elevators based on the characteristics of key components provided in this application;

[0053] Figure 3 This is a schematic diagram of the structure of the elevator electric vehicle testing model provided in this application;

[0054] Figure 4 This is a schematic diagram of an embodiment of the structured labeling of electric vehicles provided in this application;

[0055] Figure 5 This is a coordinate schematic diagram of an embodiment of the overall detection frame and component detection frame provided in this application;

[0056] Figure 6 This is a coordinate schematic diagram of another embodiment of the overall detection frame and component detection frame provided in this application;

[0057] Figure 7 This is a schematic flowchart of an embodiment of the joint detection method provided in this application;

[0058] Figure 8 This is a schematic diagram of the overall process of the combined detection method provided in this application;

[0059] Figure 9 This is a schematic diagram of an embodiment of the elevator monitoring area provided in this application;

[0060] Figure 10 This is a detailed flowchart of the priori inductive judgment alarm module provided in this application;

[0061] Figure 11 This is a schematic diagram of the structure of an embodiment of the combined testing equipment provided in this application;

[0062] Figure 12 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0064] Please refer to details. Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating an embodiment of the model training method provided in this application. Figure 2 This is a schematic diagram of the overall process of the joint detection method for illegal entry of electric vehicles into elevators based on the characteristics of key components provided in this application.

[0065] The model training method of this application is applied to a joint detection device, which can be a server or a system consisting of a server and a terminal device working together. Accordingly, the various parts of the joint detection device, such as units, sub-units, modules, and sub-modules, can be all located in the server, or they can be located separately in the server and the terminal device.

[0066] Furthermore, the aforementioned server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules, such as software or software modules used to provide distributed servers, or as a single software program or software module; no specific limitation is made here. In some possible implementations, the image processing method of this application embodiment can be implemented by a processor calling computer-readable instructions stored in memory.

[0067] like Figure 2 As shown, the proposed joint detection method for electric vehicles illegally entering elevators addresses the issues of false positives and false negatives in existing target detection algorithms in elevator scenarios. It designs a method for detecting illegal electric vehicle entry, implemented using deep learning technology based on computer vision. The main process is shown in the figure. Figure 2 .

[0068] First, we analyze the shortcomings of single-stage target detection algorithms in elevator scenarios and design an algorithm to detect illegal entry of electric vehicles into elevators. Second, we use a monocular visible light camera to acquire real-time images of people entering the elevator. Third, we input the video stream into the elevator detection algorithm. The main detection head is responsible for detecting major categories of targets such as people, bicycles, and electric vehicles, while the auxiliary detection head is responsible for detecting key components of the front and rear of electric vehicles. The recognition results of the two detection heads are passed to the next module. Fourth, we use the key feature distribution of the front and rear of electric vehicles as prior inductive knowledge and design a prior inductive judgment module. The module jointly judges the output results of the main and auxiliary detection heads. If the front and rear of the vehicle meet the judgment logic, then the illegal target is detected.

[0069] The training process of the electric vehicle unauthorized entry detection algorithm designed in this application is as follows: Figure 1 As shown, the model training method in this embodiment of the application specifically includes the following steps:

[0070] Step S11: Obtain the image to be trained and extract the target features of the image to be trained.

[0071] In this embodiment, the specific structure of the model to be trained, which combines the electric vehicle illegal entry detection algorithm, is first described. Please refer to the figure for details. Figure 3 , Figure 3 This is a schematic diagram of the elevator electric vehicle testing model provided in this application.

[0072] Existing YOLO detection models can be structurally divided into two main parts: a feature extraction network and a detection head. Taking YOLOv4 as an example, the network includes a darkNet feature extraction network and a single detection head. If this electric vehicle illegal entry detection algorithm is directly applied to elevator scene detection, it is prone to triggering false alarms for items with high similarity to electric vehicles and cannot handle deliberately occluded electric vehicles.

[0073] Therefore, this application is based on the YOLOv4 algorithm and improves upon it as follows: Figure 3 The network structure shown in this application is presented, and a joint loss function is proposed. This joint loss function guides the model to fit towards a high detection rate during the algorithm training process. The improved method proposed in this application can be extended to the YOLO series of algorithms.

[0074] like Figure 3 As shown, the elevator electric vehicle detection model of this application uses the default detection head of the YOLO algorithm as the main detection head. Based on the detection decoupling idea, an auxiliary detection head that focuses on the key components of the electric vehicle is added to the original YOLO model, and a joint loss function is added to enhance the model's ability to perceive electric vehicle targets. Figure 3In the diagram, from left to right, the Backbone represents the feature extraction network, which can be implemented using classification models such as ResNet50. The main detector head is the Detect part of the original YOLO algorithm. The newly added auxiliary detector head is implemented in the same way as the main detector head, and its function is responsible for detecting the front and rear of the electric vehicle. Both the main detector head and the auxiliary detector head are responsible for predicting the target's category information, coordinate position, and confidence level. The difference lies in the objects that different detector heads are responsible for.

[0075] In the embodiments of this application, Figure 1 The model training method shown uses data from elevator scenarios as training samples. The algorithm imposes specific requirements on the data: the samples must not only include common target categories such as electric vehicles, bicycles, and people, but also add two categories: the front and rear of electric vehicles. The joint detection equipment utilizes... Figure 3 The Backbone feature extraction network shown extracts target features from the image to be trained.

[0076] Step S12: Input the target features into the first and second detection heads of the model to be trained, respectively, to obtain the overall detection box output by the first detection head and the component detection box output by the second detection head.

[0077] In this embodiment of the application, the joint detection device inputs the target features extracted by the feature extraction network into the main detection head and the auxiliary detection head respectively. The main detection head is used to predict the vehicle body detection box, its coordinate position and the vehicle body confidence score in the input image, which is classified as electric vehicle body. The auxiliary detection head is used to predict the component detection box, its coordinate position and the component confidence score in the input image, which is classified as electric vehicle component.

[0078] Electric vehicle components include the front and rear of the electric vehicle.

[0079] Step S13: Use the positional relationship between the overall detection box and the component detection box to obtain the positional constraint loss value of the image to be trained.

[0080] In this embodiment of the application, the model loss function of the model to be trained consists of three parts, namely Loss 主 Loss 辅 Loss 联合 The mode loss function is defined as follows:

[0081] Loss = Loss 主 +Loss 辅 +Loss 联合

[0082] Among them, Loss 主 and Loss 辅These are the target loss functions for the main and auxiliary detection heads, respectively. They consist of three parts: confidence loss, localization loss, and classification loss. The definition of this loss function is consistent with that of the YOLO algorithm.

[0083] Loss of this application 联合 Specifically, it includes three different constraint loss functions, Loss 联合 The main description is as follows:

[0084] Loss 联合 =Loss constraint_1 +Loss constraint_2 +Loss constraint_3

[0085] Among them, Loss constraint_1 Loss is a loss function that includes constraints on the detection box. constraint_2 Loss is the center point offset constraint loss function. constraint_3 The loss function is the mutual supervision constraint for the detection heads.

[0086] Due to Loss constraint_1 and Loss constraint_2 All are related to location, Loss constraint_1 and Loss constraint_2 This can be collectively referred to as the position constraint loss function. In other embodiments, the position constraint loss function can be Loss. constraint_1 and Loss constraint_2 One of the constraint loss functions in this context will not be elaborated upon here.

[0087] The specific details of the constraint loss function included in the detection box are as follows:

[0088] Please see Figure 4 and Figure 5 , Figure 4 This is a schematic diagram of an embodiment of the structured labeling for electric vehicles provided in this application. Figure 5 This is a coordinate schematic diagram of an embodiment of the overall detection frame and component detection frame provided in this application.

[0089] Analysis of elevator parking case studies, such as Figure 4 As shown, box A represents the main body of the electric vehicle, box B represents the front part of the electric vehicle, and box C represents the front part of the electric vehicle.

[0090] Based on prior knowledge: the bounding box of an electric vehicle body must include the bounding boxes of the front and rear components; that is, the component bounding boxes are subsets of the main bounding boxes. A schematic diagram of the coordinates of the overall detection box and component detection boxes in the input image is shown below. Figure 5 As shown. Figure 5 The coordinates in the text represent the top-left and bottom-right positions of the detection box. Loss constraint_1The formula constraints are as follows:

[0091]

[0092] Box 车身 =|x1-x2|*|y1-y2|

[0093] Box 部件 =|x3-x4|*|y3-y4|

[0094] Among them, Box * The regions represent different target detection boxes. (x1, y1) are the coordinates of the upper left position of the vehicle body detection box, (x2, y2) are the coordinates of the lower right position of the vehicle body detection box, (x3, y3) are the coordinates of the upper left position of the component detection box, and (x4, y4) are the coordinates of the lower right position of the component detection box.

[0095] The detection box contains a constraint loss function that describes two cases: 1. The part is inside the body frame, and the result is 0; 2. The part is offset from the body frame and does not satisfy the subset relationship, and the result is between 0 and +∞. In order to reflect the detection error in the loss, the corresponding loss value is calculated using the function.

[0096] The specific details of the center point offset constraint loss function are as follows:

[0097] Please see Figure 6 , Figure 6 This is a coordinate schematic diagram of another embodiment of the overall detection frame and component detection frame provided in this application.

[0098] Figure 6 This diagram represents the coordinates of the center point of the detection frame, where (a, b) is the center point of the component (denoted by C2), (m, n) is the center point of the vehicle body (denoted by C1), and (x, y) are the coordinates of a fixed point on the vehicle body frame (denoted by C3). The basic principle of the center point constraint is that the center point of the component must be within the vehicle body detection frame. This constraint is used to impose positional constraints on the detection results, mathematically described as follows:

[0099]

[0100]

[0101] Where N is the number of component detection boxes; when the center point of the component detection box is contained within the vehicle body detection box, y in =0, otherwise 1; Dst describes the drift of the center point of the component detection box. When the component detection box satisfies the center constraint, the result is less than 0, otherwise it is greater than 0; ||*||2 is the norm, used to calculate the Euclidean distance.

[0102] Step S14: Using the overall loss value of the overall detection box and the part loss value of the part detection box, obtain the mutual supervision constraint loss value of the detection head of the image to be trained.

[0103] In this embodiment of the application, the specific description of the mutual supervision constraint loss function of the detection heads is as follows:

[0104] To address the limitations of different detection heads in the algorithm, this application proposes a mutual supervision constraint loss function for detection heads, avoiding the problem of one detection head detecting a target while another fails to do so. The mutual supervision constraint loss function in this application reuses the detection head function to reduce computation. When the primary detection head predicts the target but the secondary detection head fails to detect it, the loss function needs to increase the penalty for the secondary detection head; conversely, the same applies if the primary detection head fails to detect it.

[0105] The mathematical description of the mutual supervision constraint loss function of the detection heads in this application is as follows:

[0106] Loss constraint_3 =α·Loss 主 +β·Loss 辅

[0107] Wherein, α and β are weighting coefficients used to balance the model’s attention to different detection heads. The prior can be set as: α = 0.5, β = 1.

[0108] Step S15: Train the model to be trained based on the position constraint loss value and the mutual supervision constraint loss value of the detection heads.

[0109] In this embodiment of the application, the loss function of step S13 and step S14 is used to train the model to be trained in order to obtain the joint detection model required in the joint detection method.

[0110] In this embodiment, the joint detection device acquires the image to be trained and extracts target features from it. These target features are then input into the first and second detection heads of the model to be trained, respectively, to obtain the overall detection box output by the first detection head and the component detection boxes output by the second detection head. The positional constraint loss value of the image to be trained is obtained using the positional relationship between the overall and component detection boxes. The mutual supervision constraint loss value between the detection heads of the image to be trained is obtained using the overall loss value of the overall detection boxes and the component loss value of the component detection boxes. Based on the positional constraint loss value and the mutual supervision constraint loss value between the detection heads, the model to be trained is trained. Through this method, the joint detection device improves model training efficiency and enhances model detection performance by adding key component detection and associating key components with the overall model during training.

[0111] This application analyzes the pain points of false detection and missed detection of electric vehicles in elevator scenarios and designs a targeted electric vehicle detection model. This algorithm can not only detect regular electric vehicles and people, but also avoid false detections of similar targets, such as children's toy cars and bicycles. For electric vehicles intentionally obscured by people, the algorithm can pinpoint key components, greatly improving accuracy. This application provides structured requirements for the samples used to train the algorithm, adding annotations of key electric vehicle components (front and rear) to the original targets such as people, electric vehicles, and bicycles. This application proposes an auxiliary detection head focusing on key electric vehicle components; it designs an association loss function by linking the results of the main detection head and the auxiliary detection head to guide model learning during algorithm training; and it innovatively proposes inclusion constraints, center point offset constraints, and mutual supervision constraints between detection heads.

[0112] Please continue reading. Figure 2 , Figure 7 and Figure 8 , Figure 7 This is a schematic flowchart of an embodiment of the joint detection method provided in this application. Figure 8 This is a schematic diagram of the overall process of the combined detection method provided in this application.

[0113] Specifically, such as Figure 7 As shown, the joint detection method in this application embodiment specifically includes the following steps:

[0114] Step S21: Acquire multiple consecutive frames of input images.

[0115] In the embodiments of this application, such as Figure 2 As shown, the joint detection equipment acquires surveillance video data, including multiple consecutive input images.

[0116] Specifically, a visible light monocular smart camera is installed in the elevator to capture real-time footage after the elevator opens. The surveillance video is encoded and decoded to transmit the image information to an intelligent algorithm. Different algorithmic monitoring areas are preset based on different elevator scene angles. When an object appears in that area, such as... Figure 9 The elevator monitoring area shown in the diagram is where subsequent modules activate the alarm function for electric vehicles illegally entering the elevator.

[0117] Step S22: Use a pre-trained joint detection model to obtain the part positions and overall positions of each frame of the input image.

[0118] In the embodiments of this application, such as Figure 8 The algorithm flowchart shown illustrates how the joint detection equipment transmits the elevator footage captured by the camera to the electric vehicle detection algorithm in real time. Figure 1The trained joint detection model uses two detection heads to identify different targets: the main detection head is responsible for detecting human bodies and non-motorized vehicles, while the auxiliary detection head is responsible for predicting key components of electric vehicles.

[0119] After the main detection head detects the target, it triggers the next judgment step, and the information flow continues. The information from the dual detection heads is passed to the prior induction judgment module.

[0120] Step S23: Obtain the positional relationship between the component positions and the overall position of each frame of the input image.

[0121] For a detailed flowchart of the priori inductive judgment alarm module provided in this application, please refer to the embodiments described in this application. Figure 10 .

[0122] Specifically, the judgment logic in the prior inductive judgment alarm module of this application is determined based on the spatial positional relationship between the key components of the electric vehicle and the electric vehicle body. The prior inductive judgment alarm module determines whether to trigger an alarm based on the detection results of consecutive time frames; detecting a violation target in 10 consecutive frames is considered as detecting an electric vehicle target.

[0123] The prior inductive judgment alarm module has four detection states: Initialization (creation), Update (update), Delete (clear), and Alarm (alarm).

[0124] The joint detection equipment initiates the prior inductive judgment alarm module. When the electric vehicle algorithm detection result is input into the prior inductive judgment alarm module, the Initialization creation state is triggered. The prior inductive judgment alarm module judges the category and logic of the main and auxiliary detection heads. If the detection result meets all judgment logic, the Update state is triggered, and the continuous frame count recorder records the number of times. If the count recording is interrupted during the judgment process of the time sequence frame, the state is first updated to Delete to clear, then the recorder count is reset to zero and the state is updated to Initialization, indicating that the current detection is a false alarm and the warning is lifted. If the count is greater than the judgment threshold of the continuous frames, an alarm is finally triggered, and the module state is updated to Alarm.

[0125] Step S24: Determine whether the joint detection conditions are met based on the positional relationship.

[0126] In this embodiment, the joint detection device determines whether the image meets the joint detection conditions based on the positional relationship between the component position and the overall position.

[0127] like Figure 10 As shown, the detailed process of the priori inductive judgment alarm module is as follows:

[0128] (1) The algorithm detection result is transmitted to the prior inductive judgment alarm module, the activation module status is Initialization, and then (2) is entered.

[0129] (2) After updating the status, the data stream enters the logic of separating the auxiliary detection head results, extracts the prediction results of the auxiliary detection head from the stream, and then enters (3).

[0130] (3) The prior inductive judgment alarm module makes a preliminary judgment on the prediction result of the auxiliary detection head. If the detection result contains one or more of the categories of electric vehicle front or electric vehicle rear, it triggers the subsequent judgment and enters (4); otherwise, the judgment ends and returns to the algorithm logic.

[0131] (4) Based on the features of the front and rear of the electric vehicle, set a category filtering threshold th1. If the ConfOD confidence level is greater than the threshold th1, proceed to (5); otherwise, end.

[0132] (5) Separate the prediction results of the main detection head in the data stream. If the result is empty, then end; otherwise, proceed to (6).

[0133] (6) Set the key detection category: electric vehicles. Determine the detection results separated from the main detection head, filter all categories, and if the target category of electric vehicles is included, proceed to (7); otherwise, return and end the judgment.

[0134] (7) Set the electric vehicle judgment threshold th2. If the result passed in (6), that is, the confidence of the electric vehicle target category, is greater than the threshold th2, proceed to the next step (8); otherwise, end.

[0135] (8) Start the prior judgment logic, package the main detection head result and the auxiliary detection head result into a meta dictionary, and pass it to (9).

[0136] (9) First, the Euclidean distance from the center point of the electric vehicle key component detection box to the center point of the electric vehicle body detection box is calculated using the norm. The calculation method is based on the formula of the center point offset constraint loss function. Then, based on the coordinates of the center point and the corner of the electric vehicle body detection box, the Euclidean distance is also calculated as the dynamic threshold using the norm. Finally, the relationship between the center point distance and the dynamic threshold is judged. If the center point distance is less than the dynamic threshold th3, then proceed to (10); otherwise, the logic ends.

[0137] (10) Calculate the area of ​​the key components (front and rear) of the electric vehicle and the electric vehicle body. Based on the upper left and lower right coordinates of the vehicle body detection frame, if any x or y coordinate in the upper left coordinate of the component detection frame is less than the x or y value of the upper left coordinate of the vehicle body detection frame, it indicates that the electric vehicle component is offset from the electric vehicle body. Similarly, calculate the relationship between the lower right corner coordinate of the component detection frame and the lower right corner coordinate of the vehicle body detection frame. If it is greater, it indicates that an offset has occurred. Then calculate the intersection of the areas of the component detection frame and the vehicle body detection frame. If it is not 0, it indicates that there is a detection offset between the auxiliary detection head and the main detection head, but it is still in the target association state, and proceed to step (11); otherwise, the judgment ends. If the x and y coordinates in the upper left coordinate of the component detection frame are greater than the x and y coordinates in the upper left coordinate of the vehicle body detection frame, and the x and y coordinates in the lower right coordinate of the component detection frame are less than the x and y coordinates in the lower right coordinate of the vehicle body detection frame, it indicates that no offset has occurred, and proceed directly to step (11).

[0138] (11) Clear the prediction results of the auxiliary detection head in the data stream and pass the processing results to (12).

[0139] (12) Determine whether the electric vehicle has entered the specified judgment area. If it has, increment the count counter, record the number of detections, update the status to Update, and then proceed to (13); otherwise, end.

[0140] If the current frame image does not meet the joint detection conditions based on the positional relationship, the counter is set to 0, and the number of consecutive frames of the input image that meet the joint detection conditions is counted again.

[0141] Step S25: Obtain the number of consecutive frames of the input image that meet the joint detection conditions.

[0142] Step S26: When the number of consecutive frames is greater than or equal to a preset frame number threshold, output information indicating the presence of a target.

[0143] In the embodiments of this application, such as Figure 10 As shown, the detailed process of the priori inductive judgment alarm module also includes:

[0144] (13) Determine the relationship between the count count and the preset frame number threshold th4. If it is greater than the count count, change the alarm status to Alarm and trigger the alarm. If it is less than the count count, change the module status from Update to Delete, clear the count counter, and then update the status to Initialization to enter the next round of judgment results.

[0145] Based on algorithm design principles, this application associates electric vehicle components with the vehicle body to construct complete subsequent logic, thus improving the alarm scheme for electric vehicles illegally entering elevators. It also designs a novel prior inductive judgment alarm module that uses the predictive information from the auxiliary detection head to provide accurate results.

[0146] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0147] To implement the model training method and / or joint detection method described in the above embodiments, this application also proposes a joint detection device, which can be found in the following details. Figure 11 , Figure 11 This is a schematic diagram of an embodiment of the combined testing equipment provided in this application.

[0148] The joint detection device 300 of this application embodiment includes a memory 31 and a processor 32, wherein the memory 31 and the processor 32 are coupled together.

[0149] The memory 31 is used to store program data, and the processor 32 is used to execute the program data to implement the model training method and / or joint detection method described in the above embodiments.

[0150] In this embodiment, processor 32 can also be referred to as a CPU (Central Processing Unit). Processor 32 may be an integrated circuit chip with signal processing capabilities. Processor 32 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 32 can be any conventional processor.

[0151] To implement the model training method and / or joint detection method of the above embodiments, this application also provides a computer-readable storage medium, such as... Figure 12 As shown, the computer-readable storage medium 400 is used to store program data 41, which, when executed by a processor, is used to implement the model training method and / or joint detection method as described in the above embodiments.

[0152] This application also provides a computer program product, wherein the computer program product includes a computer program operable to cause a computer to perform the image processing method as described in the embodiments of this application. The computer program product may be a software installation package.

[0153] The image processing method described in the above embodiments of this application, when implemented as a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0154] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A model training method, characterized in that, The model training method includes: Acquire the image to be trained and extract the target features from the image to be trained; The target features are respectively input into the first detection head and the second detection head of the model to be trained to obtain the overall detection box output by the first detection head and the component detection box output by the second detection head. The positional constraint loss value of the image to be trained is obtained by utilizing the positional relationship between the overall detection box and the component detection box; Using the overall loss value of the overall detection box and the component loss value of the component detection box, the mutual supervision constraint loss value of the detection head of the image to be trained is obtained; The model to be trained is trained based on the position constraint loss value and the mutual supervision constraint loss value of the detection heads; The step of obtaining the mutual supervision constraint loss value of the detection heads for the image to be trained by using the overall loss value of the overall detection box and the component loss value of the component detection box includes: Based on the attention levels of the first and second detection heads, obtain the mutual supervision weights; Based on the mutual supervision weights, the weighted sum of the overall loss value and the component loss value is obtained to obtain the mutual supervision constraint loss value of the detection head; The position constraint loss value includes the detection box inclusion constraint loss value and the center point offset constraint loss value; The step of obtaining the positional constraint loss value of the image to be trained by utilizing the positional relationship between the overall detection box and the component detection box includes: Obtain the overlap relationship between the overall detection box and the component detection box, and obtain the constraint loss value included in the detection box of the image to be trained; Obtain the inclusion relationship between the center point of the component detection box and the overall detection box, and obtain the center point offset constraint loss value of the image to be trained.

2. The model training method according to claim 1, characterized in that, The step of obtaining the overlap relationship between the overall detection box and the component detection box, and obtaining the constraint loss value of the detection box in the image to be trained, includes: Obtain the intersection of the overall detection frame and the component detection frame; Based on the ratio of the intersection to the component detection frame, it is determined that the detection frame contains constraint loss values.

3. The model training method according to claim 1, characterized in that, The step of obtaining the inclusion relationship between the center point of the component detection box and the overall detection box, and obtaining the center point offset constraint loss value of the image to be trained, includes: Obtain the first distance between the center point of the component detection frame and the center point of the overall detection frame; Obtain the second distance between the center point and the corner point of the overall detection frame; The center point offset constraint loss value is determined based on the difference between the first distance and half of the second distance.

4. The model training method according to any one of claims 1 to 3, characterized in that, The overall detection frame is the vehicle body detection frame, and the component detection frame is the front detection frame and / or the rear detection frame.

5. A combined detection method, characterized in that, The joint detection method includes: Acquire multiple consecutive frames of input image; The component positions and overall positions of each frame of the input image are obtained using a pre-trained joint detection model. Obtain the positional relationship between the component positions and the overall position of each frame of the input image; Determine whether the joint detection conditions are met based on the positional relationship; If so, then obtain the number of consecutive frames of the input image that satisfy the joint detection conditions; When the number of consecutive frames is greater than or equal to a preset frame number threshold, information indicating the presence of a target is output. The joint detection model is trained using the model training method described in any one of claims 1 to 4.

6. The combined detection method according to claim 5, characterized in that, The step of obtaining the positional relationship between the component positions and the overall position of each frame of the input image includes: Obtain the third distance between the center point of the component's position and the center point of the overall position; Obtain the dynamic distance between the center point and the corner points of the overall position; When the third distance is less than the dynamic distance, the positional relationship between the component position and the overall position is determined to satisfy the joint detection condition.

7. The combined detection method according to claim 5 or 6, characterized in that, The step of obtaining the positional relationship between the component positions and the overall position of each frame of the input image includes: Obtain the component corner coordinates at the component's location, and the global corner coordinates at the overall location; When the upper left corner coordinate of the component is greater than the upper left corner coordinate of the overall corner coordinate, and the upper right corner coordinate of the component is less than the upper right corner coordinate of the overall corner coordinate, the positional relationship between the component position and the overall position is determined to satisfy the joint detection condition.

8. The combined detection method according to claim 7, characterized in that, After obtaining the component corner coordinates of the component position and the global corner coordinates of the overall position, the joint detection method further includes: When the upper left corner coordinate of the component is less than the upper left corner coordinate of the overall corner coordinate, or the upper right corner coordinate of the component is greater than the upper right corner coordinate of the overall corner coordinate, the area intersection of the component position and the overall position is obtained. When the area intersection is greater than or equal to a preset intersection, the positional relationship between the component position and the overall position is determined to satisfy the joint detection condition.

9. The combined detection method according to claim 5, characterized in that, After obtaining the part positions and overall positions of each frame of the input image using a pre-trained joint detection model, the joint detection method further includes: Obtain the component confidence score for the component location and the overall confidence score for the overall location; When the confidence level of a component is greater than the confidence level of a component, and the confidence level of the whole is greater than the confidence level of the whole, the positional relationship between the component position and the overall position of each frame of the input image is obtained.

10. A combined detection device, characterized in that, The joint detection device includes a processor and a memory, the memory storing program data, and the processor executing the program data to implement the model training method as described in any one of claims 1 to 4, and / or the joint detection method as described in any one of claims 5 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program data, which, when executed by a processor, is used to implement the model training method according to any one of claims 1 to 4, and / or the joint detection method according to any one of claims 5 to 9.

Images