A method, device and electronic equipment for determining spillage

Abstract

The present application provides a method, device and electronic device for determining spillage, and relates to the field of intelligent monitoring technology. In the present application, firstly, a key video frame in which spillage is detected is obtained from the monitoring video screen, and the vehicle target features and spillage features in the key video frame are extracted; then, K frames of candidate video frames adjacent to the key video frame are selected, and the features of each scale extracted from the K frames of candidate video frames are input into a multi-layer perceptron to obtain global features of each scale; the vehicle target features are spliced ​​with the global features of each scale to obtain fusion features of each scale; finally, based on the feature interaction results of the fusion features of each scale and the features of the spillage features, the vehicle target to which the spillage belongs is determined. In this way, the determination problem of the spillage and the vehicle where the spillage occurs can be converted into a target matching problem, thereby improving the accuracy and efficiency of the spillage determination.

Description

Technical Field

[0001] The present application relates to the field of intelligent monitoring technology, and in particular to a method, device and electronic equipment for determining spillage. Background Art

[0002] With the development of urbanization, there are an increasing number of large and medium-sized engineering vehicles and other motor vehicles on urban roads. The phenomenon of vehicles spilling and dripping during driving is becoming more and more frequent, which not only affects the urban environment, but also poses risks to the safe passage of passing vehicles.

[0003] In order to deal with the spillage and dripping of vehicles on urban roads and reduce the risk of spillage and dripping to the safe passage of vehicles, in the related technology, the traditional method 1 proposes a method to determine whether there is spillage in the area of ​​the object to be processed based on the feature similarity between the target area and the area of ​​the object to be processed. However, when using method 1 to detect spillage, if the mass of the spillage is light, the position of the spillage may change. At this time, the detector will mistakenly detect the same spillage at different positions as different spillages. Summary of the invention

[0004] The present invention provides a method, device and electronic device for determining the spillage determination, which are used to determine the target vehicle sending the spillage and improve the accuracy of the spillage determination. The specific technical solution is as follows:

[0005] In a first aspect, the present application provides a method for determining spillage, comprising:

[0006] From the monitoring video screen, a key video frame in which the spilled object is detected is obtained, and vehicle target features and spilled object features in the key video frame are extracted;

[0007] Selecting K frames of candidate video frames adjacent to the key video frame, and inputting the features of each scale extracted from the K frames of candidate video frames into a multi-layer perceptron to obtain global features of each scale;

[0008] Splicing the vehicle target features with the global features of each scale to obtain fused features of each scale;

[0009] Based on the feature interaction results of the fusion features of each scale and the features of the scattered objects, the vehicle target to which the scattered objects belong is determined.

[0010] By interacting the fused features of each scale (including local features of vehicle targets, position features, and global features of each scale), the problem of determining the spilled objects and the vehicle where the spillage occurs can be converted into a target matching problem. This allows for better attention to be paid to the correlation between the features of the vehicle target where the spillage occurs and the features of the spilled objects, thereby improving the accuracy and efficiency of spillage determination.

[0011] In a possible implementation, determining the vehicle target to which the scattered object belongs based on the feature interaction result of the fusion features of each scale and the features of the scattered object includes:

[0012] Performing feature interaction on the fusion features of each scale and the features of the spilled object to obtain interaction features of each scale, and splicing the interaction features of each scale to obtain target fusion features;

[0013] The target fusion features are input into the fully connected layer and the output layer to obtain the predicted value of each vehicle target to which the scattered objects belong, and based on the predicted value, the vehicle target to which the scattered objects belong is determined.

[0014] By interacting the features of the spilled objects with the fusion features of each scale, we can better focus on the correlation between the features of the vehicle target where the spillage occurs and the features of the spilled objects, thereby improving the accuracy of spillage judgment.

[0015] In a possible implementation, if the fusion features of each scale include a first fusion feature Y1, a second fusion feature Y2, and a third fusion feature Y3, then the fusion features of each scale are subjected to feature interaction with the feature of the spilled object to obtain interaction features of each scale, including:

[0016] The Y1, Y2, Y3 and the feature F of the thrown object p Input multi-layer perceptron, and get , , and ;

[0017] Calculate the , , With the The first fusion feature after feature interaction , the second fusion feature and the third fusion feature :

[0018] ;

[0019] ;

[0020] ;

[0021] Among them, λ1 is the adjustment parameter; is the sigmoid function;

[0022] Based on the , and stated , determine the characteristics of the scattered objects after interaction :

[0023] ;

[0024] ;

[0025] Among them, multi-head attention means multi-head attention; ×2 means upsampling by 2 times; ×4 means upsampling by 4 times;

[0026] Calculate the , , With the The interactive features of each scale after feature interaction , , :

[0027] ;

[0028] ;

[0029] ;

[0030] Among them, λ2 is the adjustment parameter; is the sigmoid function.

[0031] Based on the above method, the problem of determining the scattered objects and the vehicle targets where the scattered objects occur can be converted into a target matching problem through the feature interaction structure, thereby improving the accuracy and speed of the scattered objects determination.

[0032] In a possible implementation, the loss value of the spilled object belonging to the vehicle target is determined by the following loss function:

[0033]

[0034] ;

[0035] Among them, L focal is the focal loss function; i It is used to identify whether the ith vehicle has scattered. If the ith vehicle has scattered, then ɑ i =1, if the ith vehicle does not scatter, then i =0;d i 2 represents the Euclidean distance between the i-th vehicle and the scattered object; d + 2represents the Euclidean distance between the scattered objects and the target vehicle where the scattering occurs; m is the minimum distance between the target vehicle where the scattering occurs and the vehicle where no scattering occurs.

[0036] Based on the above method, the parameters of the judgment model can be optimized to improve the accuracy of the judgment model prediction.

[0037] In a possible implementation, the step of selecting K candidate video frames adjacent to the key video frame, and inputting the features of each scale extracted from the K candidate video frames into a multi-layer perceptron to obtain global features of each scale includes:

[0038] Extracting, from the K frames of video frames to be selected, a first scale feature, a second scale feature, and a third scale feature corresponding to each frame of the video frames to be selected;

[0039] The first scale feature, the second scale feature, and the third scale feature corresponding to each frame of the selected video are input into the multi-layer perceptron to obtain the global features of each scale.

[0040] Through the above method, global features of each scale can be obtained.

[0041] In a possible implementation, extracting the vehicle target features in the key video frame includes:

[0042] Determine each vehicle target in the key video frame, and extract the vehicle local features and vehicle position features of each vehicle target;

[0043] Tracking each vehicle target based on a preset target tracking algorithm to determine a tracking identifier for each vehicle target;

[0044] According to the tracking identifier, the vehicle local features and vehicle position features of each of the vehicle targets are combined to obtain the vehicle target features.

[0045] Through the above method, vehicle target features including vehicle local features and vehicle position features and spilled object features including spilled object local features and spilled object position features can be extracted from key video frames, making the extracted target features richer and preparing data for subsequent accurate spillage judgment.

[0046] In a second aspect, the present application provides a spillage determination device, comprising:

[0047] A data acquisition module is used to acquire key video frames where spilled objects are detected from the monitoring video screen, and to extract vehicle target features and spilled object features in the key video frames;

[0048] Selecting K frames of candidate video frames adjacent to the key video frame, inputting the features of each scale extracted from the K frames of candidate video frames into a multi-layer perceptron to obtain global features of each scale;

[0049] A data splicing module, used for splicing the vehicle target features with the global features of each scale to obtain fusion features of each scale;

[0050] A feature interaction module is used to determine the vehicle target to which the scattered object belongs based on the feature interaction results of the fusion features of each scale and the features of the scattered object.

[0051] In a possible implementation, the feature interaction module is specifically used to:

[0052] Performing feature interaction on the fusion features of each scale and the features of the spilled object to obtain interaction features of each scale, and splicing the interaction features of each scale to obtain target fusion features;

[0053] The target fusion features are input into the fully connected layer and the output layer to obtain the probability value of the scattered objects belonging to each vehicle target, and based on the probability value, the vehicle target to which the scattered objects belong is determined.

[0054] In a possible implementation, the feature interaction module is specifically used to:

[0055] The Y1, Y2, Y3 and the feature F of the thrown object p Input multi-layer perceptron, and get , , and ;

[0056] Calculate the , , With the The first fusion feature after feature interaction , the second fusion feature and the third fusion feature :

[0057] ;

[0058] ;

[0059] ;

[0060] Among them, λ1 is the adjustment parameter; is the sigmoid function;

[0061] Based on the , and stated , determine the characteristics of the scattered objects after interaction :

[0062] ;

[0063] ;

[0064] Among them, multi-head attention means multi-head attention; ×2 means upsampling by 2 times; ×4 means upsampling by 4 times;

[0065] Calculate the , , With the The interactive features of each scale after feature interaction , , :

[0066] ;

[0067] ;

[0068] ;

[0069] Among them, λ2 is the adjustment parameter; is the sigmoid function.

[0070] In a possible implementation, the data acquisition module is specifically used to:

[0071] Extracting, from the K frames of video frames to be selected, a first scale feature, a second scale feature, and a third scale feature corresponding to each frame of the video frames to be selected;

[0072] The first scale feature, the second scale feature, and the third scale feature corresponding to each frame of the selected video are input into the multi-layer perceptron to obtain the global features of each scale.

[0073] In a possible implementation, the data acquisition module is specifically used to:

[0074] Determine each vehicle target in the key video frame, and extract the vehicle local features and vehicle position features of each vehicle target;

[0075] Tracking each vehicle target based on a preset target tracking algorithm to determine a tracking identifier for each vehicle target;

[0076] According to the tracking identifier, the vehicle local features and vehicle position features of each of the vehicle targets are combined to obtain the vehicle target features.

[0077] In a third aspect, the present application provides an electronic device, including:

[0078] Memory, used to store computer programs;

[0079] The processor is used to implement the steps of the above-mentioned spillage determination method when executing the computer program stored in the memory.

[0080] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned spillage determination method are implemented.

[0081] For each aspect from the second to the fourth aspect and the technical effects that may be achieved by each aspect, please refer to the above description of the technical effects that can be achieved by the first aspect or various possible schemes in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 A flow chart of a method for determining a spillage provided in an embodiment of the present application;

[0083] Figure 2 A schematic diagram of the architecture of a spill determination system provided in an embodiment of the present application;

[0084] Figure 3 A flow chart of an overall spill determination provided in an embodiment of the present application;

[0085] Figure 4 A structural diagram of a feature interaction provided in an embodiment of the present application;

[0086] Figure 5 A schematic diagram of the structure of a spillage determination device provided in an embodiment of the present application;

[0087] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The specific operating methods in the method embodiments can also be applied to device embodiments or system embodiments. It should be noted that in the description of the present application, "multiple" is understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A is connected to B, which can represent: A is directly connected to B and A is connected to B through C. In addition, in the description of the present application, words such as "first" and "second" are only used to distinguish the purpose of description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0089] The embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0090] With the development of urbanization, there are an increasing number of large and medium-sized engineering vehicles and other motor vehicles on urban roads. The phenomenon of vehicles spilling and dripping during driving is becoming more and more frequent, which not only affects the urban environment, but also poses risks to the safe passage of passing vehicles.

[0091] In order to deal with the spillage and dripping of vehicles on urban roads and reduce the risks of spillage and dripping to the safe passage of vehicles, in the related technology, traditional method one proposes a method for determining whether there is spillage in the area of ​​the object to be processed based on the feature similarity between the target area and the area of ​​the object to be processed. In method one: first obtain the reference background image and the image frame to be processed of the target shooting scene; then update the reference background image based on the non-interest area of ​​the image frame to be processed to obtain the target background image, wherein the non-interest area includes other areas in the image frame to be processed except the vehicle area to be processed and the object area to be processed; determine the target object area in the target background image that has the same position information as the object area to be processed; finally, determine whether the object area to be processed contains spillage based on the feature similarity between the target object area and the object area to be processed.

[0092] In addition, in the second traditional method: first, a target image frame in the target video is acquired; when a target vehicle is detected from the target image frame, a reference image frame before the target image frame is determined from the target video; the reference image frame includes the target vehicle and is an image frame with the least target objects other than the target vehicle; based on the target image frame and the reference image frame, it is determined whether there is any spilled object falling from the target vehicle.

[0093] However, when using method one to detect scattered objects, if the mass of the scattered object is light, the position of the scattered object may change. At this time, the detector will mistakenly detect the same scattered object in different positions as different scattered objects. Similar to method one, since method two detects scattered objects in the area around the target vehicle, if the target vehicle is traveling at a high speed and the mass of the scattered object is light, the scattered object will be thrown far away from the target vehicle. This may result in the scattered object and the target vehicle not matching, resulting in false detection.

[0094] In view of this, in order to determine the vehicle target where the spillage occurs and improve the accuracy of the spillage judgment, the present application provides a spillage judgment method, which specifically includes: first, obtaining the key video frame where the spilled object is detected from the monitoring video screen, and extracting the vehicle target features and the spilled object features in the key video frame; then selecting K frames of candidate video frames adjacent to the key video frame, and inputting the features of each scale extracted from the K frames of candidate video frames into a multi-layer perceptron to obtain global features of each scale; splicing the vehicle target features with the global features of each scale to obtain fused features of each scale; finally, based on the feature interaction results of the fused features of each scale and the features of the spilled object, determining the vehicle target to which the spilled object belongs.

[0095] By interacting the fused features of each scale (including local features of vehicle targets, position features, and global features of each scale), the problem of determining the scattered objects and the vehicle sending the scattered objects can be converted into a problem of target matching. This allows the determination model to better focus on the correlation between the features of the vehicle target where the scattered objects are located and the features of the scattered objects, thereby improving the accuracy and efficiency of the spillage determination.

[0096] Reference Figure 1 As shown, it is a flow chart of a method for determining a spillage provided in an embodiment of the present application, the method comprising:

[0097] S1, from the monitoring video screen, obtain the key video frame where the spilled object is detected, and extract the vehicle target features and the spilled object features in the key video frame.

[0098] First of all, the method provided in this application can be applied to Figure 2 In the system architecture shown, the system architecture includes: an image acquisition device, an image processing device and a target terminal, and the method provided in the present application can be executed in the image processing device.

[0099] The image acquisition device is used to acquire images of urban roads and send the acquired images to the image processing device in real time for corresponding processing. The image acquisition device can be a camera or a camera. This application does not impose specific restrictions on the type and quantity of the image acquisition device.

[0100] The image processing device is used to receive images uploaded by the image acquisition device and process the images according to a preset processing logic, for example, to extract feature information of the image, identify and track various targets in the image (such as spilled objects, vehicles, pedestrians, etc.), or determine the vehicle target where the spillage occurred.

[0101] The target terminal is used to receive and record the information of the vehicle target where the spillage occurred reported by the image processing device, and notify the maintenance personnel to clean up the spilled objects in the monitored area.

[0102] In the embodiments of this application, see the attached Figure 3 As shown, when the image processing device obtains the key video frame in which the spilled objects are detected, the image processing device can first define a monitoring area for the image acquisition device, and then the image acquisition device sends the acquired monitoring video to the image processing device; the image processing device receives the monitoring video and uses a preset target detection algorithm (such as the YOLO series model, the RCNN series model) to detect targets in the monitoring video screen, and the targets include motor vehicles, non-motor vehicles, pedestrians, spilled objects, etc., and this application does not impose specific restrictions on this.

[0103] In addition, in order to reduce the computational complexity of target detection and improve the rate of target detection, the image processing device can perform target detection on the video frames every N frames, and obtain the key video frames where the spilled objects are detected from the monitoring video screen, where N is an integer greater than or equal to 1.

[0104] After obtaining the key video frame, the image processing device can extract the vehicle target features and the scattered object features in the key video frame.

[0105] Specifically, when extracting vehicle target features, the image acquisition device can first determine each vehicle target in the key video frame, and each vehicle target can be represented by a vehicle target box (x vi ,y vi , w vi ,h vi ), where vi represents the i-th vehicle target; (x vi ,y vi ) represents the center coordinate of the i-th vehicle target box, (w vi ,h vi ) represents the width and height of the i-th vehicle target box.

[0106] The image acquisition device uses a feature extraction network to extract the local features of each vehicle target. and vehicle position characteristics.

[0107] In order to enrich the features of the extracted vehicle targets, the image acquisition device extracts the local features of each vehicle target. After obtaining the vehicle position characteristics, each vehicle target can be tracked according to a preset target tracking algorithm to determine the tracking identifier of each vehicle target. The tracking identifier can be a unique identification code of the vehicle, vehicle ID, etc., and this application does not impose any specific restrictions on this.

[0108] The image acquisition device can fuse the local features and vehicle position features of each vehicle target according to the vehicle tracking identification to obtain the vehicle target features.

[0109] Based on the same method as the vehicle target feature acquisition method, when extracting the features of the scattered objects, the image acquisition device can first determine the scattered objects in the key video frame. The scattered objects can also be represented by the scattered object target box (x p ,y p , w p ,h p ) to represent it; then extract the local features of the scattered objects and the location characteristics of the spilled objects; finally, the local characteristics of the spilled objects The features of the scattered objects are combined with the features of the scattered objects’ positions to obtain the features of the scattered objects. The repeated parts will not be repeated here.

[0110] Through the above method, vehicle target features including vehicle local features and vehicle position features and spilled object features including spilled object local features and spilled object position features can be extracted from key video frames, making the extracted target features richer and preparing data for subsequent accurate spillage judgment.

[0111] S2, select K frames of candidate video frames adjacent to the key video frame, and input the features of each scale extracted from the K frames of candidate video frames into a multi-layer perceptron to obtain global features of each scale.

[0112] In the embodiment of the present application, after acquiring the key video frame, the image processing device can also select K frames of candidate video frames adjacent to the key video frame. For example, (K-1) / 2 frames are selected before and after the time node of the key video frame where the spilled object is detected, where K is an integer greater than or equal to 1; then the selected K frames of candidate video frames are input into the trained detection network to extract the first scale feature corresponding to each frame of the candidate video frame. , second scale features and third scale features ,in, The size is (C1, W, H), The size is (C2, W / 2, H / 2), The size is (C3, W / 4, H / 4); finally, each frame of the selected video frame corresponds to , and The multi-layer perceptron (MLP) is input to obtain the global features of each scale. The global features of each scale can be expressed as X1', X2', and X3', where the size of X1' is (K, N), the size of X2' is (K, N / 2), and the size of X3' is (K, N / 4).

[0113] Through the above method, the global features of each scale can be extracted to prepare data for subsequent accurate spillage judgment.

[0114] S3, concatenates the vehicle target features with the global features of each scale to obtain the fusion features of each scale.

[0115] In the embodiment of the present application, in order to make the determination model pay more attention to the correlation between the features of the scattered objects and the features of the vehicle target, after obtaining the features of the vehicle target and X1', X2', and X3', the image processing device can splice the features of the vehicle target with X1', X2', and X3' to obtain fusion features of each scale. For example, the fusion features of each scale can be represented as a first fusion feature Y1, a second fusion feature Y2, and a third fusion feature Y3, where the size of Y1 is (M, W v ×H v +4+KN), Y2 size is (M, W v ×H v +4+KN / 2), Y3 size is (M, W v ×H v +4+KN / 4).

[0116] Through the above-mentioned method, it is possible to better focus on the correlation between the characteristics of the vehicle causing the scattering and the characteristics of the scattered objects, thereby improving the accuracy of subsequent scattering judgments.

[0117] S4, based on the feature interaction results of the fusion features of each scale and the features of the scattered objects, determine the vehicle target to which the scattered objects belong.

[0118] In the embodiment of the present application, according to the feature interaction and results of the fusion features of each scale and the features of the scattered objects, determining the vehicle target to which the scattered objects belong includes the following steps 1 and 2:

[0119] Step 1: Perform feature interaction between the fusion features of each scale and the features of the spilled objects to obtain the interaction features of each scale, and then splice the interaction features of each scale to obtain the target fusion features;

[0120] Step 2: Input the target fusion features into the fully connected layer and the output layer to obtain the predicted value of each vehicle target to which the spilled objects belong, and based on the predicted value, determine the vehicle target to which the spilled objects belong.

[0121] In step 1, after obtaining the fusion features of each scale (the first fusion feature Y1, the second fusion feature Y2 and the third fusion feature Y3), the image processing device can firstly combine Y1, Y2, Y3 and the spilled object feature F p Input MLP to get the corresponding value of Y1 , Y2 corresponds to , Y3 corresponds to and F p Corresponding ; then see the attached Figure 4 As shown, , , and Enter the feature interaction structure for feature interaction.

[0122] Will , , and The process of inputting feature interaction structure for processing is as follows:

[0123] (1) Calculation , , and The first fusion feature after feature interaction , the second fusion feature and the third fusion feature :

[0124] ;

[0125] ;

[0126] ;

[0127] Among them, λ1 is the adjustment parameter; is the sigmoid function;

[0128] (2) Based on , and , determine the characteristics of the scattered objects after interaction :

[0129] ;

[0130] ;

[0131] Among them, multi-head attention means multi-head attention; ×2 means upsampling by 2 times; ×4 means upsampling by 4 times;

[0132] (3) Calculation , , and Interaction features at each scale after feature interaction , , :

[0133] ;

[0134] ;

[0135] ;

[0136] Among them, λ2 is the adjustment parameter; is the sigmoid function.

[0137] Similarly, based on , , It is also possible to calculate the features of the spilled objects after feature interaction :

[0138] ;

[0139] ;

[0140] Among them, / 4 means downsampling by 4 times; / 2 means downsampling by 2 times.

[0141] Image acquisition equipment obtains interactive features at various scales , and After that, you can , and Perform splicing (multi-scale feature merging) to obtain the target fusion feature Y:

[0142]

[0143] By interacting the features of the spilled objects with the fusion features of each scale, we can better focus on the correlation between the features of the vehicle target where the spillage occurs and the features of the spilled objects, thereby improving the accuracy of spillage judgment.

[0144] In step 2, as shown in the attached Figure 3 As shown, after obtaining the target fusion features, the image processing device can input the target fusion features into the fully connected layer (fc layer) and the output layer (softmax layer) to obtain the predicted value of each vehicle target to which the spilled objects belong, and finally determine the vehicle target to which the spilled objects belong based on the predicted values. For example, the vehicle target corresponding to the maximum predicted value is determined as the vehicle target to which the spilled objects belong.

[0145] In this application, in order to improve the accuracy of spillage determination, the image processing device can also determine the loss value L of the spilled object belonging to the vehicle target through the following loss function when training the determination model:

[0146] ;

[0147] ;

[0148] Among them, L focal is the focal loss function; i It is used to identify whether the ith vehicle has scattered. If the ith vehicle has scattered, then ɑ i =1, if the ith vehicle does not scatter, then i =0;d i 2 represents the Euclidean distance between the i-th vehicle and the scattered object; d + 2 represents the Euclidean distance between the scattered objects and the target vehicle where the scattering occurs; m is the minimum distance between the target vehicle where the scattering occurs and the vehicle where no scattering occurs.

[0149] In summary, the method provided in the present application can better focus on the correlation between the characteristics of the vehicle target where the spillage occurs and the characteristics of the spilled objects by interacting the characteristics of the spilled objects with the fusion features of each scale; by converting the judgment problem of the scattered objects and the vehicle target where the spillage occurs into a target matching problem using the feature interaction structure, and by outputting the probability of spillage for each vehicle target in the key video frame through the powerful learning ability of the model, without the need to manually design complex judgment logic, the network automatically outputs the vehicle target where the spillage occurs, thereby improving the accuracy and rate of spillage judgment.

[0150] Based on the method provided in the above embodiment, the embodiment of the present application also provides a spillage determination device, such as Figure 5 The figure is a schematic diagram of the structure of a spillage determination device in an embodiment of the present application, the device comprising:

[0151] The data acquisition module 501 is used to acquire the key video frame where the spilled object is detected from the monitoring video screen, and extract the vehicle target features and the spilled object features in the key video frame;

[0152] Selecting K frames of candidate video frames adjacent to the key video frame, inputting the features of each scale extracted from the K frames of candidate video frames into a multi-layer perceptron to obtain global features of each scale;

[0153] A data splicing module 502 is used to splice the vehicle target feature with the global features of each scale to obtain fusion features of each scale;

[0154] The feature interaction module 503 is used to determine the vehicle target to which the scattered object belongs based on the feature interaction results of the fusion features of each scale and the features of the scattered object.

[0155] In a possible implementation, the feature interaction module 503 is specifically used to:

[0156] Performing feature interaction on the fusion features of each scale and the features of the spilled object to obtain interaction features of each scale, and splicing the interaction features of each scale to obtain target fusion features;

[0157] The target fusion features are input into the fully connected layer and the output layer to obtain the probability value of the scattered objects belonging to each vehicle target, and based on the probability value, the vehicle target to which the scattered objects belong is determined.

[0158] In a possible implementation, the feature interaction module 503 is specifically used to:

[0159] The Y1, Y2, Y3 and the feature F of the thrown object p Input multi-layer perceptron, and get , , and ;

[0160] Calculate the , , With the The first fusion feature after feature interaction , the second fusion feature and the third fusion feature :

[0161] ;

[0162] ;

[0163] ;

[0164] Among them, λ1 is the adjustment parameter; is the sigmoid function;

[0165] Based on the , and stated , determine the characteristics of the scattered objects after interaction :

[0166] ;

[0167] ;

[0168] Among them, multi-head attention means multi-head attention; ×2 means upsampling by 2 times; ×4 means upsampling by 4 times;

[0169] Calculate the , , With the The interactive features of each scale after feature interaction , , :

[0170] ;

[0171] ;

[0172] ;

[0173] Among them, λ2 is the adjustment parameter; is the sigmoid function.

[0174] In a possible implementation, the data acquisition module 501 is specifically used for:

[0175] Extracting, from the K frames of video frames to be selected, a first scale feature, a second scale feature, and a third scale feature corresponding to each frame of the video frames to be selected;

[0176] The first scale feature, the second scale feature, and the third scale feature corresponding to each frame of the selected video are input into the multi-layer perceptron to obtain the global features of each scale.

[0177] In a possible implementation, the data acquisition module 501 is specifically used for:

[0178] Determine each vehicle target in the key video frame, and extract the vehicle local features and vehicle position features of each vehicle target;

[0179] Tracking each vehicle target based on a preset target tracking algorithm to determine a tracking identifier for each vehicle target;

[0180] According to the tracking identifier, the vehicle local features and vehicle position features of each of the vehicle targets are combined to obtain the vehicle target features.

[0181] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application, and the electronic device can realize the function of the aforementioned spillage determination device, referring to Figure 6 , the electronic device comprises:

[0182] At least one processor 601, and a memory 602 connected to the at least one processor 601. The specific connection medium between the processor 601 and the memory 602 is not limited in the embodiment of the present application. Figure 6 In the example, the processor 601 and the memory 602 are connected via a bus 600. The bus 600 is Figure 6 The connection between other components is shown by bold lines, and is not intended to be limiting. The bus 600 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 601 can also be called a controller, and there is no limitation on the name.

[0183] In the embodiment of the present application, the memory 602 stores instructions that can be executed by at least one processor 601. The at least one processor 601 can execute the above-mentioned spillage determination method by executing the instructions stored in the memory 602. The processor 601 can implement Figure 5 The functions of each module in the device shown.

[0184] Among them, the processor 601 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 602 and calling the data stored in the memory 602, the various functions of the device and processing data, the device can be monitored as a whole.

[0185] In one possible design, the processor 601 may include one or more processing units, and the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 601. In some embodiments, the processor 601 and the memory 602 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0186] Processor 601 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method for determining the spillage disclosed in the embodiments of the present application can be directly embodied as a hardware processor to be executed, or a combination of hardware and software modules in the processor can be executed.

[0187] The memory 602 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 602 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 602 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 602 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0188] By programming the processor 601, the code corresponding to the spillage determination method described in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 1 The steps of the method for determining the spillage in the embodiment shown are as follows: How to design and program the processor 601 is a technique known to those skilled in the art and will not be described in detail here.

[0189] Based on the same inventive concept, an embodiment of the present application further provides a storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the spillage determination method discussed above.

[0190] In some possible implementations, various aspects of the spillage determination method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on an apparatus, the program code is used to enable the control device to execute the steps of the spillage determination method according to various exemplary embodiments of the present application described above in this specification.

[0191] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0192] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0193] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0194] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0195] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for determining spillage, characterized in that: include: From the monitoring video screen, a key video frame in which the spilled object is detected is obtained, and vehicle target features and spilled object features in the key video frame are extracted; Selecting K frames of candidate video frames adjacent to the key video frame, and inputting the features of each scale extracted from the K frames of candidate video frames into a multi-layer perceptron to obtain global features of each scale; Splicing the vehicle target features with the global features of each scale to obtain fused features of each scale; Performing feature interaction on the fusion features of each scale and the features of the spilled object to obtain interaction features of each scale, and splicing the interaction features of each scale to obtain target fusion features; The target fusion features are input into the fully connected layer and the output layer to obtain the predicted value of each vehicle target to which the scattered objects belong, and based on the predicted value, the vehicle target to which the scattered objects belong is determined.

2. The method according to claim 1, characterized in that If the fusion features of each scale include a first fusion feature Y1, a second fusion feature Y2, and a third fusion feature Y3, then the fusion features of each scale are subjected to feature interaction with the feature of the spilled object to obtain interaction features of each scale, including: The Y1, Y2, Y3 and the feature F of the thrown object p Input multi-layer perceptron, and get , , and ; Calculate the , , With the The first fusion feature after feature interaction , the second fusion feature and the third fusion feature : ； ； ； Among them, λ1 is the adjustment parameter; is the sigmoid function; Based on the , and stated , determine the characteristics of the scattered objects after interaction : ； ； Among them, multi-head attention means multi-head attention; ×2 means upsampling by 2 times; ×4 means upsampling by 4 times; Calculate the , , With the The interactive features of each scale after feature interaction , , : ； ； ； Among them, λ2 is the adjustment parameter; is the sigmoid function.

3. The method according to claim 1, characterized in that The loss value of the spilled object belonging to the vehicle target is determined by the following loss function: ； ； Among them, L focal is the focal loss function; i It is used to identify whether the ith vehicle has scattered. If the ith vehicle has scattered, then ɑ i =1, if the ith vehicle does not scatter, then i =0;d i 2 represents the Euclidean distance between the i-th vehicle and the scattered object; d + 2 represents the Euclidean distance between the scattered objects and the target vehicle where the scattering occurs; m is the minimum distance between the target vehicle where the scattering occurs and the vehicle where no scattering occurs.

4. The method according to claim 1, characterized in that The step of selecting K frames of candidate video frames adjacent to the key video frame, and inputting the features of each scale extracted from the K frames of candidate video frames into a multi-layer perceptron to obtain global features of each scale includes: Extracting, from the K frames of video frames to be selected, a first scale feature, a second scale feature, and a third scale feature corresponding to each frame of the video frames to be selected; The first scale feature, the second scale feature, and the third scale feature corresponding to each frame of the selected video are input into the multi-layer perceptron to obtain the global features of each scale.

5. The method according to claim 1, characterized in that The extracting of the vehicle target features in the key video frame includes: Determine each vehicle target in the key video frame, and extract the vehicle local features and vehicle position features of each vehicle target; Tracking each vehicle target based on a preset target tracking algorithm to determine a tracking identifier for each vehicle target; According to the tracking identifier, the vehicle local features and vehicle position features of each of the vehicle targets are combined to obtain the vehicle target features.

6. A spillage determination device, characterized in that: include: A data acquisition module is used to acquire key video frames where spilled objects are detected from the monitoring video screen, and to extract vehicle target features and spilled object features in the key video frames; Selecting K frames of candidate video frames adjacent to the key video frame, inputting the features of each scale extracted from the K frames of candidate video frames into a multi-layer perceptron to obtain global features of each scale; A data splicing module, used for splicing the vehicle target features with the global features of each scale to obtain fusion features of each scale; A feature interaction module is used to perform feature interaction on the fusion features of each scale and the features of the spilled object to obtain interaction features of each scale, and to splice the interaction features of each scale to obtain target fusion features; The target fusion features are input into the fully connected layer and the output layer to obtain the probability value of the scattered objects belonging to each vehicle target, and based on the probability value, the vehicle target to which the scattered objects belong is determined.

7. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to implement the method steps of any one of claims 1 to 5 when executing the computer program stored in the memory.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 5 are implemented.

Images