Alarm image determination method and device
Patent Information
- Application Number
- CN202411026558.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art is costly and has low detection accuracy when detecting whether a motorcycle rider wears a safety helmet, which is prone to missed detection or missed reports, especially when distinguishing between riders and pedestrians on the roadside.
The multimodal model is used to combine the sliding window mechanism to obtain and process the pedestrian and vehicle area information in the image, bind the target pedestrian area and vehicle identity identification, and determine the alarm image through confidence and area to reduce false detection and misreport.
It improves the accuracy and efficiency of motorcycle riders' inspections wearing safety helmets, reduces calculation complexity and resource consumption, and enhances the reliability of riders' wearing conditions.
Smart Images

Figure CN119027644A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for determining a warning image. Background Art
[0002] Safety equipment detection for dangerous behaviors is of great significance in many fields. It is not only related to personal safety and health, but also involves social management and scientific research. First, in terms of personal safety and health: in specific occasions, such as schools, factories, hospitals, etc., by detecting whether personnel are wearing personal protective equipment such as safety helmets and protective glasses correctly, accidents can be effectively prevented and public safety can be guaranteed. In public health events such as epidemic prevention and control, by detecting whether people are wearing masks, it is helpful to strengthen public health management and reduce the spread of diseases. In terms of industrial production and quality control: by detecting whether workers are wearing necessary protective equipment, work-related accidents can be reduced and production safety can be improved.
[0003] At present, motorcycles are a popular means of transportation. There are significant characteristics in motorcycle traffic accidents. 1) The safety factor of motorcycles is often relatively low. They lack the protection of a car and seat belts. Once an accident occurs, it poses a greater threat to the safety of the rider. 2) Motorcycles only have two wheels. Compared with four-wheeled cars, motorcycles are more difficult to control and require higher driving skills from the rider. 3) Motorcycles are easily affected by external environments such as bad weather and road conditions. They are more likely to fall and cause danger on muddy roads in rural streets. 4) Some motorcycle riders drive too fast and do not wear helmets properly. They carry passengers illegally, which often poses a greater threat to life safety when a traffic accident occurs. 5) The mortality rate in motorcycle accidents is about three times that of cars. The casualty rate in motorcycle accidents is as high as 80%, while that in car accidents remains at around 20%.
[0004] In order to reduce traffic accidents involving electric vehicles or motorcycles, riders need to wear safety helmets. Currently, most methods for detecting safety helmets are to use hardware devices, using sensors installed on the helmet to take photos of the rider's head and transmit them to a microprocessor for background analysis, and then feed back the analysis results to the on-board device. If it is determined that the rider is wearing a safety helmet, the power supply system is started, and the vehicle can drive normally. Alternatively, a simple target detection algorithm is used to directly determine whether the rider is wearing a helmet. The first method is costly and cannot control whether the rider must have purchased this equipment and used it with a motorcycle. The second method has low detection accuracy for whether the rider is wearing a helmet, and is prone to misdetecting pedestrians on the roadside, resulting in alarms, and is prone to missed or false alarms. Summary of the invention
[0005] In view of the above problems, embodiments of the present application provide a method and device for determining a warning image that overcomes the above problems or at least partially solves the above problems.
[0006] In a first aspect, an embodiment of the present application provides a method for determining a warning image, the method comprising:
[0007] Acquire an i-th frame of the first image, wherein the i-th frame of the first image is marked with a pedestrian area and a vehicle area, where i is an integer greater than 0;
[0008] Inputting the first image of the i-th frame into the multimodal model, obtaining the region information of the pedestrian region in the i-th frame of the first image where the pedestrian does not wear the safety equipment, the region information including: the confidence, the angular coordinates and the area of the pedestrian region;
[0009] In the case where there is a target pedestrian region satisfying the first condition in the i-th frame of the first image, binding the target pedestrian region with a vehicle identity identifier of the vehicle region;
[0010] In a case where the number of second images in the target sliding window is equal to a preset number and the confidence of the target pedestrian area bound to the target vehicle identity in the third image of the target sliding window is less than the confidence of the target pedestrian area bound to the target vehicle identity in the i-th frame first image, the third image is replaced with the i-th frame first image, the third image being the second image with the smallest confidence of the target pedestrian area bound to the target vehicle identity in the target sliding window, and the target vehicle identity is the vehicle identity corresponding to any vehicle area in the i-th frame first image;
[0011] Let i=i+1, and repeat the above content until the first image of the i-th frame does not contain the vehicle area marked with the target vehicle identity, and determine the warning image corresponding to the target vehicle identity according to the area and confidence of the target pedestrian area bound to the target vehicle identity in each second image in the target sliding window.
[0012] Optionally, acquiring the i-th frame of the first image includes:
[0013] Get the fourth image of the i-th frame;
[0014] Vehicles and pedestrians are detected on the i-th frame fourth image, and the detected pedestrian area and vehicle area are marked in the i-th frame fourth image to obtain the i-th frame first image.
[0015] Optionally, the detecting vehicles and pedestrians on the i-th frame fourth image and marking the detected pedestrian area and vehicle area into the i-th frame fourth image to obtain the i-th frame first image includes:
[0016] Perform vehicle and pedestrian detection on the fourth image of the i-th frame by using the first model to obtain a first pedestrian region and a first vehicle region corresponding to the first model;
[0017] Perform vehicle and pedestrian detection on the fourth image of the i-th frame by using the second model to obtain a second pedestrian region and a second vehicle region corresponding to the second model;
[0018] Performing weighted average calculation on the first pedestrian region and the second pedestrian region to obtain a third pedestrian region;
[0019] Performing weighted average calculation on the first vehicle area and the second vehicle area to obtain a third vehicle area;
[0020] The third pedestrian region and the third vehicle region are marked in the i-th frame fourth image to obtain the i-th frame first image.
[0021] Optionally, when there is a target pedestrian region satisfying a first condition in the i-th frame of the first image, binding the target pedestrian region with a vehicle identity of a vehicle region includes:
[0022] A pedestrian region where the pedestrian does not wear safety equipment and the confidence is greater than a preset confidence and the center point of the angular coordinate is located in the area of interest of the i-th frame of the first image is determined as a target pedestrian region that meets the first condition;
[0023] Acquire a vehicle identity of each vehicle region in the i-th frame of the first image by a target tracking method;
[0024] The target pedestrian area is bound to the vehicle identity identifier of the vehicle area.
[0025] Optionally, binding the target pedestrian area with the vehicle identity of the vehicle area includes:
[0026] In the case where there is a target pedestrian region in the i-th frame first image, performing an intersection-and-union ratio calculation on the target pedestrian region in the i-th frame first image and each vehicle region to obtain an intersection-and-union ratio result;
[0027] The target pedestrian region whose intersection-over-combination result is greater than the first result is bound to the vehicle identity identifier of the vehicle region.
[0028] Optionally, when there is a target pedestrian region satisfying the first condition in the i-th frame of the first image, after binding the target pedestrian region with a vehicle identity identifier of the vehicle region, the method further includes:
[0029] When the number of second images in the target sliding window is less than the preset number, the i-th frame first image is added to the target sliding window, and the i-th frame first image serves as one of the second images in the target sliding window.
[0030] Optionally, determining the warning image corresponding to the target vehicle identity according to the area and confidence of the target pedestrian region bound to the target vehicle identity in each second image in the target sliding window includes:
[0031] Acquire a first number of target pedestrian regions among the target pedestrian regions bound to the target vehicle identity in each second image in the target sliding window in descending order of the area of the target pedestrian regions;
[0032] The second image corresponding to the target pedestrian area with the highest confidence among the first number of target pedestrian areas is determined as the warning image corresponding to the target vehicle identity.
[0033] In a second aspect, an embodiment of the present application further provides a device for determining a warning image, the device comprising:
[0034] An acquisition module, used for acquiring an i-th frame of a first image, wherein the i-th frame of the first image is marked with a pedestrian area and a vehicle area, and i is an integer greater than 0;
[0035] A processing module, configured to input the i-th frame first image into a multimodal model to obtain region information of a pedestrian region in the i-th frame first image where the pedestrian is not wearing safety equipment, wherein the region information includes: a confidence level, an angular coordinate, and an area of the pedestrian region;
[0036] A binding module, configured to bind the target pedestrian region to a vehicle identity identifier of a vehicle region when there is a target pedestrian region satisfying a first condition in the i-th frame of the first image;
[0037] A replacement module, configured to replace the third image with the i-th frame first image, when the number of second images in the target sliding window is equal to a preset number and the confidence of the target pedestrian area bound to the target vehicle identity in the third image of the target sliding window is less than the confidence of the target pedestrian area bound to the target vehicle identity in the i-th frame first image, wherein the third image is the second image with the smallest confidence of the target pedestrian area bound to the target vehicle identity in the target sliding window, and the target vehicle identity is a vehicle identity corresponding to any vehicle area in the i-th frame first image;
[0038] A determination module is used to set i=i+1, loop the above content until the vehicle area marked with the target vehicle identity is not included in the first image of the i-th frame, and determine the warning image corresponding to the target vehicle identity according to the area and confidence of the target pedestrian area bound to the target vehicle identity in each second image in the target sliding window.
[0039] Optionally, the acquisition module is specifically used to:
[0040] Get the fourth image of the i-th frame;
[0041] Vehicles and pedestrians are detected on the i-th frame fourth image, and the detected pedestrian area and vehicle area are marked in the i-th frame fourth image to obtain the i-th frame first image.
[0042] Optionally, when the acquisition module detects vehicles and pedestrians on the i-th frame fourth image and marks the detected pedestrian area and vehicle area into the i-th frame fourth image to obtain the i-th frame first image, it is specifically used to:
[0043] Perform vehicle and pedestrian detection on the fourth image of the i-th frame by using the first model to obtain a first pedestrian region and a first vehicle region corresponding to the first model;
[0044] Perform vehicle and pedestrian detection on the fourth image of the i-th frame by using the second model to obtain a second pedestrian region and a second vehicle region corresponding to the second model;
[0045] Performing weighted average calculation on the first pedestrian region and the second pedestrian region to obtain a third pedestrian region;
[0046] Performing weighted average calculation on the first vehicle area and the second vehicle area to obtain a third vehicle area;
[0047] The third pedestrian region and the third vehicle region are marked in the i-th frame fourth image to obtain the i-th frame first image.
[0048] Optionally, the binding module is specifically used to:
[0049] A pedestrian region where the pedestrian does not wear safety equipment and the confidence is greater than a preset confidence and the center point of the angular coordinate is located in the area of interest of the i-th frame of the first image is determined as a target pedestrian region that meets the first condition;
[0050] Acquire a vehicle identity of each vehicle region in the i-th frame of the first image by a target tracking method;
[0051] The target pedestrian area is bound to the vehicle identity identifier of the vehicle area.
[0052] Optionally, when binding the target pedestrian area with the vehicle identity of the vehicle area, the binding module is specifically used to:
[0053] In the case where there is a target pedestrian region in the i-th frame first image, performing an intersection-and-union ratio calculation on the target pedestrian region in the i-th frame first image and each vehicle region to obtain an intersection-and-union ratio result;
[0054] The target pedestrian region whose intersection-over-combination result is greater than the first result is bound to the vehicle identity identifier of the vehicle region.
[0055] Optionally, the device further comprises:
[0056] An adding module is used to add the i-th frame first image to the target sliding window when the number of second images in the target sliding window is less than the preset number, and the i-th frame first image serves as one of the second images in the target sliding window.
[0057] Optionally, when the determination module determines the warning image corresponding to the target vehicle identity according to the area and confidence of the target pedestrian area bound to the target vehicle identity in each second image in the target sliding window, it is specifically used to:
[0058] Acquire a first number of target pedestrian regions among the target pedestrian regions bound to the target vehicle identity in each second image in the target sliding window in descending order of the area of the target pedestrian regions;
[0059] The second image corresponding to the target pedestrian area with the highest confidence among the first number of target pedestrian areas is determined as the warning image corresponding to the target vehicle identity.
[0060] In a third aspect, an embodiment of the present application further provides an electronic device, including a memory, a transceiver, and a processor:
[0061] A memory for storing a computer program; a transceiver for sending and receiving data under the control of a processor; and a processor for reading the computer program in the memory and executing the method as described in the first aspect above.
[0062] In a fourth aspect, an embodiment of the present application further provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, and the computer program is used to enable the processor to execute the method described in the first aspect above.
[0063] In the above embodiment of the present application, the first image of the i-th frame marked with pedestrian areas and vehicle areas is input into the multimodal model to obtain the confidence, angular coordinates and area of the pedestrian area in the i-th frame of the first image where the pedestrian is not wearing safety equipment. The multimodal large model replaces the traditional classification network method. Compared with the traditional method, the multimodal model has image understanding capabilities, making the identified pedestrian area and vehicle area more accurate. In addition, when there is a target pedestrian area that meets the first condition in the i-th frame of the first image, the target pedestrian area is bound to the vehicle identity of the vehicle area, thereby reducing the matching calculation amount of pedestrians and vehicles. Furthermore, when the number of second images in the target sliding window is equal to a preset number and the confidence of the target pedestrian area bound to the target vehicle identity in the third image of the target sliding window is less than the confidence of the target pedestrian area bound to the target vehicle identity in the first image of the i-th frame, the third image is replaced with the i-th frame first image, the third image is the second image with the smallest confidence of the target pedestrian area bound to the target vehicle identity in the target sliding window, the target vehicle identity is the vehicle identity corresponding to any vehicle area in the i-th frame first image, let i=i+1, and repeat the above content until the confidence of the target pedestrian area bound to the target vehicle identity in the i-th frame first image is reached. The image does not contain a vehicle area marked with the target vehicle identity, and the warning image corresponding to the target vehicle identity is determined according to the area and confidence of the target pedestrian area bound to the target vehicle identity in each second image in the target sliding window. That is, by introducing a sliding window mechanism, it is only necessary to sort the second images in the sliding window, and the number of images in the sliding window is always kept at most a preset number, thereby reducing the cache resource amount of the image; and the images in the sliding window are continuously updated to obtain a preset number of images with high confidence, and then the optimal warning image is determined by the area and confidence, thereby improving the determination efficiency of the warning image and reducing the overall calculation complexity. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0065] Figure 1 One of the flowcharts of the method for determining the warning image provided in the embodiment of the present application;
[0066] Figure 2 A schematic diagram of the association of prompts, correlation values, and sample images provided in the embodiments of the present application;
[0067] Figure 3 The second flowchart of the method for determining the warning image provided in the embodiment of the present application;
[0068] Figure 4 An example diagram of feature fusion of two models provided in an embodiment of the present application;
[0069] Figure 5 A schematic diagram of a region of interest provided in an embodiment of the present application;
[0070] Figure 6 A schematic diagram of the processing flow of the target sliding window provided in an embodiment of the present application;
[0071] Figure 7 A structural block diagram of a device for determining an alarm image provided in an embodiment of the present application;
[0072] Figure 8 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0073] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0074] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0075] The following describes in detail the method for determining the warning image provided by the embodiment of the present application through specific embodiments and application scenarios in conjunction with the accompanying drawings.
[0076] Specifically, the embodiment of the present application provides a method for determining a warning image, such as Figure 1 As shown, the following steps may be specifically included:
[0077] Step 101 : obtaining an i-th frame of a first image, wherein a pedestrian area and a vehicle area are marked in the i-th frame of the first image, and i is an integer greater than 0.
[0078] Specifically, a video capture card, a network camera or other device is used to capture highway video in real time, and the captured video stream is sent to the server using protocols such as Real-Time Stream Protocol (RTSP), Real-Time Message Protocol (RTMP), HTTP-based streaming media transmission protocol (HLS), etc. After the server obtains the video stream, it needs to encode it, convert it into a standard video format, and then extract frames to obtain multiple frame images in a standard image format, and mark the pedestrian area and vehicle area in the i-th frame image, that is, the first image of the i-th frame is obtained.
[0079] The pedestrian area refers to the pedestrian rectangular frame of the pedestrian wearing safety equipment and the pedestrian matrix frame of the pedestrian not wearing safety equipment identified in the frame image. The vehicle area refers to the vehicle rectangular frame identified in the frame image.
[0080] Step 102: input the i-th frame first image into a multimodal model to obtain the area information of the pedestrian area in the i-th frame first image where the pedestrian is not wearing safety equipment, wherein the area information includes: the confidence, angular coordinates and area of the pedestrian area.
[0081] Specifically, the first image of the i-th frame is input into the trained multimodal model to obtain the confidence of the pedestrian area in the i-th frame where the pedestrian is not wearing safety equipment, the angular coordinates of the pedestrian area, and the area of the pedestrian area. The training process of the multimodal model is described below:
[0082] The server obtains the sample video stream, encodes it, converts it into a standard video format, and then extracts frames to obtain multiple frame images in a standard image format. Multiple frame images form a sample data set. The frame images in the sample data set are classified into three categories: motorcycle (Motorbike), no helmet (Nohelmet), and helmet (DoHelmet). No helmet (Nohelmet) and helmet (DoHelmet) are merged into one category with motorcycle (Motorbike) to construct two categories of detection data sets.
[0083] The sample images of not wearing a helmet (Nohelmet) and wearing a helmet (DoHelmet) are cropped after a rectangular expansion operation of a certain ratio to obtain multiple sample images. The cropped sample images of wearing a helmet (DoHelmet) are placed in folder 1 as positive samples, and the cropped samples of not wearing a helmet (Nohelmet) are placed in folder 2 as negative samples.
[0084] For the training set, add sentences describing the task of wearing safety equipment in the sample image as prompts, and increase the generalization ability of the multimodal model for different similar language prompts. For example, the prompt is a picture of an person wearing a helmet. Traverse the relationship between each prompt and each sample image and make a one-to-one correspondence. The relationship between the prompt and the sample image can be represented by a numerical value. If the prompt is related to the sample image, it is 1, and if it is not related, it is 0. Blank or tab can be used as a separator between the prompt, correlation value, and sample image and written into the folder blip_train.txt as training data. When the separator is blank, such as Figure 2 As shown in the figure, Apicture of an person wearing a helmet is the prompt, 1.img represents sample image 1, 2.img represents sample image 2, 3.img represents sample image 3, and 4.img represents sample image 4. The test set and the validation set are the same as above. The multimodal model is trained with the training set to obtain the trained multimodal model, and the trained multimodal model is tested and verified with the test set and the validation set. Figure 2 The prompts in the figure are only examples. You can modify the prompts as needed, or add or delete prompts.
[0085] The multimodal model may be a BLIP large model. In this embodiment, the image-text matching task ITM of the BLIP large model may be used.
[0086] It should be noted that Prompt, as an emerging interaction method, has demonstrated its unique value in multiple AI application scenarios. The core idea of Prompt technology is to guide the pre-trained language model to complete specific tasks by designing specific language prompts (i.e. Prompt) without the need for a large amount of task-specific training on the model.
[0087] The Bootstrapping Language-Image Pre-training (BLIP) framework aims to unify visual language understanding and generation tasks. BLIP is an encoder-decoder hybrid multimodal network, which is an important cornerstone in the field of multimodal large models. It mainly consists of three parts, corresponding to three tasks: image-text comparison task ITC, image-text matching task ITM, and language model task LM. Among them, ITM uses an image-based text encoder image-grounded text encoder, and its specific structure is also a language transformation model (Bidirectional Encoder Representations from Transformers, BERT). It uses cross-attention to fuse image features and text features, which is equivalent to a multimodal encoder encoder. The encoder's token token is used to represent the features after multimodal fusion. Finally, a multilayer perceptron (MLP) classification head is added after the feature to distinguish whether the image and text match.
[0088] Step 103 : when there is a target pedestrian region satisfying the first condition in the i-th frame of the first image, the target pedestrian region is bound to a vehicle identity identifier of the vehicle region.
[0089] Specifically, it is determined whether each pedestrian region in the first image of the i-th frame satisfies the first condition. If all pedestrian regions in the first image of the i-th frame do not satisfy the first condition, the process is terminated. If at least one pedestrian region in the first image of the i-th frame satisfies the first condition, the pedestrian region satisfying the first condition is used as the target pedestrian region. Since each vehicle region has a corresponding vehicle identity, each target pedestrian region is bound to the vehicle identity of the corresponding vehicle region, thereby establishing an association relationship between the target pedestrian region and the vehicle identity.
[0090] Step 104, when the number of second images in the target sliding window is equal to a preset number and the confidence of the target pedestrian area bound to the target vehicle identity in the third image of the target sliding window is less than the confidence of the target pedestrian area bound to the target vehicle identity in the i-th frame first image, the third image is replaced with the i-th frame first image, the third image is the second image with the smallest confidence of the target pedestrian area bound to the target vehicle identity in the target sliding window, and the target vehicle identity is the vehicle identity corresponding to any vehicle area in the i-th frame first image.
[0091] Specifically, one vehicle identification corresponds to one sliding window. For the target sliding window of the target vehicle identification, the number of second images in the target sliding window is set to a preset number, that is, the target sliding window can store a preset number of second images at most. After determining the target pedestrian area bound to the target vehicle identification, it is determined whether the number of second images in the target sliding window is equal to the preset number. If the number of second images in the target sliding window is equal to the preset number, it means that the target sliding window can no longer store additional images, and it is necessary to obtain the second image (i.e., the third image) corresponding to the target pedestrian area with the lowest confidence in the current target sliding window. If at least one target pedestrian area in the first image of the i-th frame is larger than the second image corresponding to the target pedestrian area with the lowest execution degree, the third image is replaced with the first image of the i-th frame.
[0092] It should be noted that the sliding window is a commonly used algorithmic technique for processing subsequence problems in sequence data or arrays. It defines a window of fixed size, slides on the sequence or array, and calculates or processes the data in the window at each position. The sliding window algorithm is often used to solve various problems such as string processing, array operations, and data stream processing. The basic idea of the sliding window algorithm is to maintain a window that slides on the sequence or array so that the data in the window can be processed at each position. The size of the window can be adjusted according to the needs of the specific problem. Each time the window is slid, calculations, judgments, or other operations can be performed based on the data in the window, and the results can be obtained according to the requirements of the problem.
[0093] Step 105, let i=i+1, loop the above content until the first image of the i-th frame does not contain the vehicle area marked with the target vehicle identity, and determine the warning image corresponding to the target vehicle identity according to the area and confidence of the target pedestrian area bound to the target vehicle identity in each second image in the target sliding window.
[0094] Specifically, let i=i+1 in step 101, and loop step 101 to step 104. If in step 101, the vehicle identification corresponding to the vehicle area marked in the first image of the i-th frame does not include the target vehicle identification, it means that the vehicle with the target vehicle identification has driven out of the shooting range, and the loop process is terminated. According to the area and confidence of the target pedestrian area bound to the target vehicle identification in each second image in the current target sliding window, the warning image corresponding to the target vehicle identification is determined. The use process of the above-mentioned target sliding window is described below through a specific embodiment:
[0095] For example, the target sliding window is set to S, and the preset number is 30. If the first image of the i-th frame contains a target pedestrian area that meets the first condition, the number of second images in the target sliding window S is determined. If the number is 30, the minimum confidence score Score_R among the 30 second images of the target sliding window needs to be determined. min The confidence score Score_R of the target pedestrian area that meets the first condition in the first image of the i-th frame 31 If the confidence score Score_R of the target pedestrian area in the first image of the i-th frame satisfies the first condition 31 Larger, that is, Score_R 31 Greater than Score_R min , then the confidence Score_R min The second image (ie, the third image) is replaced by the first image of the i-th frame.
[0096] In the above embodiment of the present application, the first image of the i-th frame marked with pedestrian areas and vehicle areas is input into the multimodal model to obtain the confidence, angular coordinates and area of the pedestrian area in the i-th frame of the first image where the pedestrian is not wearing safety equipment. The multimodal large model replaces the traditional classification network method. Compared with the traditional method, the multimodal model has image understanding capabilities, making the identified pedestrian area and vehicle area more accurate. In addition, when there is a target pedestrian area that meets the first condition in the i-th frame of the first image, the target pedestrian area is bound to the vehicle identity of the vehicle area, thereby reducing the matching calculation amount of pedestrians and vehicles. Furthermore, when the number of second images in the target sliding window is equal to a preset number and the confidence of the target pedestrian area bound to the target vehicle identity in the third image of the target sliding window is less than the confidence of the target pedestrian area bound to the target vehicle identity in the first image of the i-th frame, the third image is replaced with the i-th frame first image, the third image is the second image with the smallest confidence of the target pedestrian area bound to the target vehicle identity in the target sliding window, the target vehicle identity is the vehicle identity corresponding to any vehicle area in the i-th frame first image, let i=i+1, and repeat the above content until the confidence of the target pedestrian area bound to the target vehicle identity in the i-th frame first image is reached. The image does not contain a vehicle area marked with the target vehicle identity, and the warning image corresponding to the target vehicle identity is determined according to the area and confidence of the target pedestrian area bound to the target vehicle identity in each second image in the target sliding window. That is, by introducing a sliding window mechanism, it is only necessary to sort the second images in the sliding window, and the number of images in the sliding window is always kept at most a preset number, thereby reducing the cache resource amount of the image; and the images in the sliding window are continuously updated to obtain a preset number of images with high confidence, and then the optimal warning image is determined by the area and confidence, thereby improving the determination efficiency of the warning image and reducing the overall calculation complexity.
[0097] As an optional specific embodiment, the step 101 of acquiring the i-th frame of the first image includes:
[0098] Get the fourth image of the i-th frame;
[0099] Vehicles and pedestrians are detected on the i-th frame fourth image, and the detected pedestrian area and vehicle area are marked in the i-th frame fourth image to obtain the i-th frame first image.
[0100] Specifically, a video capture card, a network camera or other device is used to capture the highway video in real time, and the captured video stream is sent to the server via protocols such as RTSP, RTMP, and HLS. After the server obtains the video stream, it needs to encode it, convert it into a standard video format, and then extract frames to obtain multiple frame images in a standard image format, wherein one frame image is a fourth image, and the i-th frame image among the multiple frame images is the i-th frame fourth image.
[0101] For the i-th frame fourth image, vehicles and pedestrians are detected on the i-th frame fourth image, pedestrian regions and vehicle regions in the i-th frame fourth image are detected, and the pedestrian regions and vehicle regions are marked in the i-th frame fourth image, thereby forming the i-th frame first image with the pedestrian regions and vehicle regions marked. For each fourth image, the corresponding i-th frame first image is obtained in the same manner as the i-th frame fourth image.
[0102] Furthermore, the above-mentioned step of detecting vehicles and pedestrians on the i-th frame fourth image and marking the detected pedestrian area and vehicle area into the i-th frame fourth image to obtain the i-th frame first image specifically includes:
[0103] Perform vehicle and pedestrian detection on the fourth image of the i-th frame by using the first model to obtain a first pedestrian region and a first vehicle region corresponding to the first model;
[0104] Perform vehicle and pedestrian detection on the fourth image of the i-th frame by using the second model to obtain a second pedestrian region and a second vehicle region corresponding to the second model;
[0105] Performing weighted average calculation on the first pedestrian region and the second pedestrian region to obtain a third pedestrian region;
[0106] Performing weighted average calculation on the first vehicle area and the second vehicle area to obtain a third vehicle area;
[0107] The third pedestrian region and the third vehicle region are marked in the i-th frame fourth image to obtain the i-th frame first image.
[0108] Specifically, Figure 3As shown, the fourth image of the i-th frame is input into the trained first model, and the vehicles and pedestrians in the fourth image of the i-th frame are detected by the first model to obtain the first pedestrian area and the first vehicle area. In addition, the fourth image of the i-th frame is input into the trained second model, and the vehicles and pedestrians in the fourth image of the i-th frame are detected by the second model to obtain the second pedestrian area and the second vehicle area. The first pedestrian area output by the first model and the second pedestrian area output by the second model are weighted averaged to obtain the third pedestrian area; and the first vehicle area output by the first model and the second vehicle area output by the second model are weighted averaged to obtain the third vehicle area. The output results of the two models are feature fused and Ensembled by weighted averaging, and the final weighted average results of the two models are closer to the true value, which can reduce the deviation and variance of the model and ultimately reduce the generalization error of the model. The third pedestrian area and the third vehicle area are annotated to the fourth image of the i-th frame to obtain the first image of the i-th frame, so that the recognition of the pedestrian area and the vehicle area in the first image of the i-th frame is more accurate.
[0109] It should be noted that Ensemble is commonly explained as encountering a bottleneck in a single model. Generally, one way to improve the model's performance is to create features or to fuse models. The methods include Voting ensembles, averaging, stacking, blending, etc. Often, using these methods to fuse multiple models will make the effect better than a single model. This method is called feature fusion.
[0110] like Figure 4 As shown, the principle of the above feature fusion is explained through an example:
[0111] The first model outputs a score of 7.2 for the pedestrian area and a score of 11.4 for the vehicle area; the second model outputs a score of 7.5 for the pedestrian area and a score of 10.4 for the vehicle area. The actual score of the pedestrian area is 7.3, and the actual score of the vehicle area is 10.7. The results output by the two models are quite different. If the score of the pedestrian area of the two models is weighted averaged, the score of the pedestrian area is (7.2+7.5) / 2=7.35, which is closer to the actual score of 7.3; if the score of the vehicle area of the two models is weighted averaged, the score of the vehicle area is (11.4+10.4) / 2=10.9, which is closer to the actual score of 10.7. Therefore, the score results inferred by the two models are closer to the actual value after weighted averaging, which can reduce the deviation and variance of the model and ultimately reduce the generalization error of the model.
[0112] The training process of the first model and the second model is described below through a specific example:
[0113] The sample data set is split into a training set, a test set, and a validation set according to a certain ratio, such as 7:2:1. The first model and the second model are used for training the same batch of sample data sets, that is, the sample quantity sets are respectively passed into the first model and the second model for detecting two types of targets, pedestrians and vehicles. The pedestrian area and vehicle area identified by the two models are weighted averaged to obtain the final pedestrian area and vehicle area, and the pedestrian area is expanded and cropped according to a certain ratio and marked on the sample image. The marked sample image is used for model training to obtain the trained first model and second model.
[0114] Among them, the first model can be DETR. The DETR model is an end-to-end target detection model, and its full name is "DEtection TRansformer". The core idea of the DETR model is to use the Transformer structure for target detection, thereby realizing a direct mapping from image to detection result, without the need for a multi-stage processing flow in a traditional target detection algorithm. The DETR model directly matches the predicted bounding box with the true label through a set-to-set prediction method, calculates the loss, and then optimizes the model parameters. The advantage of this method is that it simplifies the target detection process and improves the efficiency of training and reasoning.
[0115] The second model can be YOLOv8, which is an object detection algorithm designed to provide higher accuracy and speed. It inherits the advantages of the previous generation model and makes many improvements on this basis, including a more complex network architecture, a more optimized training process, and more powerful feature extraction capabilities. These improvements work together to enable YOLOv8 to achieve unprecedented detection performance on multiple standard datasets.
[0116] As an optional specific embodiment, in step 103, when there is a target pedestrian area satisfying the first condition in the i-th frame first image, binding the target pedestrian area with the vehicle identity of the vehicle area specifically includes:
[0117] A pedestrian region where the pedestrian does not wear safety equipment and the confidence is greater than a preset confidence and the center point of the angular coordinate is located in the area of interest of the i-th frame of the first image is determined as a target pedestrian region that meets the first condition;
[0118] Acquire a vehicle identity of each vehicle region in the i-th frame of the first image by a target tracking method;
[0119] The target pedestrian area is bound to the vehicle identity identifier of the vehicle area.
[0120] Specifically, Figure 5 As shown, a preset confidence T is defined. If the confidence of the pedestrian area where the pedestrian is not wearing safety equipment in the first image of the i-th frame is less than or equal to the preset confidence T, it is determined that the cyclist (i.e., pedestrian) is wearing safety equipment, and the pedestrian area is eliminated. If the confidence of the pedestrian area where the pedestrian is not wearing safety equipment in the first image of the i-th frame is greater than the preset confidence T, it is preliminarily considered that the cyclist (i.e., pedestrian) who needs to be determined is not wearing a helmet. It is further determined whether the center point of the angular coordinates of the pedestrian area (i.e., the center point of the Box) is in the region of interest (Region Of Interest, ROI). If it does not exist in the ROI area, the pedestrian area is eliminated; if it exists in the ROI area, the pedestrian area is determined to be a target pedestrian area that meets the first condition. The setting of the ROI area improves the efficiency and accuracy of detection. Among them, the ROI area is a region of interest in a preset picture, which can be set as needed.
[0121] like Figure 3 As shown, a unique identifier is assigned to different vehicles through the SORT multi-target tracking method, that is, the vehicle identity identifier of each vehicle area in the first image of the i-th frame can be obtained through the SORT multi-target tracking method.
[0122] It should be noted that Simple Online and Realtime Tracking (SORT) is an important field in computer vision. The task of Multiple Object Tracking (MOT) is to detect multiple targets such as pedestrians and cars in the video without knowing the number of targets in advance, and assign IDs for trajectory tracking. It includes first detecting objects in a single frame, and then associating these detected results with trajectories. Now real-time tracking (SORT) is a practical MOT method. It uses the Hungarian algorithm to perform Kalman filtering in image space and frame-by-frame data association.
[0123] Furthermore, the step of binding the target pedestrian area with the vehicle identity of the vehicle area specifically includes:
[0124] In the case where there is a target pedestrian region in the i-th frame first image, performing an intersection-and-union ratio calculation on the target pedestrian region in the i-th frame first image and each vehicle region to obtain an intersection-and-union ratio result;
[0125] The target pedestrian region whose intersection-over-combination result is greater than the first result is bound to the vehicle identity identifier of the vehicle region.
[0126] Specifically, Figure 3 As shown in the figure, if the target pedestrian area that meets the first condition is named R1, the corresponding confidence is Score_R1, and the area is Area_R1, the box coordinates of R1 and the box coordinates of each vehicle area are calculated by the following formula:
[0127]
[0128] Among them, IOU represents the intersection-over-union result; A and B represent the box coordinates of R1 and the box coordinates of the vehicle area respectively. The calculation of the intersection-over-union can reduce the false positives of pedestrian recalls and increase the accuracy of the result of judging that the person is a cyclist and is not wearing a helmet.
[0129] If the intersection and union of the box coordinates of a target pedestrian area and the box coordinates of each vehicle area is greater than the first result, the target pedestrian area can be bound to the vehicle identity corresponding to the vehicle area. Among them, a vehicle identity can be bound to one or more target pedestrian areas. For example: Figure 3 As shown, if the intersection and union ratio of the box coordinates of the target pedestrian area 1 and the target pedestrian area 2 with the box coordinates of the target vehicle area are both greater than the first result, the target pedestrian area 1 and the target pedestrian area 2 are bound to the target vehicle identity corresponding to the target vehicle area.
[0130] As an optional specific embodiment, in step 103, when there is a target pedestrian area satisfying the first condition in the i-th frame first image, after binding the target pedestrian area with the vehicle identification of the vehicle area, the method further includes:
[0131] When the number of second images in the target sliding window is less than the preset number, the i-th frame first image is added to the target sliding window, and the i-th frame first image serves as one of the second images in the target sliding window.
[0132] Specifically, for the target sliding window of the target vehicle identity, the number of second images in the target sliding window is set to a preset number, that is, the target sliding window stores a preset number of second images at most. After determining the target pedestrian area bound to the target vehicle identity, it is determined whether the number of second images in the target sliding window is equal to the preset number. If the number of second images in the target sliding window is less than the preset number, the first image of the i-th frame is directly added as a second image to the target sliding window to update the target sliding window. In the above manner, the number of second images in the target sliding window is filled to the preset number.
[0133] As an optional specific embodiment, the step 105 determines the warning image corresponding to the target vehicle identity according to the area and confidence of the target pedestrian area bound to the target vehicle identity in each second image in the target sliding window, including:
[0134] Acquire a first number of target pedestrian regions among the target pedestrian regions bound to the target vehicle identity in each second image in the target sliding window in descending order of the area of the target pedestrian regions;
[0135] The second image corresponding to the target pedestrian area with the highest confidence among the first number of target pedestrian areas is determined as the warning image corresponding to the target vehicle identity.
[0136] Specifically, let i=i+1 in step 101, and loop from step 101 to step 104. If in step 101, the vehicle identification corresponding to the vehicle area marked in the first image of the i-th frame does not include the target vehicle identification, it means that the vehicle with the target vehicle identification has driven out of the shooting range, and the loop process ends. The first number (e.g., 5) of target pedestrian areas in the target sliding window are obtained in descending order of the area of the target pedestrian areas, that is, the 5 target pedestrian areas with the areas in the top5 are obtained to ensure that a larger and clearer warning image is obtained. Then, the target pedestrian area with the highest confidence is selected from the 5 target pedestrian areas in the top5, and the target pedestrian area with the highest confidence is used as the warning image corresponding to the target vehicle identification, thereby ensuring that a warning image with high confidence and clear image is obtained, which is convenient for subsequent license plate recognition and traffic police teaching.
[0137] The processing flow of the target sliding window is described in detail below through a specific embodiment:
[0138] like Figure 6 As shown, the confidence and area of the pedestrian area where the pedestrian does not wear safety equipment in the first image of the i-th frame and the vehicle identity of the vehicle area are obtained, and a preset confidence is defined. If the confidence of the pedestrian area where the pedestrian does not wear safety equipment in the first image of the i-th frame is less than or equal to the preset confidence, it is determined that the cyclist (i.e., pedestrian) is wearing safety equipment, and the pedestrian area is eliminated. If the confidence of the pedestrian area where the pedestrian does not wear safety equipment in the first image of the i-th frame is greater than the preset confidence, it is preliminarily considered that the cyclist (i.e., pedestrian) who needs to be determined is not wearing a helmet.
[0139] For the target sliding window of the target vehicle identity, the preset number of the target sliding window is set to 30, that is, a maximum of 30 second images are stored in the target sliding window. After determining the target pedestrian area bound to the target vehicle identity, it is determined whether the number of second images in the target sliding window is equal to 30. If the number of second images in the target sliding window is equal to 30, it means that the target sliding window can no longer save additional images, and it is necessary to obtain the second image (i.e., the third image) corresponding to the target pedestrian area with the lowest confidence in the current target sliding window. If at least one target pedestrian area in the i-th frame of the first image is larger than the second image corresponding to the target pedestrian area with the lowest execution degree, the third image is replaced with the i-th frame of the first image. If the number of second images in the target sliding window is less than 30, the i-th frame of the first image is directly added as a second image to the target sliding window to update the target sliding window. i=i+1 loop the above steps, if the vehicle identity corresponding to the vehicle area marked in the i-th frame of the first image does not contain the target vehicle identity, that is, the target vehicle identity disappears, it means that the vehicle with the target vehicle identity has driven out of the shooting range, and the loop process ends.
[0140] In summary, in the above embodiment of the present application, the output results of the two models are feature fused by weighted averaging, and the final weighted average result of the two models is closer to the true value, which can reduce the deviation and variance of the model, and finally reduce the generalization error of the model, and can also improve the precision and recall of vehicles and pedestrians. In addition, by tracking the target in the vehicle area, DeepSort adds cascade matching (Matching Cascade) on this basis compared with the traditional method based on Kalman filtering and Hungarian algorithm. It is mainly based on the Hungarian algorithm for matching, and the weighted matching degree of the appearance model and the motion model is combined for matching. For the track that is not lost, matching is performed first to improve the accuracy and stability of the matching. This cascade matching method can effectively handle the situation of target occlusion and reappearance, and improve the accuracy and robustness of multi-target tracking. In addition, the vehicle area and the pedestrian area are matched by IOU association, which reduces the situation of false detection of pedestrians. In addition, the ITM method in BLIP is used to improve the accuracy of the alarm as a whole, greatly reducing the cost of manual review. In addition, the application has strong generalization capabilities, and different business scenario requirements can be quickly met through different prompts, such as: whether construction workers are wearing safety ropes; whether there are people fishing in the Yangtze River Basin; whether there are people holding knives or weapons in campus scenes, etc.
[0141] The above describes the method for determining the warning image provided in the embodiment of the present application. The following will describe the device for determining the warning image provided in the embodiment of the present application in conjunction with the accompanying drawings.
[0142] like Figure 7 As shown, the embodiment of the present application further provides a device 700 for determining a warning image, the device comprising:
[0143] An acquisition module 701 is used to acquire an i-th frame of a first image, wherein the i-th frame of the first image is marked with a pedestrian area and a vehicle area, and i is an integer greater than 0;
[0144] The processing module 702 is used to input the i-th frame first image into the multimodal model to obtain the area information of the pedestrian area in the i-th frame first image where the pedestrian does not wear the safety equipment, wherein the area information includes: the confidence, angular coordinates and area of the pedestrian area;
[0145] A binding module 703 is used to bind the target pedestrian area with the vehicle identification of the vehicle area when there is a target pedestrian area satisfying the first condition in the i-th frame of the first image;
[0146] A replacement module 704 is used to replace the third image with the first image of the i-th frame, when the number of second images in the target sliding window is equal to a preset number and the confidence of the target pedestrian area bound to the target vehicle identity in the third image of the target sliding window is less than the confidence of the target pedestrian area bound to the target vehicle identity in the i-th frame first image, wherein the third image is the second image with the smallest confidence of the target pedestrian area bound to the target vehicle identity in the target sliding window, and the target vehicle identity is the vehicle identity corresponding to any vehicle area in the i-th frame first image;
[0147] Determination module 705 is used to set i=i+1, loop the above content until the vehicle area marked with the target vehicle identity is not included in the first image of the i-th frame, and determine the warning image corresponding to the target vehicle identity according to the area and confidence of the target pedestrian area bound to the target vehicle identity in each second image in the target sliding window.
[0148] In the above embodiment of the present application, the first image of the i-th frame marked with pedestrian areas and vehicle areas is input into the multimodal model to obtain the confidence, angular coordinates and area of the pedestrian area in the i-th frame of the first image where the pedestrian is not wearing safety equipment. The multimodal large model replaces the traditional classification network method. Compared with the traditional method, the multimodal model has image understanding capabilities, making the identified pedestrian area and vehicle area more accurate. In addition, when there is a target pedestrian area that meets the first condition in the i-th frame of the first image, the target pedestrian area is bound to the vehicle identity of the vehicle area, thereby reducing the matching calculation amount of pedestrians and vehicles. Furthermore, when the number of second images in the target sliding window is equal to a preset number and the confidence of the target pedestrian area bound to the target vehicle identity in the third image of the target sliding window is less than the confidence of the target pedestrian area bound to the target vehicle identity in the first image of the i-th frame, the third image is replaced with the i-th frame first image, the third image is the second image with the smallest confidence of the target pedestrian area bound to the target vehicle identity in the target sliding window, the target vehicle identity is the vehicle identity corresponding to any vehicle area in the i-th frame first image, let i=i+1, and repeat the above content until the confidence of the target pedestrian area bound to the target vehicle identity in the i-th frame first image is reached. The image does not contain a vehicle area marked with the target vehicle identity, and the warning image corresponding to the target vehicle identity is determined according to the area and confidence of the target pedestrian area bound to the target vehicle identity in each second image in the target sliding window. That is, by introducing a sliding window mechanism, it is only necessary to sort the second images in the sliding window, and the number of images in the sliding window is always kept at most a preset number, thereby reducing the cache resource amount of the image; and the images in the sliding window are continuously updated to obtain a preset number of images with high confidence, and then the optimal warning image is determined by the area and confidence, thereby improving the determination efficiency of the warning image and reducing the overall calculation complexity.
[0149] Optionally, the acquisition module 701 is specifically configured to:
[0150] Get the fourth image of the i-th frame;
[0151] Vehicles and pedestrians are detected on the i-th frame fourth image, and the detected pedestrian area and vehicle area are marked in the i-th frame fourth image to obtain the i-th frame first image.
[0152] Optionally, when the acquisition module 701 detects vehicles and pedestrians on the i-th frame fourth image and marks the detected pedestrian area and vehicle area into the i-th frame fourth image to obtain the i-th frame first image, it is specifically used to:
[0153] Perform vehicle and pedestrian detection on the fourth image of the i-th frame by using the first model to obtain a first pedestrian region and a first vehicle region corresponding to the first model;
[0154] Perform vehicle and pedestrian detection on the fourth image of the i-th frame by using the second model to obtain a second pedestrian region and a second vehicle region corresponding to the second model;
[0155] Performing weighted average calculation on the first pedestrian region and the second pedestrian region to obtain a third pedestrian region;
[0156] Performing weighted average calculation on the first vehicle area and the second vehicle area to obtain a third vehicle area;
[0157] The third pedestrian region and the third vehicle region are marked in the i-th frame fourth image to obtain the i-th frame first image.
[0158] Optionally, the binding module 703 is specifically configured to:
[0159] A pedestrian region where the pedestrian does not wear safety equipment and the confidence is greater than a preset confidence and the center point of the angular coordinate is located in the area of interest of the i-th frame of the first image is determined as a target pedestrian region that meets the first condition;
[0160] Acquire a vehicle identity of each vehicle region in the i-th frame of the first image by a target tracking method;
[0161] The target pedestrian area is bound to the vehicle identity identifier of the vehicle area.
[0162] Optionally, when binding the target pedestrian area with the vehicle identity of the vehicle area, the binding module 703 is specifically used to:
[0163] In the case where there is a target pedestrian region in the i-th frame first image, performing an intersection-and-union ratio calculation on the target pedestrian region in the i-th frame first image and each vehicle region to obtain an intersection-and-union ratio result;
[0164] The target pedestrian region whose intersection-over-combination result is greater than the first result is bound to the vehicle identity identifier of the vehicle region.
[0165] Optionally, the device further comprises:
[0166] An adding module is used to add the i-th frame first image to the target sliding window when the number of second images in the target sliding window is less than the preset number, and the i-th frame first image serves as one of the second images in the target sliding window.
[0167] Optionally, when determining the warning image corresponding to the target vehicle identity according to the area and confidence of the target pedestrian area bound to the target vehicle identity in each second image in the target sliding window, the determination module 705 is specifically used to:
[0168] Acquire a first number of target pedestrian regions among the target pedestrian regions bound to the target vehicle identity in each second image in the target sliding window in descending order of the area of the target pedestrian regions;
[0169] The second image corresponding to the target pedestrian area with the highest confidence among the first number of target pedestrian areas is determined as the warning image corresponding to the target vehicle identity.
[0170] In summary, in the above embodiment of the present application, the output results of the two models are feature fused by weighted averaging, and the final weighted average result of the two models is closer to the true value, which can reduce the deviation and variance of the model, and finally reduce the generalization error of the model, and can also improve the precision and recall of vehicles and pedestrians. In addition, by tracking the target in the vehicle area, DeepSort adds cascade matching (Matching Cascade) on this basis compared with the traditional method based on Kalman filtering and Hungarian algorithm. It is mainly based on the Hungarian algorithm for matching, and the weighted matching degree of the appearance model and the motion model is combined for matching. For the track that is not lost, matching is performed first to improve the accuracy and stability of the matching. This cascade matching method can effectively handle the situation of target occlusion and reappearance, and improve the accuracy and robustness of multi-target tracking. In addition, the vehicle area and the pedestrian area are matched by IOU association, which reduces the situation of false detection of pedestrians. In addition, the ITM method in BLIP is used to improve the accuracy of the alarm as a whole, greatly reducing the cost of manual review. In addition, the application has strong generalization capabilities, and different business scenario requirements can be quickly met through different prompts, such as: whether construction workers are wearing safety ropes; whether there are people fishing in the Yangtze River Basin; whether there are people holding knives or weapons in campus scenes, etc.
[0171] It should be noted here that the above-mentioned alarm image determination device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned alarm image determination method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.
[0172] It should be noted that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0173] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.
[0174] like Figure 8 As shown, an embodiment of the present application further provides an electronic device, including a memory 820, a transceiver 810, and a processor 800:
[0175] A memory 820, for storing computer programs;
[0176] A transceiver 810, configured to transmit and receive data under the control of the processor;
[0177] The processor 800 is configured to read the computer program in the memory and execute the steps of the method for determining the warning image as described in any of the above embodiments.
[0178] Among them, Figure 8In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 800 and memory represented by memory 820. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 810 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium, which transmission medium may include a wireless channel, a wired channel, an optical cable, and other transmission media. The processor 800 is responsible for managing the bus architecture and general processing, and the memory 820 may store data used by the processor 800 when performing operations.
[0179] The processor 800 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD). The processor may also adopt a multi-core architecture.
[0180] The processor calls the computer program stored in the memory to execute the method for determining the warning image provided in the embodiment of the present application according to the obtained executable instructions. The processor and the memory can also be arranged physically separately.
[0181] It should be noted here that the above-mentioned electronic device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned alarm image determination method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.
[0182] An embodiment of the present application further provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, and the computer program is used to enable the processor to execute the above-mentioned method for determining the warning image.
[0183] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD)), etc.
[0184] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.
[0185] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer executable instructions. These computer executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0186] These processor executable instructions may also be stored in a processor readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the processor readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0187] These processor-executable instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0188] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A method for determining a warning image, characterized in that: The method comprises: Acquire an i-th frame of the first image, wherein the i-th frame of the first image is marked with a pedestrian area and a vehicle area, where i is an integer greater than 0; Inputting the first image of the i-th frame into the multimodal model, obtaining the region information of the pedestrian region in the i-th frame of the first image where the pedestrian does not wear the safety equipment, the region information including: the confidence, the angular coordinates and the area of the pedestrian region; In the case where there is a target pedestrian region satisfying the first condition in the i-th frame of the first image, binding the target pedestrian region with a vehicle identity identifier of the vehicle region; In a case where the number of second images in the target sliding window is equal to a preset number and the confidence of the target pedestrian area bound to the target vehicle identity in the third image of the target sliding window is less than the confidence of the target pedestrian area bound to the target vehicle identity in the i-th frame first image, the third image is replaced with the i-th frame first image, the third image being the second image with the smallest confidence of the target pedestrian area bound to the target vehicle identity in the target sliding window, and the target vehicle identity is the vehicle identity corresponding to any vehicle area in the i-th frame first image; Let i=i+1, and repeat the above content until the first image of the i-th frame does not contain the vehicle area marked with the target vehicle identity, and determine the warning image corresponding to the target vehicle identity according to the area and confidence of the target pedestrian area bound to the target vehicle identity in each second image in the target sliding window.
2. The method according to claim 1, characterized in that: The step of acquiring the first image of the i-th frame comprises: Get the fourth image of the i-th frame; Vehicles and pedestrians are detected on the i-th frame fourth image, and the detected pedestrian area and vehicle area are marked in the i-th frame fourth image to obtain the i-th frame first image.
3. The method according to claim 2, characterized in that The detecting of vehicles and pedestrians on the i-th frame fourth image and marking the detected pedestrian area and vehicle area on the i-th frame fourth image to obtain the i-th frame first image includes: Perform vehicle and pedestrian detection on the fourth image of the i-th frame by using the first model to obtain a first pedestrian region and a first vehicle region corresponding to the first model; Perform vehicle and pedestrian detection on the fourth image of the i-th frame by using the second model to obtain a second pedestrian region and a second vehicle region corresponding to the second model; Performing weighted average calculation on the first pedestrian region and the second pedestrian region to obtain a third pedestrian region; Performing weighted average calculation on the first vehicle area and the second vehicle area to obtain a third vehicle area; The third pedestrian region and the third vehicle region are marked in the i-th frame fourth image to obtain the i-th frame first image.
4. The method according to claim 1, characterized in that: When there is a target pedestrian region satisfying the first condition in the i-th frame of the first image, binding the target pedestrian region with a vehicle identity identifier of the vehicle region includes: A pedestrian region where the pedestrian does not wear safety equipment and the confidence is greater than a preset confidence and the center point of the angular coordinate is located in the area of interest of the i-th frame of the first image is determined as a target pedestrian region that meets the first condition; Acquire a vehicle identity of each vehicle region in the i-th frame of the first image by a target tracking method; The target pedestrian area is bound to the vehicle identity identifier of the vehicle area.
5. The method according to claim 4, characterized in that The step of binding the target pedestrian area with the vehicle identity identifier of the vehicle area includes: In the case where there is a target pedestrian region in the i-th frame first image, performing an intersection-and-union ratio calculation on the target pedestrian region in the i-th frame first image and each vehicle region to obtain an intersection-and-union ratio result; The target pedestrian region whose intersection-over-combination result is greater than the first result is bound to the vehicle identity identifier of the vehicle region.
6. The method according to claim 1, characterized in that In the case where there is a target pedestrian region satisfying the first condition in the i-th frame of the first image, after binding the target pedestrian region with the vehicle identity identifier of the vehicle region, the method further includes: When the number of second images in the target sliding window is less than the preset number, the i-th frame first image is added to the target sliding window, and the i-th frame first image serves as one of the second images in the target sliding window.
7. The method according to claim 1, characterized in that The step of determining the warning image corresponding to the target vehicle identity according to the area and the confidence of the target pedestrian region bound to the target vehicle identity in each second image in the target sliding window comprises: Acquire a first number of target pedestrian regions among the target pedestrian regions bound to the target vehicle identity in each second image in the target sliding window in descending order of the area of the target pedestrian regions; The second image corresponding to the target pedestrian area with the highest confidence among the first number of target pedestrian areas is determined as the warning image corresponding to the target vehicle identity.
8. A device for determining a warning image, characterized in that: The device comprises: An acquisition module, used for acquiring an i-th frame of a first image, wherein the i-th frame of the first image is marked with a pedestrian area and a vehicle area, and i is an integer greater than 0; A processing module, configured to input the i-th frame first image into a multimodal model to obtain region information of a pedestrian region in the i-th frame first image where the pedestrian is not wearing safety equipment, wherein the region information includes: a confidence level, an angular coordinate, and an area of the pedestrian region; A binding module, configured to bind the target pedestrian region to a vehicle identity identifier of a vehicle region when there is a target pedestrian region satisfying a first condition in the i-th frame of the first image; A replacement module, configured to replace the third image with the i-th frame first image, when the number of second images in the target sliding window is equal to a preset number and the confidence of the target pedestrian area bound to the target vehicle identity in the third image of the target sliding window is less than the confidence of the target pedestrian area bound to the target vehicle identity in the i-th frame first image, wherein the third image is the second image with the smallest confidence of the target pedestrian area bound to the target vehicle identity in the target sliding window, and the target vehicle identity is a vehicle identity corresponding to any vehicle area in the i-th frame first image; A determination module is used to set i=i+1, loop the above content until the vehicle area marked with the target vehicle identity is not included in the first image of the i-th frame, and determine the warning image corresponding to the target vehicle identity according to the area and confidence of the target pedestrian area bound to the target vehicle identity in each second image in the target sliding window.
9. The device according to claim 8, characterized in that The acquisition module is specifically used for: Get the fourth image of the i-th frame; Vehicles and pedestrians are detected on the i-th frame fourth image, and the detected pedestrian area and vehicle area are marked in the i-th frame fourth image to obtain the i-th frame first image.
10. The device according to claim 9, characterized in that When the acquisition module detects vehicles and pedestrians on the i-th frame fourth image and marks the detected pedestrian area and vehicle area on the i-th frame fourth image to obtain the i-th frame first image, it is specifically used to: Perform vehicle and pedestrian detection on the fourth image of the i-th frame by using the first model to obtain a first pedestrian region and a first vehicle region corresponding to the first model; Perform vehicle and pedestrian detection on the fourth image of the i-th frame by using the second model to obtain a second pedestrian region and a second vehicle region corresponding to the second model; Performing weighted average calculation on the first pedestrian region and the second pedestrian region to obtain a third pedestrian region; Performing weighted average calculation on the first vehicle area and the second vehicle area to obtain a third vehicle area; The third pedestrian region and the third vehicle region are marked in the i-th frame fourth image to obtain the i-th frame first image.
11. The device according to claim 8, characterized in that The binding module is specifically used for: A pedestrian region where the pedestrian does not wear safety equipment and the confidence is greater than a preset confidence and the center point of the angular coordinate is located in the area of interest of the i-th frame of the first image is determined as a target pedestrian region that meets the first condition; Acquire a vehicle identity of each vehicle region in the i-th frame of the first image by a target tracking method; The target pedestrian area is bound to the vehicle identity identifier of the vehicle area.
12. The device according to claim 11, characterized in that When binding the target pedestrian area with the vehicle identification mark of the vehicle area, the binding module is specifically used to: In the case where there is a target pedestrian region in the i-th frame first image, performing an intersection-and-union ratio calculation on the target pedestrian region in the i-th frame first image and each vehicle region to obtain an intersection-and-union ratio result; The target pedestrian region whose intersection-over-combination result is greater than the first result is bound to the vehicle identity identifier of the vehicle region.
13. The device according to claim 8, characterized in that The device also includes: An adding module is used to add the i-th frame first image to the target sliding window when the number of second images in the target sliding window is less than the preset number, and the i-th frame first image serves as one of the second images in the target sliding window.
14. The device according to claim 8, characterized in that When the determination module determines the warning image corresponding to the target vehicle identity according to the area and confidence of the target pedestrian area bound to the target vehicle identity in each second image in the target sliding window, the determination module is specifically used to: Acquire a first number of target pedestrian regions among the target pedestrian regions bound to the target vehicle identity in each second image in the target sliding window in descending order of the area of the target pedestrian regions; The second image corresponding to the target pedestrian area with the highest confidence among the first number of target pedestrian areas is determined as the warning image corresponding to the target vehicle identity.
15. An electronic device, characterized in that: Including memory, transceiver, processor: Memory for storing computer programs; a transceiver, for transmitting and receiving data under the control of the processor; A processor, configured to read the computer program in the memory and execute the method for determining the warning image according to any one of claims 1 to 7.
16. A processor-readable storage medium, characterized in that: The processor-readable storage medium stores a computer program, and the computer program is used to enable the processor to execute the method for determining a warning image according to any one of claims 1 to 7.
Citation Information
Cited By
Container damage information detection and counting method based on yolov9 and DeepSort
CN119229221A
A container damage information detection and counting method based on yolov9 and DeepSort
CN119229221B
