A safety warning method, device, electronic device and storage medium
By detecting the key points of the human body and detecting the outside image area of the vehicle, safety alarm information is output, and the dangerous problem of people in the vehicle sticking their bodies out of the window during travel is solved, and the prevention and safety guarantee of accidents is achieved.
Patent Information
- Application Number
- CN202111056039.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-09
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-09-09
AI Technical Summary
During travel, it is dangerous for drivers or passengers to stick their bodies out of the window, and it is difficult to effectively detect and remind and prevent accidents in the prior art.
By detecting the human body key points in the car image, if it is detected that the key points are located in the set area, the outside image is obtained and the area detection model determines whether there is a human body image area. If it exists, safety alarm information is output.
Accurate judgment on whether the personnel in the car are close to or extending their bodies out of the window, and timely warnings are issued to reduce accidents and ensure personnel safety.
Smart Images

Figure CN113762168B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and more specifically, to a security alarm method, device, electronic device and storage medium. Background Art
[0002] With the development of computer network technology, smart taxi-hailing, such as online ride-hailing, has gradually become the main means of transportation.
[0003] During the trip, both the driver and the passenger may stick their body out of the window, which is extremely dangerous and can easily lead to an accident. Therefore, how to detect whether the passenger and the driver stick their body out of the window during driving is an urgent problem to be solved. Summary of the invention
[0004] In order to solve the above-mentioned problems in the prior art, an embodiment of the present application provides a safety alarm method, which can monitor in real time whether the driver or passenger in the car has any behavior of putting their body close to the window or sticking their body out of the window, and give corresponding alarms to ensure the safety of people.
[0005] In a first aspect, an embodiment of the present application provides a security alarm method, the method comprising:
[0006] Performing human body key point detection on the acquired in-vehicle image of the target vehicle to obtain the coordinate positions of multiple human body key points;
[0007] If the coordinate position of at least one of the plurality of human body key points is located within a set area, then acquiring an exterior image of the target vehicle; the set area is used to indicate an area corresponding to a window of the target vehicle in the interior image; and the exterior image includes the window of the target vehicle;
[0008] If the image outside the vehicle contains a human body image area, a safety warning message is output.
[0009] In a possible implementation, the step of performing human key point detection on the acquired in-vehicle image of the target vehicle to obtain coordinate positions of multiple human key points includes:
[0010] The key point detection model is used to detect the key points of the human body in the in-vehicle image, and the coordinate positions of multiple key points of the human body output by the key point detection model are obtained; the key point detection model is trained based on sample images with human body position annotation information.
[0011] In a possible implementation, the key point detection model includes a first convolutional subnetwork and a first output subnetwork; the key point detection model is used to perform human key point detection on the in-vehicle image to obtain coordinate positions of multiple human key points output by the key point detection model, including:
[0012] Inputting the in-vehicle image into the first convolutional subnetwork, performing feature extraction on the in-vehicle image through the first convolutional subnetwork, and obtaining a first in-vehicle image feature map;
[0013] The first in-vehicle image feature map is input into the first output sub-network to obtain the coordinate positions of the multiple human body key points output by the first output sub-network.
[0014] In a possible implementation, the key point detection model includes a second convolution subnetwork, a first classification subnetwork, a first regression subnetwork, and a second output subnetwork; the key point detection model is used to perform human key point detection on the in-vehicle image to obtain the coordinate positions of multiple human key points output by the key point detection model, including:
[0015] Inputting the in-vehicle image into the second convolutional subnetwork, performing feature extraction on the in-vehicle image through the second convolutional subnetwork, and obtaining a second in-vehicle image feature map;
[0016] Inputting the second in-vehicle image feature map into the first classification subnetwork, and classifying objects contained in the second in-vehicle image feature map through the first classification subnetwork to obtain a first classification feature map;
[0017] Inputting the first classification feature map into the first regression sub-network, and obtaining an in-vehicle marked feature map at a position where the human body area is marked on the first classification feature map through the first regression sub-network;
[0018] The in-vehicle annotated feature map is input into the second output sub-network to obtain the coordinate positions of the multiple human body key points output by the second output sub-network based on the in-vehicle annotated feature map.
[0019] In a possible manner, if the coordinate position of at least one of the plurality of human body key points is located within a set area, acquiring the external image of the target vehicle includes:
[0020] If the coordinate position of the at least one human body key point is located in the set area corresponding to the left window of the target vehicle, then obtaining the vehicle exterior image of the left side of the target vehicle;
[0021] If the coordinate position of the at least one human body key point is located in the set area corresponding to the right side window of the target vehicle, an exterior image of the right side of the target vehicle is acquired.
[0022] In one possible manner, if the image outside the vehicle includes a human body image area, before outputting the safety warning information, the method includes:
[0023] Human body detection is performed on the image outside the vehicle through a region detection model, and whether the image outside the vehicle contains a human body image region is determined based on the output of the region detection model; the region detection model is trained based on a training image with human body key point position annotation information.
[0024] In a possible manner, the region detection model includes a third convolution subnetwork and a second regression subnetwork; performing human body detection on the external vehicle image through the region detection model, and determining whether the external vehicle image contains a human body image region according to an output of the region detection model, includes:
[0025] Inputting the vehicle exterior image into the third convolutional subnetwork, performing feature extraction on the vehicle exterior image through the third convolutional subnetwork, and obtaining a first vehicle exterior image feature map;
[0026] The first vehicle exterior image feature map is input into the second regression subnetwork. If an image of the vehicle exterior with marked human body regions is obtained, it is determined that the image of the vehicle exterior contains human body image regions; if an image of the vehicle exterior without marked human body regions is obtained, it is determined that the image of the vehicle exterior does not contain human body image regions.
[0027] In a possible implementation, the region detection model includes a fourth convolution subnetwork, a second classification subnetwork, and a third regression subnetwork; performing human body detection on the external vehicle image through the region detection model, and determining whether the external vehicle image contains a human body image region according to an output of the region detection model, includes:
[0028] Inputting the vehicle exterior image into the fourth convolutional subnetwork, performing feature extraction on the vehicle exterior image through the fourth convolutional subnetwork, and obtaining a second vehicle exterior image feature map;
[0029] Inputting the second vehicle exterior image feature map into the second classification subnetwork, and classifying objects contained in the second vehicle exterior image feature map through the second classification subnetwork to obtain a second classification feature map;
[0030] The second classification feature map is input into the third regression subnetwork. If an image outside the vehicle with a human body area marked is obtained, it is determined that the image outside the vehicle contains a human body image area; if an image without a human body area marked is obtained, it is determined that the image outside the vehicle does not contain a human body image area.
[0031] In a second aspect, an embodiment of the present application provides a safety alarm device, including:
[0032] A determination unit, used to detect key points of a human body on the acquired in-vehicle image of the target vehicle, and obtain coordinate positions of multiple key points of the human body;
[0033] A judgment unit, configured to obtain an exterior image of the target vehicle if the coordinate position of at least one of the plurality of human body key points is within a set area; the set area is used to indicate an area corresponding to a window of the target vehicle in the interior image; and the exterior image includes the window of the target vehicle;
[0034] The warning unit is used to output safety warning information if the image outside the vehicle contains a human body image area.
[0035] In a possible implementation manner, the determining unit is further configured to:
[0036] The key point detection model is used to detect the key points of the human body in the in-vehicle image, and the coordinate positions of multiple key points of the human body output by the key point detection model are obtained; the key point detection model is trained based on sample images with human body position annotation information.
[0037] In a possible implementation manner, the determining unit is further configured to:
[0038] Inputting the in-vehicle image into the first convolutional subnetwork, performing feature extraction on the in-vehicle image through the first convolutional subnetwork, and obtaining a first in-vehicle image feature map;
[0039] The first in-vehicle image feature map is input into the first output sub-network to obtain the coordinate positions of the multiple human body key points output by the first output sub-network.
[0040] In a possible implementation manner, the determining unit is further configured to:
[0041] Inputting the in-vehicle image into the second convolutional subnetwork, performing feature extraction on the in-vehicle image through the second convolutional subnetwork, and obtaining a second in-vehicle image feature map;
[0042] Inputting the second in-vehicle image feature map into the first classification subnetwork, and classifying objects contained in the second in-vehicle image feature map through the first classification subnetwork to obtain a first classification feature map;
[0043] Inputting the first classification feature map into the first regression sub-network, and obtaining an in-vehicle marked feature map at a position where the human body area is marked on the first classification feature map through the first regression sub-network;
[0044] The in-vehicle annotated feature map is input into the second output sub-network to obtain the coordinate positions of the multiple human body key points output by the second output sub-network based on the in-vehicle annotated feature map.
[0045] In a possible implementation manner, the determining unit is further configured to:
[0046] If the coordinate position of at least one key point of the human body is located in the set area corresponding to the left window of the target vehicle, the image outside the left side of the target vehicle is obtained; if the coordinate position of at least one key point of the human body is located in the set area corresponding to the right window of the target vehicle, the image outside the right side of the target vehicle is obtained.
[0047] In a possible implementation manner, the safety warning device further includes:
[0048] The detection unit is used to perform human body detection on the image outside the vehicle through a region detection model, and determine whether the image outside the vehicle contains a human body image region based on the output of the region detection model; the region detection model is obtained by training based on a training image with human body key point position annotation information.
[0049] In a possible implementation manner, the detection unit is further used for:
[0050] Inputting the vehicle exterior image into the third convolutional subnetwork, performing feature extraction on the vehicle exterior image through the third convolutional subnetwork, and obtaining a first vehicle exterior image feature map;
[0051] The first vehicle exterior image feature map is input into the second regression subnetwork. If an image of the vehicle exterior with marked human body regions is obtained, it is determined that the image of the vehicle exterior contains human body image regions; if an image of the vehicle exterior without marked human body regions is obtained, it is determined that the image of the vehicle exterior does not contain human body image regions.
[0052] In a possible implementation, the region detection model includes a fourth convolution subnetwork, a second classification subnetwork, and a third regression subnetwork; the detection unit is further used to:
[0053] Inputting the vehicle exterior image into the fourth convolutional subnetwork, performing feature extraction on the vehicle exterior image through the fourth convolutional subnetwork, and obtaining a second vehicle exterior image feature map;
[0054] Inputting the second vehicle exterior image feature map into the second classification subnetwork, and classifying objects contained in the second vehicle exterior image feature map through the second classification subnetwork to obtain a second classification feature map;
[0055] The second classification feature map is input into the third regression subnetwork. If an image outside the vehicle with a human body area marked is obtained, it is determined that the image outside the vehicle contains a human body image area; if an image without a human body area marked is obtained, it is determined that the image outside the vehicle does not contain a human body image area.
[0056] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the steps of any one of the security alarm methods in the first aspect are implemented.
[0057] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the security alarm methods in the first aspect are implemented.
[0058] The safety warning method provided in the embodiment of the present application obtains the interior image of the target vehicle, performs human body key point detection on the interior image, and obtains the coordinate positions of multiple human body key points. If the coordinate position of at least one of the multiple human body key points is within the set area, it proves that a passenger or driver in the car has placed a part of his body close to the window or stretched out of the window. Then, an exterior image of the target vehicle is obtained, and the exterior image contains the window of the target vehicle. If the exterior image contains a human body image area, it is detected whether the exterior image contains a human body area image. If so, it proves that a passenger or driver has indeed stretched his body out of the window. Then, a safety warning message is output to remind the people in the car to pay attention, reduce the occurrence of accidents, and ensure the safety of the people. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0060] Figure 1 A flowchart of a security warning method provided in an embodiment of the present application;
[0061] Figure 2 A schematic diagram of the structure of a key point detection model provided in an embodiment of the present application;
[0062] Figure 3 A schematic diagram of the structure of another key point detection model provided in an embodiment of the present application;
[0063] Figure 4 A schematic diagram of the structure of a region detection model provided in an embodiment of the present application;
[0064] Figure 5 A schematic diagram of the structure of another area detection model provided in an embodiment of the present application;
[0065] Figure 6 A structural schematic diagram used in the key point detection model training process provided in an embodiment of the present application;
[0066] Figure 7 A structural schematic diagram used in the region detection model training process provided in an embodiment of the present application;
[0067] Figure 8 A flowchart of another security warning method provided by an embodiment of the present application;
[0068] Fig. 9 A schematic diagram of the structure of a safety alarm device provided in an embodiment of the present application;
[0069] Fig.10 A schematic diagram of the structure of another safety alarm device provided in an embodiment of the present application;
[0070] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0071] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0072] It should be noted that the terms "including" and "having" and their variations involved in the documents of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0073] During the trip, both the driver and the passenger may stick their body out of the window. Such behavior is extremely dangerous and can easily lead to accidents. In order to remind passengers or drivers in real time, the embodiment of the present application provides a safety warning method, by acquiring the in-vehicle image of the target vehicle, detecting the key points of the human body on the in-vehicle image, and obtaining the coordinate positions of multiple key points of the human body. If the coordinate position of at least one of the multiple key points of the human body is located in the set area, it proves that there is a passenger or driver in the car who has put a part of his body close to the window or stretched it out of the window, then the image outside the vehicle of the target vehicle is acquired, and the set area is used to indicate the area corresponding to the window of the target vehicle in the in-vehicle image. The window of the target vehicle is included in the image outside the vehicle. If the image outside the vehicle contains the human body image area, it is detected whether the image outside the vehicle contains the human body area image. If it exists, it proves that there is indeed a passenger or driver who has stretched his body out of the window, and then a safety warning message is output to warn the people in the car. The safety warning method in the present application can accurately determine whether the people in the car have the behavior of putting their body close to the window or stretching it out of the window, so as to ensure the safety of the people in the car.
[0074] Figure 1 The flowchart of a safety warning method provided by an embodiment of the present application is shown; the safety warning method can be applied to electronic devices, specifically electronic devices such as vehicle-mounted terminals or driver's mobile phones, and the following description will take the vehicle-mounted terminal as an example. Figure 1 As shown, the security alarm method includes the following steps:
[0075] Step S101: performing human body key point detection on the acquired in-vehicle image of the target vehicle to obtain the coordinate positions of multiple human body key points.
[0076] In a possible embodiment, the driver can choose to open the vehicle terminal of the target vehicle during driving, and the vehicle terminal is connected to the in-vehicle camera and the outside camera installed on the outside rearview mirror. The in-vehicle camera can choose to use a DVR (Digital Video Recorder) camera to obtain the in-vehicle image of the target vehicle; the outside camera can choose to use a BSD (Blind Spot Detection) camera to obtain the outside image of the target vehicle.
[0077] The vehicle terminal uses the in-vehicle camera to obtain the in-vehicle image of the target vehicle, and performs human key point detection on the obtained in-vehicle image of the target vehicle, and can output the coordinate positions of multiple human key points in the in-vehicle image. Among them, the human key points can be key points of different human parts, such as key points of the head or key points of the limbs. By performing human key point detection on the obtained in-vehicle image, the coordinate positions of all human key points contained in the in-vehicle image can be obtained.
[0078] In other embodiments, when the target vehicle is stationary, an image inside the target vehicle may be acquired to detect whether a limb of a person inside the vehicle is sticking out of the vehicle. For example, when the target vehicle is waiting for a red light or when the target vehicle is parked in a parking lot, it may be detected whether a limb of a person inside the vehicle is sticking out of the vehicle.
[0079] Step S102: If the coordinate position of at least one of the plurality of human body key points is within the set area, an external image of the target vehicle is acquired.
[0080] In a possible embodiment, after obtaining the coordinate positions of multiple key points of the human body, the vehicle-mounted terminal can determine whether the coordinate positions of the multiple key points of the human body are located in a set area. The set area is a pre-calibrated area that can be used to indicate the area corresponding to the window of the target vehicle in the in-vehicle image. If the installation position and angle of the in-vehicle camera do not change, the position of the set area in the in-vehicle image remains unchanged; if the installation position or angle of the in-vehicle camera is adjusted, it is necessary to recalibrate the position of the set area in the in-vehicle image captured by the in-vehicle camera.
[0081] If the coordinate position of any one or more key points of the human body is within the set area, it means that the person in the car has put part of his body close to the window or stretched out of the window, and the external camera is used to obtain the external image of the target vehicle. The external image contains the window of the target vehicle, which is used to further determine whether there is part of the body sticking out of the window.
[0082] When the coordinate position of at least one of the multiple human key points is within the set area, the camera outside the car is turned on to collect images outside the car. If the coordinate positions of the multiple human key points are not within the set area, it means that the person in the car has not put his body close to or stretched out of the window, and there is no need to turn on the camera outside the car, which can avoid wasting resources by keeping the camera outside the car turned on all the time. If the camera outside the car is turned on all the time, pedestrians passing by the car may also be detected, resulting in false triggering of safety alarm information. Therefore, the embodiment of the present application can reduce the situation of false triggering.
[0083] In a possible implementation, the target vehicle may be equipped with an exterior camera on the left rearview mirror and the right rearview mirror of the vehicle, respectively. For the interior image captured by the interior camera, two set areas may be calibrated, one of which is the set area corresponding to the left window, and the other is the set area corresponding to the right window.
[0084] If the coordinate position of at least one key point of the human body is located in the set area corresponding to the left window of the target vehicle, it means that the person in the car has put part of his body close to the window on the left side of the vehicle or stretched it out of the window. At this time, the external image of the left side of the target vehicle can be obtained through the external camera on the left side of the vehicle. If the coordinate position of at least one key point of the human body is located in the set area corresponding to the right window of the target vehicle, it means that the person in the car has put part of his body close to the window on the right side of the vehicle or stretched it out of the window. At this time, the external image of the right side of the target vehicle can be obtained through the external camera on the right side of the vehicle. If the position coordinates of at least one key point of the human body are detected in the set area corresponding to the left window and the set area corresponding to the right window, then choose to open the external cameras on both sides to collect the external image. This targeted collection of external images can save resources.
[0085] Step S103: If the image outside the vehicle contains a human body image area, a safety warning message is output.
[0086] In a possible embodiment, after acquiring the image outside the vehicle, it is detected whether there is a human image area in the image outside the vehicle. If there is a human image area, it means that a person in the vehicle has stretched his body out of the window, and a safety warning message can be issued through voice prompts or other prompts. For example, a voice prompt "Please do not stick your head or body out of the window" can be issued.
[0087] The safety warning method in this application can accurately determine whether the person in the car has placed his body close to the window or stretched his body out of the window, and issue a warning in time to reduce the occurrence of accidents and ensure the safety of the people in the car.
[0088] In a possible implementation, in the above step S101, key point detection of human body may be performed on the in-vehicle image by using a key point detection model to obtain coordinate positions of multiple key points of human body output by the key point detection model.
[0089] For example, a trained key point detection model can be installed on the vehicle terminal. After the in-vehicle camera obtains the in-vehicle image, the in-vehicle image can be input into the key point detection model, and the trained key point detection model can be used to detect the key points of the human body in the in-vehicle image to obtain the coordinate positions of multiple key points of the human body. The key point detection model is trained based on sample images with human body position annotated information.
[0090] In one embodiment, the structure of the key point detection model can be as follows Figure 2As shown, it includes a first convolution subnetwork and a first output subnetwork. The process of processing the in-car image through the key point detection model includes: inputting the in-car image into the first convolution subnetwork, extracting features of the in-car image through the first convolution subnetwork to obtain a first in-car image feature map, inputting the first in-car image feature map into the first output subnetwork, and obtaining the coordinate positions of multiple human key points output by the first output subnetwork. By judging whether the coordinate position of at least one of the multiple human key points is within the preset area, it can be judged whether there is a person in the car with a body part close to or extending out of the window.
[0091] In another embodiment, the structure of the key point detection model can be as follows Figure 3 As shown, it includes a second convolution subnetwork, a first classification subnetwork, a first regression subnetwork and a second output subnetwork. The process of processing the in-car image through the key point detection model includes: inputting the in-car image into the second convolution subnetwork, extracting features of the in-car image through the second convolution subnetwork, obtaining a second in-car image feature map, inputting the second in-car image feature map into the first classification subnetwork, classifying the objects contained in the second in-car image feature map through the first classification subnetwork, obtaining a first classification feature map, inputting the first classification feature map into the first regression subnetwork, and obtaining an in-car marked feature map at the position of the human body area in the first classification feature map through the first regression subnetwork, and inputting the in-car marked feature map into the second output subnetwork to obtain the coordinate positions of multiple human key points output by the second output subnetwork based on the in-car marked feature map. By judging whether the coordinate position of at least one of the multiple human key points is within the preset area, it can be judged whether there is a person in the car with a body part close to or extending out of the window.
[0092] In a possible implementation, when determining whether the image outside the vehicle contains a human image area, a human body detection can be performed on the image outside the vehicle through a region detection model, and based on the output of the region detection model, it is determined whether the image outside the vehicle contains a human body image area.
[0093] Specifically, the vehicle terminal can be equipped with a trained region detection model. After the vehicle camera obtains the vehicle image, the vehicle image can be input into the region detection model, and the trained region detection model can be used to perform human image region detection on the vehicle image to determine whether the vehicle image contains a human image region, which can be an image region of a key point of the human body. The region detection model is trained based on training images with labeled information of the key points of the human body.
[0094] In one embodiment, the processing process of the area detection model for the outside vehicle image is: input the outside vehicle image into the third convolution sub-network, extract features of the outside vehicle image through the third convolution sub-network, and obtain a first outside vehicle image feature map; input the first outside vehicle image feature map into the second regression sub-network, and if an outside vehicle labeled image with a human body area is obtained, it is determined that the outside vehicle image contains a human body image area; if an image without a human body area is obtained, it is determined that the outside vehicle image does not contain a human body image area.
[0095] In another embodiment, the processing process of the area detection model for the outside vehicle image is: inputting the outside vehicle image into the fourth convolution subnetwork, extracting features of the outside vehicle image through the fourth convolution subnetwork, and obtaining a second outside vehicle image feature map; inputting the second outside vehicle image feature map into the second classification subnetwork, and classifying the objects contained in the second outside vehicle image feature map through the second classification subnetwork to obtain a second classification feature map; inputting the second classification feature map into the third regression subnetwork, and if an outside vehicle labeled image with a human body area is obtained, it is determined that the outside vehicle image contains a human body image area; if an image without a human body area is obtained, it is determined that the outside vehicle image does not contain a human body image area.
[0096] The following introduces the training processes of the key point detection model and the region detection model used in the above embodiments.
[0097] In one possible embodiment, during training Figure 2 When the key point detection model is shown, the classification sub-network and the regression sub-network can be used to assist in the training of the key point detection model. Specifically, Figure 6 As shown, the auxiliary training classification subnetwork, the auxiliary training regression subnetwork and the first output subnetwork are connected in parallel. During the training process, the output of the first convolution subnetwork is used as the input of the auxiliary training classification subnetwork, the auxiliary training regression subnetwork and the first output subnetwork respectively.
[0098] in, Figure 2 The specific training process of the key point detection model shown includes:
[0099] Step a1, obtaining a first training data set, wherein the sample images in the first training data set are sample images with human body position annotation information. The human body position annotation information includes human body category labels, annotation boxes of human body key points, and specific coordinate positions of human body key points.
[0100] Step b1, extracting sample images from the first training data set, inputting the extracted sample images into the first convolutional subnetwork, and obtaining a first convolutional sample feature map.
[0101] Step c1, input the first convolution sample feature map into the auxiliary training classification subnetwork to obtain an auxiliary classification human sample feature image containing the human body position and a classification loss value.
[0102] The auxiliary training classification subnetwork is used to classify the feature maps of human body parts and non-human body parts in the in-car image, and outputs the auxiliary classification human body sample feature map, that is, the sample feature image of the human body in the in-car image. The loss is calculated using the output of the auxiliary training classification subnetwork and the sample image containing the human body category label to determine the classification loss value.
[0103] Exemplarily, the loss function formula for determining the classification loss value is as follows:
[0104]
[0105] Among them, y is the auxiliary classification feature map output by the auxiliary training classification subnetwork, y * It is the feature map with human category labels in the first training data set.
[0106] Step d1, input the first convolution sample feature map into the auxiliary training regression subnetwork to obtain the auxiliary human key point sample feature map and regression loss value.
[0107] The auxiliary training regression sub-network is used to determine the specific positions of the key points of the human body in the first convolution sample feature map, and mark the key points of the human body in the first convolution sample feature map with a label frame. The output of the auxiliary training regression sub-network and the image with human body position labeling information are used to calculate the loss and determine the regression loss value.
[0108] The loss function that determines the regression loss value is as follows:
[0109]
[0110] t is the auxiliary regression feature map output by the regression subnetwork of auxiliary training, t * It is a feature map of the labeled box with key points of the human body in the first training data set.
[0111] Among them, the loss function for determining the regression loss value specifically includes:
[0112]
[0113] β 2 (t,t * )=(x p1 -x p2 ) 2 +(y p1 -y p2 ) 2
[0114] x p1 =x t2 -x t1 ,y p1 = y t2 -y t1
[0115] x p2 =x t*2 -x t*1 ,y p2 = y t*2 -y t*1
[0116] c 2 =(x c1 -x c2 ) 2 +(y c1 -y c2 ) 2
[0117] x c1 =min(x t1 , x t*1 ), x c2 =min(x t2 , x t*2 )
[0118] y c1 =min(y t1 ,y t*1 ), y c2 =min(y t2 ,y t*2 )
[0119] Among them, IOU (Intersection over Union) is the intersection of the image with the labeled boxes of the key points of the human body and the image output by the regression subnetwork divided by their union. The larger the value, the more overlapping areas there are, which proves that the more overlaps the output image has with the image with the labeled boxes of the key points of the human body, the better the training effect, and the more the loss value meets the preset requirements.
[0120] The m in the loss function of the above key point detection model is specifically determined according to the intersection-and-union ratio in the loss function of the regression subnetwork. If the intersection-and-union ratio is greater than 0.5, the value of m is 1; if the intersection-and-union ratio is less than 0.5, the value of m is 0.
[0121] Among them, x p1 , x t2 , x t1 , x p2The contents such as p and t in the content only serve to distinguish. (x t1 ,y t1 ), (x t2 ,y t2 ) refers to the coordinates of the upper left corner and the lower right corner of the image output by the regression subnetwork, (x t*1 ,y t*1 ), (x t*2 ,y t*2 ) are the coordinates of the upper left corner and the lower right corner of the image with the labeled box of the key points of the human body. 2 =(x c1 -x c2 ) 2 +(y c1 -Y c2 ) 2 To calculate the distance between coordinate points.
[0122] Step e1, inputting the first convolution sample feature image into the first output subnetwork to obtain the coordinate positions of multiple human body key points corresponding to the first output subnetwork and the first output loss value.
[0123] The first output subnetwork is used to obtain the coordinate positions of multiple human body key points, and use the output coordinate positions of multiple human body key points and the image annotated with the coordinate positions of the human body key points to perform loss calculation to determine the first output loss value.
[0124] The loss function that determines the first output loss value is as follows:
[0125]
[0126] l is the coordinate position of multiple human key points output by the first output sub-network, l * It is the specific coordinate position of the key points of the human body.
[0127] Step f1, adjusting the network parameters of the key point detection model according to the classification loss value, the regression loss value and the first output loss value until a trained key point detection model is obtained.
[0128] Specifically, the total loss value of the key point detection model can be determined according to the classification loss value, the regression loss value and the first output loss value.
[0129] Exemplarily, the total loss value LOSS of the key point detection model can be determined by the following formula:
[0130] LOSS 关键点 =k1L class (y,y * )+k2mL bbox (t,t * )+k3mLpts (l,l * )
[0131] L class (y,y * ) is the loss function of the auxiliary training classification sub-network, and k1 represents the weight of the loss function of the auxiliary training classification sub-network.
[0132] L bbox (t,t * ) is the loss function of the regression sub-network for auxiliary training, k2 represents the weight of the loss function of the regression sub-network for auxiliary training; m is determined according to the size of the loss function of the regression sub-network.
[0133] L pts (l,l * ) is the loss function of the first output sub-network, and k3 represents the weight of the loss function of the first output sub-network.
[0134] According to the total loss value of the key point detection model, the network parameters of the key point detection model are adjusted. If the total loss value of the key point detection model meets the preset range, it means that the training of the key point detection model is completed; if the total loss value of the key point detection model does not meet the preset range, return to step b1 and continue to train the key point detection model until the total loss value of the key point detection model meets the preset range, and output the trained key point detection model.
[0135] After the key point detection model is trained, the coordinate positions of multiple key points of the human body can be obtained in the application process, so you can choose to turn off the auxiliary training classification sub-network and auxiliary training regression sub-network, which can save data processing pressure on the vehicle terminal and save resources. If you want to get the output results of the auxiliary training classification sub-network and auxiliary training regression sub-network, you can also choose to use the classification sub-network and regression sub-network in the application process.
[0136] In one possible embodiment, during training Figure 3 When the key point detection model is shown, Figure 3 As shown, the first classification sub-network, the first regression sub-network and the second output sub-network are connected in sequence.
[0137] in, Figure 3 The specific training process of the key point detection model shown includes:
[0138] Step a2, obtaining a first training data set, wherein the training data set includes sample images with human body position annotation information, wherein the human body position annotation information includes human body category labels, annotation boxes of human body key points, and specific coordinate positions of human body key points.
[0139] Step b2: extract sample images from the first training data set, input the extracted sample images into the second convolutional subnetwork, and obtain a second convolutional sample feature map.
[0140] Step c2: input the second convolution sample feature image into the first classification subnetwork to obtain a first classification human sample feature image including the human body position and a first classification loss value.
[0141] The network structure of the first classification subnetwork may be the same as the network structure of the classification subnetwork for auxiliary training in the above content, and the first classification subnetwork determines the first classification loss value using the same loss function as the classification subnetwork for auxiliary training.
[0142] Step d2: input the first classified human sample feature image into the first regression subnetwork to obtain the first regression human key point sample feature image and the first regression loss value.
[0143] Among them, the network structure of the first regression sub-network can be the same as the network structure of the regression sub-network of the auxiliary training in the above content, and the first regression sub-network uses the same loss function as the regression sub-network of the auxiliary training to determine the first regression loss value.
[0144] Step e2, inputting the first regressed human key point sample feature image into the second output subnetwork to obtain the coordinate positions of multiple human key points corresponding to the second output subnetwork and the second output loss value.
[0145] The network structure of the second output sub-network may be the same as the network structure of the first output sub-network in the above content, and the second output sub-network determines the second output loss value using the same loss function as the first output sub-network.
[0146] Step f2, determine the total loss value of the key point detection model based on the first classification loss value, the first regression loss value and the second output loss value, adjust the network parameters of the key point detection model, if the total loss value of the key point detection model meets the preset range, it means that the training of the key point detection model is completed; if the total loss value of the key point detection model does not meet the preset range, return to step b2, continue to train the key point detection model until the total loss value of the key point detection model meets the preset range, and output the trained key point detection model.
[0147] Among them, the key point detection model uses the same loss function as the key point detection model in the above content to determine the network parameters of the key point detection model.
[0148] The sub-networks included in the key point detection model are connected in series. By using the output of the previous sub-network as input, the loss value can be determined more accurately, thereby obtaining a better key point detection model and obtaining more accurate coordinate positions of multiple human key points.
[0149] In one possible embodiment, during training Figure 4 When the key point region detection model is used as shown, the classification sub-network can be used to assist in training the region detection model. Specifically, Figure 7 As shown, the auxiliary training classification sub-network and the second regression sub-network are connected in parallel. During the training process, the output of the third convolutional sub-network is used as the input of the auxiliary training classification sub-network and the second regression sub-network. Figure 4 The specific training process of the region detection model shown includes:
[0150] Step a3, obtaining a second training data set, wherein the second training data set includes sample images with human body key point position annotation information.
[0151] Step b3, extracting sample images from the second training data set, inputting the extracted sample images into the third convolutional subnetwork, and obtaining a third convolutional sample feature map.
[0152] Step c3, input the third convolution sample feature image into the classification subnetwork of auxiliary training to obtain an auxiliary classification human key point sample feature map containing the positions of human key points and the classification loss value of auxiliary training.
[0153] Specifically, the input of the auxiliary training classification subnetwork is the third convolution sample feature image, and the output is the auxiliary classification human key point sample feature map, that is, the classification feature map classified according to the category of human body parts. The auxiliary training classification subnetwork in the region detection model can be the same as the classification subnetwork in the key point detection model, and the auxiliary training classification subnetwork in the region detection model uses the same loss function as the classification subnetwork in the key point detection model to determine the classification loss value.
[0154] Step d3, input the third convolution sample feature image into the second regression subnetwork to obtain the second regression human key point sample feature image and the second regression loss value.
[0155] Specifically, the input of the second regression subnetwork is the third convolution sample feature image, and the output is the second regression human key point sample feature image, which is a feature map annotated with a human image region annotation box. The second regression subnetwork can be the same as the regression subnetwork in the key point detection model, and the second regression subnetwork uses the same loss function as the regression subnetwork in the key point detection model to determine the second regression loss value.
[0156] Step e3, determine the loss value of the first region detection model based on the classification loss value and the second regression loss value of the auxiliary training, adjust the network parameters of the first region detection model, if the total loss value of the region detection model meets the preset range, it means that the region detection model training is completed; if the total loss value of the region detection model does not meet the preset range, return to step b3, continue to train the region detection model until the total loss value of the region detection model meets the preset range, and output the trained region detection model.
[0157] The network parameters of the region detection model are adjusted by the loss value obtained by the loss function until the region detection model training is completed. The loss function of the region detection model is as follows:
[0158]
[0159] is the loss function of the auxiliary training classification sub-network, which is the same as the loss function used by the auxiliary training classification sub-network in the key point detection model. i The classification feature map output by the classification subnetwork for auxiliary training is classified according to the category of human body parts. is a sample image annotated with category labels of various parts of the human body, and k1 represents the weight of the loss function of the classification sub-network for auxiliary training.
[0160] is the loss function of the second regression sub-network, which is the same as the loss function used by the auxiliary training regression sub-network in the key point detection model. i It is the feature map output by the second regression sub-network with the labeled box of the human image area. is a sample image with a human body image region annotation box. k2 represents the weight of the loss function of the second regression sub-network; m is determined according to the size of the loss function of the second regression sub-network.
[0161] After the regional detection model training is completed, in the application process, it is only necessary to determine whether there is a human image area in the image outside the vehicle. Therefore, you can choose to turn off the auxiliary training classification sub-network, which can save data processing pressure on the vehicle terminal and save resources. If you want to get the output result of the auxiliary training classification sub-network, you can also choose to use the auxiliary training classification sub-network in the application process.
[0162] In one possible embodiment, during training Figure 5 When the area detection model is shown, Figure 5 As shown, the second classification sub-network and the third regression sub-network are connected sequentially.
[0163] in, Figure 5 The specific training process of the region detection model shown includes:
[0164] Step a4, obtaining a second training data set, the second training data set including a second sample image with human key point position annotation information. Step b4, extracting a sample image from the second training data set, inputting the extracted sample image into a fourth convolutional subnetwork, and obtaining a fourth convolutional sample feature map.
[0165] Step c4, input the fourth convolution sample feature image into the second classification subnetwork to obtain a second classification human sample feature image containing the human key point positions and a second classification loss value.
[0166] Specifically, the second classification subnetwork may be the same as the classification subnetwork auxiliary trained in the region detection model, and the second classification subnetwork determines the second classification loss value using the same loss function as the classification subnetwork auxiliary trained in the region detection model.
[0167] Step d4, inputting the second classified human sample feature image into the third regression subnetwork to obtain the third regression human key point sample feature image and the third regression loss value.
[0168] Specifically, the third regression subnetwork may be the same as the second regression subnetwork in the region detection model, and the third regression subnetwork determines the third regression loss value using the same loss function as the second regression subnetwork in the region detection model.
[0169] Based on the second classification loss value and the third regression loss value, the loss value of the region detection model is determined, and the network parameters of the region detection model are adjusted. If the total loss value of the region detection model meets the preset range, it means that the training of the region detection model is completed; if the total loss value of the region detection model does not meet the preset range, return to step b4, continue to train the region detection model until the total loss value of the region detection model meets the preset range, and output the trained region detection model.
[0170] Among them, the region detection model uses the same loss function as the region detection model in the above content to determine the network parameters of the region detection model.
[0171] Region Detection Model The network structure of the sub-networks included in the region detection model is connected in series. By using the output of the previous sub-network as input, the loss value can be determined more accurately, thereby obtaining a better region detection model, and the coordinate positions of multiple key points of the human body are more accurate.
[0172] In order to facilitate understanding of the embodiments of the present application, Figure 8 The specific implementation process of a security alarm method is shown as follows: Figure 8 As shown, the method may include the following steps:
[0173] Step S801: Acquire the interior image of the target vehicle.
[0174] Step S802: Determine the coordinate positions of multiple human key points in the interior image of the target vehicle using a key point detection model.
[0175] Step S803: Determine whether the preset area contains the coordinate position of at least one of the coordinate positions of multiple human body key points; if so, execute step S804; if not, return to execute step S801 to obtain the in-vehicle image at the next moment.
[0176] Step S804: Acquire the exterior image of the target vehicle.
[0177] Step S805: Determine whether the exterior image of the target vehicle contains a human body image region through the region detection model; if so, execute step S806; if not, return to execute step S801 to obtain the interior image of the vehicle at the next moment.
[0178] Step S806: Output security warning information.
[0179] In a possible embodiment, the vehicle terminal calls the DVR camera to capture the in-vehicle image, inputs the in-vehicle image into the trained key point detection model, obtains the coordinate positions of multiple human key points, and determines whether the coordinate position of at least one of the human key points is within the preset area, that is, determines whether the body parts of the person in the vehicle are close to the window or extended out of the window. If not, the in-vehicle camera continues to obtain the in-vehicle image, and continues to determine whether the coordinate positions of multiple human key points in the in-vehicle image obtained at the next moment are within the preset area. If so, the BSD camera is called to obtain the external image of the target vehicle, and the external image is input into the area detection model to detect whether there is a human image area in the external image, that is, to detect whether there is a body part of the person in the vehicle in the external image. If not, the in-vehicle camera continues to obtain the in-vehicle image, and the external camera is turned off and waits for calling; if it exists, it proves that someone has approached or extended his body out of the window, and a safety warning message is issued to warn the people in the vehicle to pay attention to safety and retract their bodies into the vehicle.
[0180] Based on the same concept, the embodiment of the present application also provides a safety alarm device, Fig. 9 A schematic diagram of the structure of a safety alarm device provided in an embodiment of the present application; Fig. 9 As shown, the safety warning device comprises:
[0181] The determination unit 901 is used to detect key points of the human body on the acquired in-vehicle image of the target vehicle to obtain the coordinate positions of multiple key points of the human body;
[0182] The judgment unit 902 is used to obtain an external image of the target vehicle if the coordinate position of at least one of the multiple human body key points is located within a set area; the set area is used to indicate the area corresponding to the window of the target vehicle in the internal image; the external image includes the window of the target vehicle;
[0183] The warning unit 903 is used to output safety warning information if the image outside the vehicle contains a human body image area.
[0184] In a possible implementation manner, the determining unit 901 is further configured to:
[0185] The key point detection model is used to detect key points of the human body in the in-vehicle image, and the coordinate positions of multiple key points of the human body output by the key point detection model are obtained; the key point detection model is trained based on sample images with human body position annotation information.
[0186] In a possible implementation manner, the determining unit 901 is further configured to:
[0187] Inputting the in-vehicle image into the first convolutional subnetwork, performing feature extraction on the in-vehicle image through the first convolutional subnetwork, and obtaining a first in-vehicle image feature map;
[0188] The first in-vehicle image feature map is input into the first output sub-network to obtain the coordinate positions of multiple human body key points output by the first output sub-network.
[0189] In a possible implementation manner, the determining unit 901 is further configured to:
[0190] Inputting the in-vehicle image into the second convolutional subnetwork, performing feature extraction on the in-vehicle image through the second convolutional subnetwork, and obtaining a second in-vehicle image feature map;
[0191] Inputting the second in-vehicle image feature map into the first classification subnetwork, and classifying the objects contained in the second in-vehicle image feature map through the first classification subnetwork to obtain a first classification feature map;
[0192] The first classification feature map is input into the first regression sub-network, and through the first regression sub-network, at the position where the human body area is marked in the first classification feature map, an in-vehicle marked feature map is obtained;
[0193] The in-vehicle annotated feature map is input into the second output sub-network to obtain the coordinate positions of multiple human body key points output by the second output sub-network based on the in-vehicle annotated feature map.
[0194] In a possible implementation manner, the determining unit 902 is further configured to:
[0195] If the coordinate position of at least one key point of the human body is located in the set area corresponding to the left window of the target vehicle, the exterior image of the left side of the target vehicle is obtained; if the coordinate position of at least one key point of the human body is located in the set area corresponding to the right window of the target vehicle, the exterior image of the right side of the target vehicle is obtained.
[0196] In a possible implementation, Fig.10 As shown, the above safety warning device may also include:
[0197] The detection unit 1001 is used to perform human body detection on the image outside the vehicle through a region detection model, and determine whether the image outside the vehicle contains a human body image region based on the output of the region detection model; the region detection model is trained based on a training image with human body key point position annotation information.
[0198] In a possible implementation manner, the detection unit 1001 is further configured to:
[0199] Inputting the vehicle exterior image into the third convolutional sub-network, performing feature extraction on the vehicle exterior image through the third convolutional sub-network, and obtaining a first vehicle exterior image feature map;
[0200] The first vehicle exterior image feature map is input into the second regression subnetwork. If an image of the vehicle exterior with marked human body regions is obtained, it is determined that the vehicle exterior image contains human body image regions. If an image without marked human body regions is obtained, it is determined that the vehicle exterior image does not contain human body image regions.
[0201] In a possible implementation manner, the detection unit 1001 is further configured to:
[0202] Inputting the vehicle exterior image into the fourth convolutional subnetwork, performing feature extraction on the vehicle exterior image through the fourth convolutional subnetwork, and obtaining a second vehicle exterior image feature map;
[0203] Inputting the second vehicle exterior image feature map into the second classification subnetwork, and classifying the objects contained in the second vehicle exterior image feature map through the second classification subnetwork to obtain a second classification feature map;
[0204] The second classification feature map is input into the third regression subnetwork. If an image outside the vehicle with a human body region marked is obtained, it is determined that the image outside the vehicle contains a human body image region; if an image without a human body region marked is obtained, it is determined that the image outside the vehicle does not contain a human body image region.
[0205] Corresponding to the above method embodiment, the embodiment of the present application also provides an electronic device. The electronic device can be a server, or a terminal device such as a mobile terminal or a computer, and the electronic device at least includes a memory for storing data and a processor for data processing. Among them, for the processor for data processing, when executing the processing, a microprocessor, CPU, GPU (Graphics Processing Unit), DSP or FPGA can be used. For the memory, the memory stores an operation instruction, which can be a computer executable code, and the various steps in the process of the security alarm method of the above embodiment of the present application are implemented by the operation instruction.
[0206] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application; Fig.11 As shown, in the embodiment of the present application, the electronic device 1100 includes: a processor 1110, a display 1120, a memory 1130, an input device 1160, a bus 1105 and a communication module 1140; the processor 1110, the memory 1130, the input device 1160, the display 1120 and the communication module 1140 are all connected via a bus 1150, and the bus 1150 is used to transmit data between the processor 1110, the memory 1130, the display 1120, the communication module 1140 and the input device 1160.
[0207] Among them, the memory 1130 can be used to store software programs and modules, and the processor 1110 executes various functional applications and data processing of the electronic device 1100 by running the software programs and modules stored in the memory 1130, such as the security alarm method provided in the embodiment of the present application. The memory 1130 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application application, etc.; the data storage area may store data created according to the use of the electronic device 1100, etc. In addition, the memory 1130 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0208] The processor 1110 is the control center of the electronic device 1100, and uses the bus 1150 and various interfaces and lines to connect various parts of the entire electronic device 1100, and executes various functions and processes data of the electronic device 1100 by running or executing software programs and / or modules stored in the memory 1130, and calling data stored in the memory 1130. Optionally, the processor 1100 may include one or more processing units, such as a CPU, a GPU (Graphics Processing Unit), a digital processing unit, etc.
[0209] The processor 1110 can also connect to the in-vehicle camera and the out-vehicle camera through the communication module 1140 by means of Bluetooth or the like, and send the acquired in-vehicle images and out-vehicle images to the electronic device.
[0210] The input device 1160 is mainly used to obtain the user's input operation. When the electronic device is different, the input device 1160 may also be different. For example, when the electronic device is a portable device such as a smart phone or a tablet computer, the input device 1160 may be a touch screen.
[0211] An embodiment of the present application further provides a computer storage medium, in which computer executable instructions are stored. The computer executable instructions are used to implement the security alarm method recorded in any embodiment of the present application.
[0212] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0213] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.
Claims
1. A security warning method, characterized in that: The method comprises: Through the key point detection model, the key points of the human body are detected on the acquired in-vehicle image of the target vehicle to obtain the coordinate positions of multiple key points of the human body; If the coordinate position of at least one of the plurality of human body key points is located within a set area, then acquiring an exterior image of the target vehicle; the set area is used to indicate an area corresponding to a window of the target vehicle in the interior image; and the exterior image includes the window of the target vehicle; If the image outside the vehicle contains a human body image area, outputting a safety warning message; The key point detection model includes a first convolution subnetwork and a first output subnetwork; the key point detection model is used to perform human key point detection on the acquired in-vehicle image of the target vehicle to obtain the coordinate positions of multiple human key points, including: Inputting the in-vehicle image into the first convolutional subnetwork, performing feature extraction on the in-vehicle image through the first convolutional subnetwork, and obtaining a first in-vehicle image feature map; Inputting the first in-vehicle image feature map into the first output sub-network to obtain the coordinate positions of the plurality of human body key points output by the first output sub-network; The key point detection model is obtained by training based on an auxiliary training regression subnetwork and an auxiliary training classification subnetwork; Before performing human key point detection on the acquired in-vehicle image of the target vehicle through the key point detection model, closing the auxiliary training regression subnetwork and the auxiliary training classification subnetwork; If the coordinate position of at least one of the plurality of human body key points is located within a set area, obtaining an external image of the target vehicle includes: If the coordinate position of the at least one human body key point is located in the set area corresponding to the left window of the target vehicle, then obtaining the vehicle exterior image of the left side of the target vehicle; If the coordinate position of the at least one human body key point is located in the set area corresponding to the right side window of the target vehicle, an exterior image of the right side of the target vehicle is acquired.
2. The method according to claim 1, characterized in that The method of performing human body key point detection on the acquired in-vehicle image of the target vehicle to obtain coordinate positions of multiple human body key points includes: The key point detection model is used to detect the key points of the human body in the in-vehicle image, and the coordinate positions of multiple key points of the human body output by the key point detection model are obtained; the key point detection model is trained based on sample images with human body position annotation information.
3. The method according to claim 1, characterized in that If the image outside the vehicle contains a human body image area, before outputting the safety warning information, the method includes: Human body detection is performed on the image outside the vehicle through a region detection model, and whether the image outside the vehicle contains a human body image region is determined based on the output of the region detection model; the region detection model is trained based on a training image with human body key point position annotation information.
4. The method according to claim 3, characterized in that The region detection model includes a third convolution subnetwork and a second regression subnetwork; the method of performing human body detection on the vehicle exterior image through the region detection model, and determining whether the vehicle exterior image contains a human body image region according to the output of the region detection model, includes: Inputting the vehicle exterior image into the third convolutional subnetwork, performing feature extraction on the vehicle exterior image through the third convolutional subnetwork, and obtaining a first vehicle exterior image feature map; The first vehicle exterior image feature map is input into the second regression subnetwork. If an image of the vehicle exterior with marked human body regions is obtained, it is determined that the image of the vehicle exterior contains human body image regions. If an image of the vehicle exterior without marked human body regions is obtained, it is determined that the image of the vehicle exterior does not contain human body image regions.
5. The method according to claim 3, characterized in that: The region detection model includes a fourth convolution subnetwork, a second classification subnetwork and a third regression subnetwork; the method of performing human body detection on the vehicle exterior image through the region detection model, and determining whether the vehicle exterior image contains a human body image region according to the output of the region detection model, includes: Inputting the vehicle exterior image into the fourth convolutional subnetwork, performing feature extraction on the vehicle exterior image through the fourth convolutional subnetwork, and obtaining a second vehicle exterior image feature map; Inputting the second vehicle exterior image feature map into the second classification subnetwork, and classifying objects contained in the second vehicle exterior image feature map through the second classification subnetwork to obtain a second classification feature map; The second classification feature map is input into the third regression subnetwork. If an image outside the vehicle with a human body area marked is obtained, it is determined that the image outside the vehicle contains a human body image area; if an image without a human body area marked is obtained, it is determined that the image outside the vehicle does not contain a human body image area.
6. A safety warning device, characterized in that: The device comprises: A determination unit, configured to perform human key point detection on the acquired in-vehicle image of the target vehicle through a key point detection model to obtain coordinate positions of multiple human key points; A judgment unit, configured to obtain an exterior image of the target vehicle if the coordinate position of at least one of the plurality of human body key points is within a set area; the set area is used to indicate an area corresponding to a window of the target vehicle in the interior image; and the exterior image includes the window of the target vehicle; The key point detection model includes a first convolution subnetwork and a first output subnetwork; the key point detection model is used to perform human key point detection on the acquired in-vehicle image of the target vehicle to obtain the coordinate positions of multiple human key points, including: Inputting the in-vehicle image into the first convolutional subnetwork, performing feature extraction on the in-vehicle image through the first convolutional subnetwork, and obtaining a first in-vehicle image feature map; Inputting the first in-vehicle image feature map into the first output sub-network to obtain the coordinate positions of the plurality of human body key points output by the first output sub-network; The key point detection model is obtained by training based on an auxiliary training regression subnetwork and an auxiliary training classification subnetwork; Before performing human key point detection on the acquired in-vehicle image of the target vehicle through the key point detection model, closing the auxiliary training regression subnetwork and the auxiliary training classification subnetwork; An alarm unit, configured to output a safety alarm message if the image outside the vehicle contains a human body image area; The judgment unit is specifically configured to obtain an exterior image of the left side of the target vehicle if the coordinate position of the at least one key point of the human body is within a set area corresponding to the left window of the target vehicle; If the coordinate position of the at least one human body key point is located in the set area corresponding to the right side window of the target vehicle, an exterior image of the right side of the target vehicle is acquired.
7. The device according to claim 6, characterized in that The device comprises: The judgment unit is also used to obtain the exterior image of the left side of the target vehicle if the coordinate position of at least one key point of the human body is within the set area corresponding to the left window of the target vehicle; and to obtain the exterior image of the right side of the target vehicle if the coordinate position of at least one key point of the human body is within the set area corresponding to the right window of the target vehicle.
8. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the method according to any one of claims 1 to 5 is implemented.
9. A computer-readable storage medium having a computer program stored therein, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.
