Method and device for identifying dangerous acts, electronic device, and storage medium
The method and device in the vehicle interior analyze images to identify dangerous acts like extending body parts out of the window, issuing warnings or controlling vehicle systems to prevent accidents, thereby enhancing safety.
Patent Information
- Application Number
- JP2023544368
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-06-30
- Filing Date
- 2021-10-28
- Publication Date
- 2024-12-04
- Estimated Expiration
- 2041-10-28
AI Technical Summary
Current vehicle safety systems fail to accurately identify dangerous acts, such as passengers putting body parts out of the window, which can lead to accidents.
A method and device that acquire an image of the vehicle interior, detect passengers, and analyze their position to identify dangerous acts like extending body parts out of the window, using image processing techniques to determine the probability of such actions and issue warnings or control vehicle systems to prevent them.
Accurately identifies dangerous acts and issues warnings or controls vehicle systems to prevent accidents, enhancing passenger safety by preventing body parts from being exposed outside the vehicle.
Smart Images

Figure 0007594677000001 
Figure 0007594677000002 
Figure 0007594677000003
Abstract
Description
[Technical field]
[0001] This application claims priority to a Chinese patent application filed with the China Patent Office on June 30, 2021, bearing application number 202110735201.6 and entitled "Method and apparatus for identifying dangerous activities, electronic devices and storage media," the entire contents of which are incorporated herein by reference.
[0002] The present disclosure relates to the technical field of intelligent vehicle cabins, and in particular to a method and device for identifying dangerous activities, an electronic device, and a storage medium. [Background technology]
[0003] At present, the car electronics industry is developing rapidly, providing people with a convenient and comfortable cabin environment for riding. Cabin intelligence is an important development direction of the current automobile industry. Cabin intelligence includes intelligence in such aspects as multi-mode interaction, personalized services, and safety detection. In terms of safety detection, intelligent cars focus on providing a safe cabin environment for passengers. Identifying dangerous behaviors of people in the cabin is very important to the safety of passengers in the cabin. Summary of the Invention
[0004] The present disclosure proposes an invention relating to identifying risky activities.
[0005] According to one aspect of the present disclosure, there is provided a method for identifying dangerous behavior, the method including: acquiring an image of a vehicle interior; detecting an occupant in the vehicle interior based on the image, and obtaining a detection result of the occupant in the vehicle interior; and, if the detection result of the occupant indicates that a occupant has been detected, identifying a dangerous behavior based on position information of the occupant contained in the image and the detection result of the occupant, and obtaining an identification result of the dangerous behavior of the occupant, wherein the dangerous behavior is an act of sticking a predetermined body part out of the vehicle window. Effect of the Invention
[0006] In one possible embodiment, when the occupant detection result indicates that a occupant has been detected, identifying a dangerous behavior based on the image and location information of the occupant contained in the occupant detection result, and obtaining an identification result of the dangerous behavior of the occupant includes identifying a dangerous behavior in the vehicle cabin based on the image and obtaining estimated information of the dangerous behavior in the vehicle cabin, and when the occupant detection result indicates that a occupant has been detected, obtaining an identification result of the dangerous behavior of the occupant based on the location information of the occupant included in the occupant detection result and the estimated information of the dangerous behavior.
[0007] In one possible embodiment, the occupant's position information includes coordinates of an estimated body center point in the image, and the risky behavior estimation information includes a probability that the occupant's behavior corresponds to each of N (N is an integer equal to or greater than 1) types of risky behaviors that are preset, As described above, when the occupant detection result indicates that a occupant has been detected, obtaining an identification result of the dangerous behavior of the occupant based on the occupant's position information and the estimated information of the dangerous behavior includes: determining that the occupant has been detected when the occupant detection result indicates the presence of an estimated human body center point; obtaining from the estimated information of the dangerous behavior a probability that the occupant's behavior corresponding to the estimated human body center point falls under each of N predetermined types of dangerous behavior based on the coordinates of the estimated human body center point included in the occupant detection result; and obtaining an identification result of the dangerous behavior of the occupant corresponding to the estimated human body center point based on the probability that the occupant's behavior corresponding to the estimated human body center point falls under each of the N predetermined types of dangerous behavior.
[0008] In one possible embodiment, the occupant position information comprises coordinates of an estimated center of the human body in the image; The above-mentioned detecting an occupant in the vehicle cabin based on the image and obtaining a detection result of the occupant in the vehicle cabin includes estimating a probability that a pixel in the image corresponds to a human body center point based on a first feature map corresponding to the image, and determining coordinates of an estimated human body center point in the image based on the probability that a pixel in the image corresponds to a human body center point.
[0009] In one possible embodiment, estimating the probability that a pixel in the image corresponds to a body center point based on a first feature map corresponding to the image includes determining, based on a first feature map corresponding to the image, coordinates of a first candidate point for a body center point in the image and a probability that the first candidate point corresponds to the body center point, and determining, based on the probability that a pixel in the image corresponds to a body center point, coordinates of an estimated body center point in the image includes determining coordinates of an estimated body center point in the image based on the coordinates of the first candidate point and the probability that the first candidate point corresponds to the body center point.
[0010] In one possible embodiment, determining the coordinates of a first candidate point for a body center point in the image and the probability that the first candidate point corresponds to the body center point based on a first feature map corresponding to the image includes: performing a convolution operation on the first feature map corresponding to the image to obtain a second feature map corresponding to the image; and performing a max pooling operation based on the second feature map to obtain the coordinates of a first candidate point for a body center point in the image and the probability that the first candidate point corresponds to the body center point.
[0011] In one possible embodiment, the step of performing a max pooling operation based on the second feature map to obtain a coordinate of a first candidate point for a body center point in the image and a probability that the first candidate point corresponds to the body center point includes performing an activation process on the second feature map to obtain a third feature map corresponding to the image, and performing a max pooling operation on the third feature map to obtain a coordinate of a first candidate point for a body center point in the image and a probability that the first candidate point corresponds to the body center point.
[0012] In one possible embodiment, performing a max pooling operation on the third feature map to obtain a coordinate of a first candidate point for a body center point in the image and a probability that the first candidate point corresponds to the body center point comprises performing an overlap max pooling operation on the third feature map to obtain a coordinate of a first candidate point for a body center point in the image and a probability that the first candidate point corresponds to the body center point; The step of determining the coordinates of an estimated body center point in the image based on the coordinates of the first candidate point and the probability that the first candidate point corresponds to the body center point includes: merging the first candidate points having the same coordinates to obtain the coordinates of a second candidate point and the probability that the second candidate point corresponds to the body center point; and determining the coordinates of the estimated body center point in the image based on the coordinates of the second candidate point and the probability that the second candidate point corresponds to the body center point.
[0013] In one possible embodiment, identifying a dangerous activity in the vehicle cabin based on the image and obtaining estimated information of the dangerous activity in the vehicle cabin includes performing a convolution operation and a full combination operation on a first feature map corresponding to the image, followed by a classification operation to obtain estimated information of the dangerous activity in the vehicle cabin.
[0014] In one possible embodiment, the method for identifying dangerous behavior further includes, when the occupant detection result indicates that a occupant has been detected in the front seat area of the vehicle cabin, identifying dangerous behavior based on the image and the occupant's position information contained in the occupant detection result, and obtaining an identification result of the occupant's dangerous behavior.
[0015] In one possible embodiment, the predefined body parts include at least one of the following: hands, arms, heads, feet, and thighs.
[0016] In one possible embodiment, the method for identifying dangerous behavior further includes, after obtaining the identification result of the dangerous behavior of the occupant as described above, issuing warning information if the identification result of the dangerous behavior indicates that the occupant's behavior includes any one of the predetermined dangerous behaviors.
[0017] In one possible embodiment, issuing the warning information includes at least one of controlling an in-vehicle voice dialogue device to issue voice warning information, issuing a command to control the raising and lowering of a vehicle window corresponding to a passenger engaging in the pre-defined dangerous behavior, and issuing a command to turn on a hazard warning light.
[0018] According to one aspect of the present disclosure, there is provided a dangerous behavior identification device that includes an acquisition module for acquiring an image of a vehicle interior, a passenger detection module for detecting an occupant in the vehicle interior based on the image and obtaining a detection result of the occupant in the vehicle interior, and a first dangerous behavior identification module for, when the occupant detection result indicates that a passenger has been detected, identifying a dangerous behavior based on the image and position information of the occupant contained in the occupant detection result and obtaining an identification result of the dangerous behavior of the occupant, wherein the dangerous behavior is an act of sticking a predetermined body part out of the vehicle window.
[0019] In one possible embodiment, the first dangerous activity identification module identifies dangerous activities in the vehicle cabin based on the image and obtains estimated information of the dangerous activities in the vehicle cabin, and when the occupant detection result indicates that a occupant has been detected, obtains an identification result of the occupant's dangerous activities based on the occupant's location information included in the occupant detection result and the estimated information of the dangerous activity.
[0020] In one possible embodiment, the occupant's position information includes coordinates of an estimated body center point in the image, and the risky behavior estimation information includes a probability that the occupant's behavior corresponds to each of N (N is an integer equal to or greater than 1) types of risky behaviors that are preset, When the occupant detection result indicates the presence of an estimated human body center point, the first dangerous behavior identification module determines that an occupant has been detected, and obtains from the dangerous behavior estimation information a probability that the occupant's behavior corresponding to the estimated human body center point falls under each of N predetermined dangerous behaviors based on the coordinates of the estimated human body center point included in the occupant detection result, and obtains an identification result of the dangerous behavior of the occupant corresponding to the estimated human body center point based on the probability that the occupant's behavior corresponding to the estimated human body center point falls under each of N predetermined dangerous behaviors.
[0021] In one possible embodiment, the occupant position information includes coordinates of an estimated body center point in the image, and the occupant detection module estimates a probability that a pixel in the image corresponds to the body center point based on a first feature map corresponding to the image, and determines the coordinates of the estimated body center point in the image based on the probability that a pixel in the image corresponds to the body center point.
[0022] In one possible embodiment, the occupant detection module comprises: Based on a first feature map corresponding to the image, coordinates of a first candidate point for the human body center point in the image and a probability that the first candidate point corresponds to the human body center point are determined, and based on the coordinates of the first candidate point and the probability that the first candidate point corresponds to the human body center point, coordinates of an estimated human body center point in the image are determined.
[0023] In one possible embodiment, the occupant detection module performs a convolution operation on a first feature map corresponding to the image to obtain a second feature map corresponding to the image, and performs a max pooling operation based on the second feature map to obtain coordinates of a first candidate point for a body center point in the image and a probability that the first candidate point corresponds to the body center point.
[0024] In one possible embodiment, the occupant detection module performs an activation process on the second feature map to obtain a third feature map corresponding to the image, and performs a max pooling operation on the third feature map to obtain the coordinates of a first candidate point for a body center point in the image and the probability that the first candidate point corresponds to the body center point.
[0025] In one possible embodiment, the occupant detection module performs an overlap maximum pooling operation on the third feature map to obtain coordinates of a first candidate point for a body center point in the image and a probability that the first candidate point corresponds to the body center point, merges the first candidate points with the same coordinates, obtains coordinates of a second candidate point and a probability that the second candidate point corresponds to the body center point, and determines coordinates of an estimated body center point in the image based on the coordinates of the second candidate point and the probability that the second candidate point corresponds to the body center point.
[0026] In one possible embodiment, the first risky activity identification module performs a convolution operation and a full combination operation on a first feature map corresponding to the image, followed by a classification operation to obtain estimated information of risky activities in the vehicle cabin.
[0027] In one possible embodiment, the dangerous behavior identification device further includes a second dangerous behavior identification module that, when the occupant detection result indicates that a occupant has been detected in the front seat area of the vehicle cabin, identifies dangerous behavior based on the image and the occupant's position information contained in the occupant detection result, and obtains an identification result of the occupant's dangerous behavior.
[0028] In one possible embodiment, the predefined body parts include at least one of the following: hands, arms, heads, feet, and thighs.
[0029] In one possible embodiment, the dangerous behavior identification device further includes a warning module that issues warning information when the dangerous behavior identification result indicates that the occupant's behavior includes any one of the predetermined dangerous behaviors.
[0030] In one possible embodiment, the warning module is used for at least one of controlling an in-vehicle voice dialogue device to issue voice warning information, issuing a command to control the raising and lowering of a vehicle window corresponding to a passenger engaging in the pre-defined dangerous behavior, and issuing a command to turn on a hazard warning light.
[0031] According to one aspect of the present disclosure, there is provided an electronic device including one or more processors and a memory for storing executable instructions, the one or more processors configured to execute the above-mentioned method for identifying dangerous activities by invoking the executable instructions stored in the memory.
[0032] According to one aspect of the present disclosure, there is provided a computer-readable storage medium having computer program instructions stored thereon, the computer-readable storage medium realizing the above-described method for identifying risky activities when executed by a processor.
[0033] According to one aspect of the present disclosure, there is provided a computer program product including computer readable code, or a non-volatile computer readable storage medium on which computer readable code is stored, which, when executed within an electronic device, causes a processor of the electronic device to execute the above-mentioned method for identifying risky activities.
[0034] In an embodiment of the present disclosure, an image of a vehicle interior is acquired, an occupant in the vehicle interior is detected based on the image to obtain a detection result of the occupant in the vehicle interior, and when the detection result of the occupant indicates that the occupant has been detected, a dangerous behavior is identified based on the image and position information of the occupant included in the detection result of the occupant, and an identification result of the dangerous behavior of the occupant is obtained, the dangerous behavior being an act of sticking a predetermined body part out of a vehicle window. This makes it possible to accurately identify the act of the occupant sticking a predetermined body part out of a vehicle window based on the position of the occupant, thereby improving the safety of the occupant in the vehicle interior.
[0035] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.
[0036] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments taken in conjunction with the accompanying drawings. [Brief description of the drawings]
[0037] The drawings, which are included as part of the specification, illustrate embodiments of the present disclosure and, together with the specification, explain the technical means of the present disclosure. [Figure 1] 1 is a flowchart of a method for identifying risky behavior according to an embodiment of the present disclosure. [Diagram 2] 1 is a schematic diagram showing a passenger sticking his / her head out of a vehicle window in a vehicle interior image in a method for identifying dangerous behavior relating to an embodiment of the present disclosure. [Diagram 3] 1 is a schematic diagram showing an occupant in a vehicle interior image sticking his / her hand or arm out of a vehicle window in a method for identifying dangerous behavior relating to an embodiment of the present disclosure. [Figure 4] 1 is a schematic diagram of an application scenario of the method for identifying risky behavior according to the present disclosure. [Diagram 5] FIG. 2 is a block diagram of a device for identifying risky activities according to an embodiment of the present disclosure. [Figure 6] FIG. 8 is a block diagram of an electronic device 800 according to an embodiment of the present disclosure. [Figure 7] FIG. 19 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0038] Various exemplary embodiments, features, and aspects of the present disclosure will now be described in detail with reference to the drawings, in which like reference numerals indicate elements of the same or similar function. While illustrating various aspects of the embodiments in the drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0039] The word "exemplary" herein means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or superior over other embodiments.
[0040] In this specification, the term "and / or" merely describes a related relationship between related objects and indicates that three relationships can exist, for example, A and / or B can indicate three cases: only A exists, A and B exist simultaneously, and only B exists. In addition, in this specification, the term "at least one" indicates any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, and C can indicate including any one or more elements selected from a set consisting of A, B, and C.
[0041] In addition, in order to better explain the present disclosure, many specific details are shown in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can be similarly implemented without any specific details. In some embodiments, in order to emphasize the gist of the present disclosure, detailed descriptions are not provided for methods, means, elements and circuits that are well known to those skilled in the art.
[0042] While the vehicle is moving, the driver or other passengers may take the risk of sticking their hands, head or other body parts outside the vehicle window, which could result in a serious accident.
[0043] An embodiment of the present disclosure provides a method and device for identifying a dangerous behavior, an electronic device, and a storage medium, which acquire an image of a vehicle interior, detect an occupant in the vehicle interior based on the image, obtain a detection result of the occupant in the vehicle interior, and if the detection result of the occupant indicates that a occupant has been detected, identify a dangerous behavior based on position information of the occupant contained in the image and the detection result of the occupant, and obtain an identification result of the dangerous behavior of the occupant, the dangerous behavior indicating an action of the occupant sticking a predetermined body part out of a vehicle window. This makes it possible to accurately identify the action of the occupant sticking a predetermined body part out of a vehicle window based on the position of the occupant, thereby improving the safety of the occupant in the vehicle interior.
[0044] Hereinafter, a method for identifying risky actions according to an embodiment of the present disclosure will be described in detail with reference to the drawings.
[0045] FIG. 1 is a flowchart of a method for identifying a risky act according to an embodiment of the present disclosure. In one possible embodiment, the method for identifying a risky act may be executed by a terminal device or a server or other processing device. Here, the terminal device may be an in-vehicle device, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, or a wearable device, etc. The in-vehicle device may be an in-vehicle infotainment, a domain controller or a processor in a vehicle cabin, or a device host for performing processing operations on data such as images in a driver monitoring system (DMS) or an occupant monitoring system (OMS). In some possible embodiments, the method for identifying a risky act is realized by a processor calling a computer-readable instruction stored in a memory. As shown in FIG. 1, the method for identifying a risky act includes steps S11 to S13.
[0046] In step S11, an image of the vehicle interior is acquired.
[0047] In step S12, an occupant in the vehicle compartment is detected based on the image, and a detection result of the occupant in the vehicle compartment is obtained.
[0048] In step S13, if the passenger detection result indicates that a passenger has been detected, a dangerous behavior is identified based on the image and the passenger's position information contained in the passenger detection result, and an identification result of the dangerous behavior of the passenger is obtained, where the dangerous behavior is an act of sticking a predetermined body part out of the vehicle window.
[0049] The embodiments of the present disclosure may be applied to any type of vehicle, such as a passenger car, a taxi, a ride-hailing car, or a shared car. The embodiments of the present disclosure are not limited to the type of vehicle, and may be, for example, a compact car or an SUV (Sport Utility Vehicle). In the embodiments of the present disclosure, an image of the camera cabin can be acquired by an on-board camera. The on-board camera may be any camera installed in a vehicle. The number of on-board cameras may be one or more. The on-board camera may be installed inside and / or outside the cabin. The type of on-board camera may be a DMS camera, an OMS camera, a normal camera, or the like.
[0050] The image of the vehicle interior may be an image of the vehicle interior environment captured by a camera installed inside and / or outside the vehicle interior, such as a DMS camera, an OMS camera, or a normal camera. This image includes at least image information of the seating area of the passengers and the window area in the vehicle interior. In other words, at least a part of the seating area of the passengers and at least a part of the window area must be included in the viewing angle range of the camera.
[0051] In an embodiment of the present disclosure, a human body and / or a face in the vehicle cabin can be detected based on the image, and a detection result of a human body and / or a face in the vehicle cabin can be obtained. A detection result of an occupant in the vehicle cabin can be obtained based on the detection result of a human body and / or a face in the vehicle cabin. For example, the detection result of a human body and / or a face in the vehicle cabin can be used as the detection result of an occupant in the vehicle cabin. As another example, the detection result of a human body and / or a face in the vehicle cabin can be processed to obtain a detection result of an occupant in the vehicle cabin.
[0052] In an embodiment of the present disclosure, when a passenger is detected, the passenger detection result may include the passenger's location information. For example, when one passenger is detected, the passenger detection result may include the passenger's location information, and when multiple passengers are detected, the passenger detection result may include the location information of each detected passenger.
[0053] In the embodiment of the present disclosure, the position information of the passenger may be indicated by the coordinates of any one or more points of the passenger, and / or the position information of the passenger may be indicated by the position information of the bounding box of the passenger. In one possible embodiment, the position information of the passenger may include the coordinates of the estimated body center point of the passenger. Here, the estimated body center point may indicate a point estimated as the body center point. The body center point may be a point representing the position of the human body, and the number of body center points of any human body may be one. For example, the body center point may be a pixel where the center of gravity of the human body is located, or may be a pixel where any one of the key points of the human body is located. In another possible embodiment, the position information of the passenger may include the coordinates of the estimated body center point of the passenger and the size of the body frame, and the size of the body frame may include the length and width of the body frame. In this embodiment, any one of the estimated body center points may be the geometric center of the body frame corresponding to the estimated body center point. In another possible embodiment, the position information of the passenger may include the position information of the body frame. For example, the position information of the body frame may include the coordinates of any one vertex of the body frame and the size of the body frame. As another example, the position information of the body frame may include the coordinates of four vertices of the body frame.
[0054] In one possible embodiment, the occupant position information includes coordinates of an estimated body center point in the image, and detecting an occupant in the vehicle cabin based on the image and obtaining a detection result of the occupant in the vehicle cabin includes estimating a probability that a pixel in the image corresponds to a body center point based on a first feature map corresponding to the image, and determining the coordinates of the estimated body center point in the image based on the probability that a pixel in the image corresponds to a body center point.
[0055] In this embodiment, a human body in the vehicle interior can be detected based on the image, a detection result of the human body in the vehicle interior can be obtained, and a detection result of an occupant in the vehicle interior can be obtained based on the detection result of the human body in the vehicle interior. For example, a human body in the vehicle interior can be detected based on the image, and coordinates of an estimated human body center point in the image can be obtained. As another example, a human body in the vehicle interior can be detected based on the image, and coordinates of an estimated human body center point in the image and a size of a human body frame to which the estimated human body center point belongs can be obtained.
[0056] As an example of this embodiment, the image is input to a backbone network, and feature extraction is performed on the image through the backbone network to obtain a first feature map corresponding to the image. The backbone network may use a network structure such as ResNet, MobileNet, etc., and is not limited thereto. As another example of this embodiment, feature extraction is performed on the first image using a pre-designed first function to obtain a first feature map corresponding to the image.
[0057] As an example of this embodiment, the first feature map can be input to a first estimation sub-network, and the first estimation sub-network can estimate the probability that a pixel in the image corresponds to a body center point. As another example of this embodiment, the first feature map can be processed using a pre-designed second function to obtain the probability that a pixel in the image corresponds to a body center point.
[0058] As an example of this embodiment, if the probability that any one pixel in the image corresponds to the human body center point is greater than a first threshold, the pixel can be determined as the human body center estimated point. That is, the coordinates of the pixel can be determined as the coordinates of the human body center estimated point. For example, the first threshold may be 0.5. Of course, those skilled in the art can flexibly set the magnitude of the first threshold according to the needs of actual application scenarios, and the first threshold is not limited here. As another example of this embodiment, if the probability that any one pixel in the image corresponds to the human body center point is greater than the first threshold, and the pixel is one of M pixels having the highest probability of corresponding to the human body center point in the image, the pixel can be determined as the human body center estimated point. That is, the coordinates of the pixel can be determined as the coordinates of the human body center estimated point. Here, M is a maximum number of human body center estimated points set in advance, and M is 1 or more.
[0059] Identifying risky actions based on the coordinates of the estimated center point of the human body in the image obtained by this embodiment is advantageous in improving the accuracy of identifying risky actions.
[0060] As an example of this embodiment, estimating the probability that a pixel in the image corresponds to a human body center point based on a first feature map corresponding to the image includes determining the coordinates of a first candidate point of the human body center point in the image and the probability that the first candidate point corresponds to the human body center point based on a first feature map corresponding to the image, and determining the coordinates of the human body center estimated point in the image based on the probability that a pixel in the image corresponds to the human body center point includes determining the coordinates of the human body center estimated point in the image based on the coordinates of the first candidate point and the probability that the first candidate point corresponds to the human body center point. In this example, the probability that a pixel in the image corresponds to a human body center point can be estimated based on a first feature map corresponding to the image, and a first candidate point of the human body center point in the image can be screened from the pixels of the image. The first candidate point can indicate a pixel screened from the image that has a relatively high probability of corresponding to the human body center point. The number of first candidate points may be multiple. In one example, each pixel in an image may be set as a first candidate point, or in another example, a human body may be detected based on a first feature map, and pixels within the detected human body outline may be set as first candidate points.
[0061] In one example, if the probability that any one of the first candidate points corresponds to the human body center point is greater than a first threshold, the first candidate point can be determined as the human body center estimated point. That is, the coordinates of the first candidate point can be determined as the coordinates of the human body center estimated point. In another example, if the probability that any one of the first candidate points corresponds to the human body center point is greater than a first threshold, and the first candidate point is one of the M first candidate points having the highest probability of corresponding to the human body center point in the image, the first candidate point can be determined as the human body center estimated point. That is, the coordinates of the first candidate point can be determined as the coordinates of the human body center estimated point. In this example, based on a first feature map corresponding to the image, the coordinates of the first candidate point of the human body center point in the image and the probability that the first candidate point corresponds to the human body center point are determined, and the coordinates of the human body center estimated point in the image are determined based on the coordinates of the first candidate point and the probability that the first candidate point corresponds to the human body center point. This can improve the accuracy of the determined estimated body center point.
[0062] In one example, the determining of the coordinates of the first candidate point of the human body center point in the image and the probability that the first candidate point corresponds to the human body center point based on the first feature map corresponding to the image includes performing a convolution operation on the first feature map corresponding to the image to obtain a second feature map corresponding to the image, and performing a max pooling operation based on the second feature map to obtain the coordinates of the first candidate point of the human body center point in the image and the probability that the first candidate point corresponds to the human body center point. In this example, the convolution operation can be performed once or more times on the first feature map to obtain the second feature map corresponding to the image. The max pooling operation can be performed on the second feature map to obtain the coordinates of the first candidate point of the human body center point in the image and the probability that the first candidate point corresponds to the human body center point, or the max pooling operation can be performed after processing the second feature map to obtain the coordinates of the first candidate point of the human body center point in the image and the probability that the first candidate point corresponds to the human body center point. In this example, a convolution operation is performed on a first feature map corresponding to the image to extract deeper features of the image, and the resulting second feature map can more accurately indicate the position information of the human body in the image. A max pooling operation is performed based on the second feature map to accurately screen the first candidate point from the pixels of the image.
[0063] In one example, performing a maximum pooling operation based on the second feature map to obtain the coordinates of the first candidate point of the human body center point in the image and the probability that the first candidate point corresponds to the human body center point includes performing an activation process on the second feature map to obtain a third feature map corresponding to the image, and performing a maximum pooling operation on the third feature map to obtain the coordinates of the first candidate point of the human body center point in the image and the probability that the first candidate point corresponds to the human body center point. For example, a sigmoid process can be performed on the second feature map to convert pixel values of the second feature map into numerical values between 0 and 1. Of course, other activation functions can be used to perform the activation process on the second feature map, and are not limited here. In this example, by performing an activation process on the second feature map, pixel values of the second feature map can be converted into probability values, so that the pixel can be used to indicate the probability that the pixel corresponds to the human body center point.
[0064] For example, performing a maximum pooling operation on the third feature map to obtain the coordinates of a first candidate point of the human body center point in the image and the probability that the first candidate point corresponds to the human body center point includes performing an overlap maximum pooling operation on the third feature map to obtain the coordinates of a first candidate point of the human body center point in the image and the probability that the first candidate point corresponds to the human body center point, and determining the coordinates of the estimated human body center point in the image based on the coordinates of the first candidate point and the probability that the first candidate point corresponds to the human body center point includes merging the first candidate points having the same coordinates, obtaining the coordinates of a second candidate point and the probability that the second candidate point corresponds to the human body center point, and determining the coordinates of the estimated human body center point in the image based on the coordinates of the second candidate point and the probability that the second candidate point corresponds to the human body center point. For example, the size of the pooling window is P×P, the step size is Q, and P>Q (P and Q are both integers equal to or greater than 1, for example, P=3, Q=1). In this example, the first candidate points having the same coordinates can be merged to obtain the second candidate point. That is, the second candidate point can indicate the result of merging the first candidate points. The number of the second candidate points is equal to or less than the number of the first candidate points, the number of the second candidate points is 1 or more, and the second candidate points do not include candidate points having the same coordinates. In this example, the third feature map is subjected to an overlap maximum pooling operation to improve the accuracy of detecting the human body center point. The first candidate points having the same coordinates are merged to obtain the coordinates of the second candidate point and the probability that the second candidate point corresponds to the human body center point, and the coordinates of the estimated human body center point in the image are determined based on the coordinates of the second candidate point and the probability that the second candidate point corresponds to the human body center point. This can further improve the accuracy of detecting the human body center point and improve the efficiency of identifying subsequent risky actions.
[0065] In another example, the above-mentioned performing a max pooling operation based on the second feature map to obtain a coordinate of a first candidate point for a body center point in the image and a probability that the first candidate point corresponds to the body center point includes performing a max pooling operation on the second feature map to obtain a fourth feature map, and performing an activation process on the fourth feature map to obtain a coordinate of a first candidate point for a body center point in the image and a probability that the first candidate point corresponds to the body center point.
[0066] In an embodiment of the present disclosure, when the occupant detection result indicates that one occupant has been detected, dangerous behavior can be identified based on the image and the location information of the occupant contained in the occupant detection result, and an identification result of the dangerous behavior of the occupant can be obtained; when the occupant detection result indicates that multiple occupants have been detected, dangerous behavior can be identified based on the image and the location information of each occupant contained in the occupant detection result, and an identification result of the dangerous behavior corresponding to each occupant can be obtained.
[0067] In one possible embodiment, the predefined body part includes at least one of a hand, an arm, a head, a foot, and a crotch. FIG. 2 shows a schematic diagram of an occupant in an image of a vehicle interior sticking his / her head out of a vehicle window in a method for identifying a dangerous behavior according to an embodiment of the present disclosure. FIG. 3 shows a schematic diagram of an occupant in an image of a vehicle interior sticking his / her hand or arm out of a vehicle window in a method for identifying a dangerous behavior according to an embodiment of the present disclosure. In this embodiment, when the detection result of the occupant indicates that the occupant has been detected, the occupant's action of sticking at least one of a hand, an arm, a head, a foot, and a crotch out of a vehicle window is identified based on the image and the position information of the occupant included in the detection result of the occupant, and a result of identifying the dangerous behavior of the occupant is obtained. This makes it possible to accurately identify the occupant's action of sticking at least one of a hand, an arm, a head, a foot, and a crotch out of a vehicle window based on the position of the occupant, thereby improving the safety of the occupant in the vehicle interior.
[0068] In one possible embodiment, when the passenger detection result indicates that a passenger has been detected, identifying a dangerous act based on the image and the passenger's position information included in the passenger detection result and obtaining an identification result of the dangerous act of the passenger includes identifying a dangerous act in the vehicle cabin based on the image and obtaining estimated information of the dangerous act in the vehicle cabin, and when the passenger detection result indicates that a passenger has been detected, obtaining an identification result of the dangerous act of the passenger based on the passenger's position information included in the passenger detection result and the estimated information of the dangerous act. As an example of this embodiment, a first feature map corresponding to the image is input to a second estimation sub-network, and estimated information of the dangerous act in the vehicle cabin can be obtained via the second estimation sub-network. As another example of this embodiment, a first feature map corresponding to the image is processed using a predesigned third function to obtain estimated information of the dangerous act in the vehicle cabin. Here, the estimated information of the dangerous act may include estimated information indicating whether a passenger corresponding to at least one position in the image is performing a dangerous act. For example, the estimated information of the dangerous behavior may include estimated information indicating whether a passenger corresponding to each pixel in the image is engaging in a dangerous behavior. In this embodiment, a dangerous behavior in the vehicle cabin is identified based on the image, estimated information of the dangerous behavior in the vehicle cabin is obtained, and when the passenger detection result indicates that a passenger has been detected, an identification result of the dangerous behavior of the passenger is obtained based on the passenger's position information included in the passenger detection result and the estimated information of the dangerous behavior. This allows the passenger's position and the estimated information of the dangerous behavior in the vehicle cabin to be combined to accurately identify the passenger's behavior of sticking a predetermined body part out of the vehicle window, thereby improving the safety of the passenger in the vehicle cabin.
[0069] As an example of this embodiment, the passenger position information includes coordinates of an estimated human body center point in the image, and the estimated information of the dangerous behavior includes a probability that the passenger's behavior corresponds to each of N (N is an integer equal to or greater than 1) types of dangerous behavior set in advance. When the passenger detection result indicates that a passenger has been detected, obtaining an identification result of the dangerous behavior of the passenger based on the passenger position information and the estimated information of the dangerous behavior included in the passenger detection result includes determining that a passenger has been detected when the passenger detection result indicates that a human body center point exists, obtaining from the estimated information of the dangerous behavior a probability that the passenger's behavior corresponding to the estimated human body center point corresponds to each of N types of dangerous behavior set in advance based on the coordinates of the estimated human body center point included in the passenger detection result, and obtaining an identification result of the dangerous behavior of the passenger corresponding to the estimated human body center point based on the probability that the passenger's behavior corresponding to the estimated human body center point corresponds to each of the N types of dangerous behavior set in advance. In this example, the estimated information of the dangerous behavior may include a probability that the behavior of the passenger corresponding to all or a part of the pixels in the image corresponds to each of the N types of dangerous behaviors set in advance. For example, the estimated information of the dangerous behavior may include a probability that the behavior of the passenger corresponding to each pixel in the image corresponds to each of the N types of dangerous behaviors set in advance. For example, the estimated information of the dangerous behavior may be a feature map of H×W×N (H indicates the height of the image, and W indicates the width of the image) or a three-dimensional array. Based on the coordinates of the estimated human body center point, the probability that the behavior of the passenger corresponding to the estimated human body center point corresponds to each of the N types of dangerous behaviors set in advance can be obtained from the estimated information of the dangerous behavior, so that the identification result of the dangerous behavior of the passenger corresponding to the estimated human body center point can be obtained. In one example, when the probability that the behavior of the passenger corresponding to any one of the estimated human body center points corresponds to any one of the N types of dangerous behaviors set in advance is greater than a second threshold, it can be determined that the passenger is performing a dangerous behavior as an identification result of the dangerous behavior of the passenger corresponding to the estimated human body center point.If the probability that the passenger's behavior corresponding to any one of the estimated human body center points corresponds to each of the N types of pre-set risky behaviors is equal to or less than the second threshold, the identification result of the risky behavior of the passenger corresponding to the estimated human body center point may be determined to be that the passenger is not performing the risky behavior. For example, the second threshold may be 0.5. Of course, those skilled in the art may flexibly set the magnitude of the second threshold according to the needs of the actual application scenario, and the magnitude is not limited here. According to this example, the identification result of the risky behavior of the passenger corresponding to each estimated human body center point in the image may be accurately determined.
[0070] As an example of this embodiment, identifying a dangerous behavior in the vehicle cabin based on the image and obtaining estimated information of the dangerous behavior in the vehicle cabin includes performing a convolution operation and a full combination operation on a first feature map corresponding to the image, and then performing a classification operation to obtain estimated information of the dangerous behavior in the vehicle cabin. In this example, a convolution operation is performed on the first feature map to obtain a fifth feature map, a full combination operation is performed on the fifth feature map to obtain a sixth feature map, and a classification operation is performed on the sixth feature map to obtain estimated information of the dangerous behavior in the vehicle cabin. Here, the convolution operation is performed once or twice or more times on the first feature map to obtain the fifth feature map. For example, the convolution operation is performed twice on the first feature map to obtain the fifth feature map. The full combination operation is performed once or twice or more times on the fifth feature map to obtain the sixth feature map. For example, the full combination operation is performed once on the fifth feature map to obtain the sixth feature map. Those skilled in the art can flexibly determine the number of convolution operations and the number of full-connection operations according to the needs of actual application scenarios, and are not limited here. In this example, the convolution operation is performed on the first feature map to extract deeper features of the image. The fifth feature map thus obtained can more accurately indicate the features of risky behavior in the image. The use of the full-connection layer operation can improve the fitting ability of the network, and therefore the accuracy of the obtained estimation information of risky behavior can be improved.
[0071] In one possible embodiment, after obtaining a first feature map corresponding to an image, the location of the center point of the human body and the classification of the dangerous behavior are performed through two branches, respectively. The location branch of the human body center point performs a further convolution operation on the first feature map to obtain a second feature map, and performs a maximum pooling operation, activation processing, etc. based on the second feature map to obtain the coordinates of the human body center point. The dangerous behavior branch performs a convolution operation, a full connection operation, and a classification operation on the first feature map to obtain information on the type of behavior corresponding to each human body center point, thereby obtaining the detection result of the dangerous behavior of each passenger in the vehicle cabin. In addition, the location of the human body center point and the classification of the dangerous behavior share the same feature extraction network, which is favorable for improving the reliability of the results and saving computing resources.
[0072] In one possible embodiment, the method further includes, when the passenger detection result indicates that a passenger is detected in the front seat area of the vehicle cabin, identifying a dangerous behavior based on the image and the passenger's position information included in the passenger detection result, and obtaining an identification result of the dangerous behavior of the passenger. In this embodiment, the passenger detected in the front seat area of the vehicle cabin may include a driver and / or a passenger. Since both the driver's dangerous behavior and the passenger's dangerous behavior have a significant impact on the driver's driving, in this embodiment, when the passenger detection result indicates that a passenger is detected in the front seat area of the vehicle cabin, identifying a dangerous behavior based on the image and the passenger's position information included in the passenger detection result, and obtaining an identification result of the dangerous behavior of the passenger, which is advantageous to improving driving safety.
[0073] In one possible embodiment, after obtaining the identification result of the dangerous behavior of the occupant, the method further includes issuing a warning information when the identification result of the dangerous behavior indicates that the behavior of the occupant includes any one of the preset dangerous behaviors. In this embodiment, when the identification result of the dangerous behavior indicates that the behavior of the occupant includes any one of the preset dangerous behaviors, the method further includes issuing a warning information. This can realize a safety alert, which is advantageous to improving the safety of the occupant in the vehicle cabin.
[0074] As an example of this embodiment, issuing the warning information includes at least one of controlling an in-vehicle voice dialogue device to issue a voice warning information, issuing a command to control the raising and lowering of a vehicle window corresponding to the passenger engaging in the preset dangerous behavior, and issuing a command to turn on a hazard warning light. In one example, when the identification result of the dangerous behavior indicates that the behavior of the passenger includes any one of the preset dangerous behaviors, the in-vehicle voice dialogue device is controlled to issue a voice warning information. For example, the in-vehicle voice dialogue device can be controlled to issue a voice warning information of "Don't lean out of the window, it stinks." According to this example, since the passenger can be warned by voice, the passenger can obtain the warning information without looking at the display screen of the infotainment. In one example, when the identification result of the dangerous behavior indicates that the behavior of the passenger includes any one of the preset dangerous behaviors, a command to control the raising and lowering of a vehicle window corresponding to the passenger engaging in the preset dangerous behavior can be issued. For example, when the passenger engaging in the preset dangerous behavior is a driver, the vehicle window corresponding to the passenger engaging in the preset dangerous behavior may be the front left vehicle window. When the passenger performing the predetermined dangerous behavior is a passenger assistant, the vehicle window corresponding to the passenger performing the predetermined dangerous behavior may be the front right vehicle window. When the passenger performing the predetermined dangerous behavior is a passenger performing the predetermined dangerous behavior on the rear left side, the vehicle window corresponding to the passenger performing the predetermined dangerous behavior may be the rear left vehicle window. When the passenger performing the predetermined dangerous behavior is a passenger performing the predetermined dangerous behavior on the rear right side, the vehicle window corresponding to the passenger performing the predetermined dangerous behavior may be the rear right vehicle window. In this example, by issuing a command to control the raising and lowering of the vehicle window corresponding to the passenger performing the predetermined dangerous behavior, the effect of warning can be strengthened, and the passenger is prompted to naturally retract the body part that is exposed outside the vehicle window at the moment the vehicle window rises or falls. In one example, when the identification result of the dangerous behavior indicates that the behavior of the passenger includes any one of the predetermined dangerous behaviors, a command to turn on a hazard warning light can be issued.This has the effect of alerting nearby vehicles, thereby improving the safety of passengers in the vehicle.
[0075] The method for identifying dangerous actions according to the embodiment of the present disclosure will be described below by taking one specific application scenario as an example. FIG. 4 shows a schematic diagram of an application scenario of the method for identifying dangerous actions according to the present disclosure. In the example shown in FIG. 4, an image of a vehicle interior can be acquired. For example, the size of the image of the vehicle interior may be 640×480. The image is input to a backbone network, and feature extraction is performed on the image via the backbone network to obtain a first feature map. Here, the size of the first feature map may be 80×60×C (C indicates the number of channels of the first feature map, and C may be 3 or more).
[0076] A convolution operation is performed on the first feature map by the first estimation sub-network to obtain a second feature map. Here, the size of the second feature map may be 80×60×3. An activation process is performed on the 0th channel of the second feature map by the Sigmoid function to obtain a third feature map. A maximum value pooling operation with a pooling window size of 3×3 and a step size of 1 is performed on the third feature map to obtain the coordinates of a first candidate point of the human body center point in the image and the probability that the first candidate point corresponds to the human body center point. Here, the number of first candidate points may be 60×80. The first candidate points having the same coordinates are merged to obtain the coordinates of a second candidate point and the probability that the second candidate point corresponds to the human body center point. Among the M second candidate points having the highest probability of corresponding to the human body center point, the second candidate point having a probability of corresponding to the human body center point exceeding 0.5 may be determined as the human body center estimation point. The length of the body frame corresponding to the estimated body center point can be obtained from the first channel of the second feature map, and the width of the body frame corresponding to the estimated body center point can be obtained from the second channel of the second feature map.
[0077] The first feature map is input to a second estimation sub-network, and a convolution operation and a full combination operation are performed on the first feature map via the second estimation sub-network, followed by a classification operation, to obtain estimated information of the dangerous behavior in the vehicle cabin. Here, the estimated information of the dangerous behavior may include a probability that the behavior of the occupant corresponding to each pixel in the image corresponds to each of N types of dangerous behaviors set in advance. For example, the estimated information of the dangerous behavior may be a 640×480×N feature map or a three-dimensional array. Based on the coordinates of the estimated human body center point, the estimated information of the dangerous behavior can obtain a probability that the behavior of the occupant corresponding to the estimated human body center point corresponds to each of N types of dangerous behaviors set in advance, and therefore, an identification result of the dangerous behavior of the occupant corresponding to the estimated human body center point can be obtained.
[0078] It should be understood that the above method embodiments mentioned in this disclosure can be combined with each other to form an embodiment without violating the principles and logic. Due to the limited number of pages, detailed descriptions are omitted in this disclosure. In the above method of a specific embodiment, it can be understood by those skilled in the art that the specific execution order of each step should be determined according to its function and possible internal logic.
[0079] The present disclosure further provides a device for identifying risky behavior, an electronic device, a computer-readable storage medium, and a program, all of which can be used for any one of the methods for identifying risky behavior according to the present disclosure. For the corresponding technical means and explanations, reference can be made to the corresponding description of the method, and detailed explanations will be omitted.
[0080] 5 shows a block diagram of a dangerous behavior identification device according to an embodiment of the present disclosure. As shown in FIG. 5, the dangerous behavior identification device includes an acquisition module 51 for acquiring an image of a vehicle interior, an occupant detection module 52 for detecting an occupant in the vehicle interior based on the image and obtaining a detection result of the occupant in the vehicle interior, and a first dangerous behavior identification module 53 for identifying a dangerous behavior based on the image and position information of the occupant included in the detection result of the occupant when the detection result of the occupant indicates that the occupant has been detected, and obtaining an identification result of the dangerous behavior of the occupant, where the dangerous behavior is an action of sticking a predetermined body part out of a vehicle window.
[0081] In one possible embodiment, the first dangerous behavior identification module 53 identifies dangerous behavior in the vehicle cabin based on the image and obtains estimated information of the dangerous behavior in the vehicle cabin, and if the occupant detection result indicates that a occupant has been detected, obtains an identification result of the occupant's dangerous behavior based on the occupant's location information included in the occupant detection result and the estimated information of the dangerous behavior.
[0082] In one possible embodiment, the occupant's position information includes coordinates of an estimated body center point in the image, and the estimated risky behavior information includes a probability that the occupant's behavior corresponds to each of N (N is an integer greater than or equal to 1) predetermined types of risky behavior.
[0083] When the occupant detection result indicates the presence of an estimated human body center point, the first dangerous behavior identification module 53 determines that an occupant has been detected, and obtains from the estimated dangerous behavior information a probability that the occupant's behavior corresponding to the estimated human body center point falls under each of N predetermined dangerous behaviors based on the coordinates of the estimated human body center point included in the occupant detection result, and obtains an identification result of the dangerous behavior of the occupant corresponding to the estimated human body center point based on the probability that the occupant's behavior corresponding to the estimated human body center point falls under each of N predetermined dangerous behaviors.
[0084] In one possible embodiment, the occupant position information includes coordinates of an estimated body center point in the image, and the occupant detection module 52 estimates a probability that a pixel in the image corresponds to a body center point based on a first feature map corresponding to the image, and determines the coordinates of the estimated body center point in the image based on the probability that a pixel in the image corresponds to a body center point.
[0085] In one possible embodiment, the occupant detection module 52 determines the coordinates of a first candidate point for a human body center point in the image and a probability that the first candidate point corresponds to the human body center point based on a first feature map corresponding to the image, and determines the coordinates of an estimated human body center point in the image based on the coordinates of the first candidate point and the probability that the first candidate point corresponds to the human body center point.
[0086] In one possible embodiment, the occupant detection module 52 performs a convolution operation on a first feature map corresponding to the image to obtain a second feature map corresponding to the image, and performs a max pooling operation based on the second feature map to obtain the coordinates of a first candidate point for a body center point in the image and a probability that the first candidate point corresponds to the body center point.
[0087] In one possible embodiment, the occupant detection module 52 performs an activation process on the second feature map to obtain a third feature map corresponding to the image, and performs a max pooling operation on the third feature map to obtain the coordinates of a first candidate point for the body center point in the image and the probability that the first candidate point corresponds to the body center point.
[0088] In one possible embodiment, the occupant detection module 52 performs an overlap maximum pooling operation on the third feature map to obtain coordinates of a first candidate point for a body center point in the image and a probability that the first candidate point corresponds to the body center point, merges the first candidate points with the same coordinates, obtains coordinates of a second candidate point and a probability that the second candidate point corresponds to the body center point, and determines the coordinates of an estimated body center point in the image based on the coordinates of the second candidate point and the probability that the second candidate point corresponds to the body center point.
[0089] In one possible embodiment, the first risky activity identification module 53 performs a convolution operation and a full combination operation on a first feature map corresponding to the image, followed by a classification operation to obtain estimated information of risky activities in the vehicle cabin.
[0090] In one possible embodiment, the dangerous behavior identification device further includes a second dangerous behavior identification module that, when the occupant detection result indicates that a occupant has been detected in the front seat area of the vehicle cabin, identifies dangerous behavior based on the image and the occupant's position information contained in the occupant detection result, and obtains an identification result of the occupant's dangerous behavior.
[0091] In one possible embodiment, the predefined body parts include at least one of the following: hands, arms, heads, feet, and thighs.
[0092] In one possible embodiment, the dangerous behavior identification device further includes a warning module that issues warning information when the dangerous behavior identification result indicates that the occupant's behavior includes any one of the predetermined dangerous behaviors.
[0093] In one possible embodiment, the warning module is used for at least one of controlling an in-vehicle voice dialogue device to issue voice warning information, issuing a command to control the raising and lowering of a vehicle window corresponding to a passenger engaging in the pre-defined dangerous behavior, and issuing a command to turn on a hazard warning light.
[0094] In an embodiment of the present disclosure, an image of a vehicle interior is acquired, an occupant in the vehicle interior is detected based on the image to obtain a detection result of the occupant in the vehicle interior, and when the detection result of the occupant indicates that the occupant has been detected, a dangerous behavior is identified based on the image and position information of the occupant included in the detection result of the occupant, and an identification result of the dangerous behavior of the occupant is obtained, the dangerous behavior being an act of sticking a predetermined body part out of a vehicle window. This makes it possible to accurately identify the act of the occupant sticking a predetermined body part out of a vehicle window based on the position of the occupant, thereby improving the safety of the occupant in the vehicle interior.
[0095] In some embodiments, the functions or modules provided by the apparatuses provided in the embodiments of the present disclosure can be used to perform the methods described in the above method embodiments. For specific embodiments and technical effects, please refer to the descriptions in the above method embodiments, and redundant descriptions will be omitted here for brevity.
[0096] An embodiment of the present disclosure further provides a computer-readable storage medium having computer program instructions stored thereon, the computer-readable storage medium implementing the method for identifying risky activities when executed by a processor. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium.
[0097] An embodiment of the present disclosure further provides a computer program comprising computer readable code, the computer program causing a processor of an electronic device to implement the method for identifying risky activities described above when the computer readable code is executed in the electronic device.
[0098] An embodiment of the present disclosure further provides a computer program product including computer readable code, or a non-volatile computer readable storage medium having computer readable code stored thereon, which, when executed by an electronic device, causes a processor of the electronic device to perform the above-mentioned method for identifying risky activities.
[0099] An embodiment of the present disclosure further provides an electronic device including one or more processors and a memory for storing executable instructions, the one or more processors configured to execute the above-mentioned method for identifying risky activities by invoking the executable instructions stored in the memory.
[0100] The electronic device may be provided as a terminal, a server, or any other form of device.
[0101] 6 is a block diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, the device 800 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.
[0102] Referring to FIG. 6, electronic device 800 may include one or more of a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.
[0103] The processing component 802 typically controls the overall operation of the electronic device 800, such as operations related to display, phone calls, data communication, camera operation, and recording operation. The processing component 802 may include one or more processors 820 that execute instructions to perform all or some of the steps of the above methods. The processing component 802 may also include one or more modules for interaction with other components. For example, the processing component 802 may include a multimedia module for interaction with the multimedia component 808.
[0104] The memory 804 is configured to store various types of data to support operation on the electronic device 800. These data include, by way of example, instructions for any application programs or methods operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by various types of volatile or non-volatile storage devices, such as, for example, static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disks, or optical disks, or combinations thereof.
[0105] The power component 806 provides power to each component of the electronic device 800. The power component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the electronic device 800.
[0106] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and a user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). When the screen includes a touch panel, it may be implemented as a touch screen that receives input signals from a user. The touch panel may include one or more touch sensors to detect touches, slides, and gestures on the touch panel. The touch sensors may detect not only the boundaries of a touch or slide motion, but also the duration and pressure of the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operational mode, such as a shooting mode or a capture mode, the front camera and / or the rear camera may receive external multimedia data. Each of the front and rear cameras may have a fixed optical lens system or a focal length and optical zoom capability.
[0107] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) configured to receive an external audio signal when the electronic device 800 is in an operational mode, such as a call mode, a record mode, or a voice recognition mode. The received audio signal may be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting the audio signal.
[0108] The I / O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include, but are not limited to, a home button, volume buttons, a start button, and a lock button.
[0109] The sensor component 814 includes one or more sensors for evaluating the state of each surface of the electronic device 800. For example, the sensor component 814 can detect the on / off state of the electronic device 800, the relative positioning of components such as the display and keypad of the electronic device 800, and the sensor component 814 can further detect changes in the position of the electronic device 800 or certain components of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration or deceleration of the electronic device 800, and temperature changes of the electronic device 800. The sensor component 814 includes a proximity sensor configured to detect the presence of a nearby object in the absence of any physical contact. The sensor component 814 may further include an optical sensor for use in imaging applications, such as a complementary metal oxide semiconductor (CMOS) or charge-coupled device (CCD) image sensor. In some embodiments, the sensor component 814 may further include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0110] The communication component 816 is configured to realize wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile communication technology (2G), or a third generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system through a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be realized by radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0111] In an exemplary embodiment, electronic device 800 may be implemented with one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components and may be used to perform the methods described above.
[0112] In an exemplary embodiment, a non-volatile computer readable storage medium, such as a memory 804, is further provided that includes computer program instructions that, when executed by the processor 820 of the electronic device 800, can cause the above-described method to be performed.
[0113] FIG. 7 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. Referring to FIG. 7, the electronic device 1900 further includes a processing component 1922 including one or more processors, and a memory resource, represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. The processing component 1922 is also configured to perform the above method by executing the instructions.
[0114] The electronic device 1900 may further include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in memory 1932, such as Microsoft Windows Server operating system (Windows Server™), Apple's graphical user interface operating system Mac OS X™, a multitasking, multiuser operating system for computers (Unix™), a free and open source Unix-based operating system (Linux™), an open source Unix-based operating system (FreeBSD™), or the like.
[0115] In an exemplary embodiment, a non-volatile computer-readable storage medium, such as a memory 1932 containing computer program instructions, which when executed by the processing component 1922 of the electronic device 1900, can cause the above-described method to be performed, is further provided.
[0116] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to implement aspects of the present disclosure.
[0117] Computer-readable storage media A computer-readable storage medium may be a tangible device capable of storing and storing instructions for use in an instruction execution device. A computer-readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. Further specific examples (non-exhaustive list) of computer-readable storage media include portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or slot-in protrusion structures on which instructions are stored, and any suitable combination of the above. As used herein, a computer-readable storage medium is not to be construed as being a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated via a waveguide or other transmission medium (e.g., pulsed light passing through a fiber optic cable), or electrical signals transmitted via electrical wires.
[0118] The computer readable program instructions described herein may be downloaded from a computer readable storage medium into each computing / processing device, or may be downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium within each computing / processing device.
[0119] Computer program instructions for carrying out the operations of the present disclosure may be assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or target code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and general procedural programming languages such as the "C" language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. When a remote computer is involved, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, state information of the computer readable program instructions may be used to personalize electronic circuitry, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), to execute the computer readable program instructions to implement aspects of the present disclosure.
[0120] Although various aspects of the present disclosure have been described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present disclosure, it should be understood that each block of the flowcharts and / or block diagrams, and combinations of each block in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.
[0121] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus such that, when these instructions are executed by the processor of the computer or other programmable data processing apparatus, it produces an apparatus that performs the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may be stored on a computer readable storage medium to cause a computer, programmable data processing apparatus and / or other device to operate in a determined manner. Thus, a computer readable storage medium having instructions stored thereon includes an article of manufacture having instructions that implement each aspect of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.
[0122] The computer readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device, and cause the computer, other programmable data processing apparatus, or other device to perform a series of operational steps to generate a computer executable process, such that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.
[0123] The flowcharts and block diagrams in the drawings illustrate possible system architectures, functions, and operations of systems, methods, and computer program products according to several embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, program segment, or part of instructions, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions depicted in the blocks may be implemented in a different order than that depicted in the drawings. For example, two consecutive blocks may be substantially executed in parallel, or may be executed in a reverse order depending on the function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated system based on hardware that executes the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.
[0124] The computer program product may be specifically implemented by hardware, software or a combination thereof. In one possible embodiment, the computer program product may be specifically a computer storage medium. In another possible embodiment, the computer program product may be specifically a software product, such as a Software Development Kit (SDK).
[0125] Although the embodiments of the present disclosure have been described above, the above description is merely illustrative and not exhaustive, and is not limited to the embodiments disclosed. Various modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms selected in this specification are intended to appropriately interpret the principles, practical applications, or technical improvements to the prior art of the embodiments, or to allow other skilled in the art to understand the embodiments disclosed in this specification.
Claims
1. A method for identifying a dangerous act, comprising: Obtaining an image of the vehicle interior; Detecting an occupant in the vehicle cabin based on the image, and obtaining a detection result of the occupant in the vehicle cabin, wherein when the detection result of the occupant indicates that an occupant has been detected, the detection result of the occupant includes coordinates of an estimated center point of a human body in the image; Identifying a dangerous behavior in the vehicle cabin based on the image, and obtaining estimated information of the dangerous behavior in the vehicle cabin, the estimated information including estimated information indicating whether an occupant corresponding to each pixel in the image is engaging in a dangerous behavior; When the occupant detection result indicates that an occupant has been detected, based on coordinates of an estimated human body center point included in the occupant detection result, acquisition of inferred information indicating whether the occupant corresponding to the estimated human body center point is engaging in a dangerous behavior from inferred information of the dangerous behavior; and obtaining a result of identifying a dangerous behavior of the occupant based on estimated information indicating whether the occupant corresponding to the estimated human body center point is engaging in a dangerous behavior; A method for identifying risky behavior, characterized in that the risky behavior is an action of sticking a predetermined body part out of a vehicle window.
2. the estimated information on the risky behavior includes a probability that the behavior of the occupant corresponding to each pixel in the image corresponds to each of N (N is an integer equal to or greater than 1) types of risky behaviors that are set in advance; obtaining, based on coordinates of an estimated human body center point included in the detection result of the occupant, estimated information indicating whether the occupant corresponding to the estimated human body center point is engaging in a dangerous behavior from estimated information on the dangerous behavior, and acquiring, based on coordinates of an estimated human body center point included in the detection result of the occupant, a probability that the action of the occupant corresponding to the estimated human body center point corresponds to each of N types of predefined risky actions from the estimated information of the risky actions; Obtaining a result of identifying a dangerous behavior of the occupant based on estimated information indicating whether the occupant corresponding to the estimated human body center point is engaging in a dangerous behavior, The method for identifying dangerous actions as described in claim 1, further comprising obtaining an identification result of the dangerous action of the occupant corresponding to the estimated human body center point based on the probability that the action of the occupant corresponding to the estimated human body center point corresponds to each of N predetermined types of dangerous actions.
3. The above-mentioned, detecting an occupant in the vehicle cabin based on the image, and obtaining a detection result of the occupant in the vehicle cabin, estimating a probability that a pixel in the image corresponds to a body center point based on a first feature map corresponding to the image; The method for identifying risky actions according to claim 1 or 2, further comprising determining coordinates of an estimated center point of the human body in the image based on a probability that a pixel in the image corresponds to the center point of the human body.
4. The step of estimating a probability that a pixel in the image corresponds to a body center point based on a first feature map corresponding to the image includes determining, based on the first feature map corresponding to the image, coordinates of a first candidate point for the body center point in the image and a probability that the first candidate point corresponds to the body center point; The method for identifying risky actions described in claim 3, characterized in that determining the coordinates of an estimated human body center point in the image based on the probability that a pixel in the image corresponds to a human body center point includes determining the coordinates of an estimated human body center point in the image based on the coordinates of the first candidate point and the probability that the first candidate point corresponds to a human body center point.
5. Determining the coordinates of a first candidate point for a human body center point in the image and a probability that the first candidate point corresponds to the human body center point based on the first feature map corresponding to the image, performing a convolution operation on a first feature map corresponding to the image to obtain a second feature map corresponding to the image; The method for identifying risky actions as described in claim 4, further comprising: performing a maximum pooling operation based on the second feature map to obtain coordinates of a first candidate point for a center point of the human body in the image and a probability that the first candidate point corresponds to the center point of the human body.
6. The above-mentioned, performing a maximum pooling operation based on the second feature map to obtain a coordinate of a first candidate point of a human body center point in the image and a probability that the first candidate point corresponds to the human body center point, performing an activation process on the second feature map to obtain a third feature map corresponding to the image; The method for identifying risky actions as described in claim 5, further comprising: performing a maximum pooling operation on the third feature map to obtain coordinates of a first candidate point for a center point of the human body in the image and a probability that the first candidate point corresponds to the center point of the human body.
7. performing a maximum pooling operation on the third feature map to obtain a coordinate of a first candidate point for a human body center point in the image and a probability that the first candidate point corresponds to the human body center point includes performing an overlap maximum pooling operation on the third feature map to obtain a coordinate of a first candidate point for a human body center point in the image and a probability that the first candidate point corresponds to the human body center point; The method for identifying risky actions according to claim 6, characterized in that determining the coordinates of an estimated human body center point in the image based on the coordinates of the first candidate point and the probability that the first candidate point corresponds to a human body center point includes: merging the first candidate points having the same coordinates to obtain the coordinates of a second candidate point and the probability that the second candidate point corresponds to a human body center point; and determining the coordinates of the estimated human body center point in the image based on the coordinates of the second candidate point and the probability that the second candidate point corresponds to a human body center point.
8. The method of identifying a dangerous behavior in the vehicle cabin based on the image and obtaining estimated information on the dangerous behavior in the vehicle cabin includes: The method for identifying dangerous behaviors as described in claim 1 or 2, characterized in that it includes performing a convolution operation and a full combination operation on a first feature map corresponding to the image, and then performing a classification operation to obtain estimated information of dangerous behaviors in the vehicle cabin.
9. The method for identifying dangerous behavior described in any one of claims 1 to 8, further comprising, when the occupant detection result indicates that an occupant has been detected in the front seat area of the vehicle compartment, identifying a dangerous behavior based on the image and the occupant's position information contained in the occupant detection result, and obtaining an identification result of the dangerous behavior of the occupant.
10. 10. The method for identifying risky behavior according to claim 1, wherein the predetermined body parts include at least one of a hand, an arm, a head, a foot, and a groin.
11. After obtaining the identification result of the dangerous behavior of the passenger, A method for identifying dangerous behavior described in any one of claims 1 to 10, characterized in that when the dangerous behavior identification result indicates that the occupant's behavior includes any one of predetermined dangerous behaviors, it further includes issuing warning information.
12. As mentioned above, issuing warning information is Controlling an in-vehicle voice dialogue device to output voice alert information; issuing a command to control the raising and lowering of a vehicle window corresponding to the passenger engaging in the predetermined dangerous behavior; and issuing a command to turn on a hazard warning light.
13. A device for identifying dangerous acts, comprising: an acquisition module for acquiring an image of the vehicle interior; an occupant detection module for detecting an occupant in the vehicle cabin based on the image and obtaining a detection result of the occupant in the vehicle cabin, where when the detection result of the occupant indicates that an occupant has been detected, the detection result of the occupant includes coordinates of an estimated center point of a human body in the image; a first dangerous behavior identification module for identifying a dangerous behavior in the vehicle cabin based on the image, obtaining estimated information of the dangerous behavior in the vehicle cabin including estimated information indicating whether an occupant corresponding to each pixel in the image is engaging in a dangerous behavior, and when the occupant detection result indicates that an occupant has been detected, obtaining estimated information from the dangerous behavior estimated information based on coordinates of an estimated human body center point included in the occupant detection result indicating whether the occupant corresponding to the estimated human body center point is engaging in a dangerous behavior, and obtaining an identification result of the dangerous behavior of the occupant based on the estimated information indicating whether the occupant corresponding to the estimated human body center point is engaging in a dangerous behavior, The device for identifying dangerous behavior is characterized in that the dangerous behavior is an action of sticking a predetermined body part out of a vehicle window.
14. An electronic device, one or more processors; a memory for storing executable instructions; The electronic device is characterized in that the one or more processors are configured to execute the method for identifying dangerous activities described in any one of claims 1 to 12 by calling up executable instructions stored in the memory.
15. A computer-readable storage medium having computer program instructions stored thereon, the computer-readable storage medium being characterized in that, when the computer program instructions are executed by a processor, the computer-readable storage medium realizes the method for identifying risky acts described in any one of claims 1 to 12.
16. 1. A computer program product comprising: A computer program product comprising computer-readable code or a non-volatile computer-readable storage medium on which computer-readable code is stored, the computer-readable code being executed within an electronic device to cause a processor of the electronic device to execute the method for identifying risky activities described in any one of claims 1 to 12.
Citation Information
Patent Citations
Fellow passenger sitting posture detecting / determining apparatus and method
JP2008199515A
Control apparatus and program
JP2019098932A
Method and apparatus for identifying dangerous actions of persons in vehicle, electronic device and storage medium
WO2020173213A1
