Action detection method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a motion detection method and device, electronic equipment and storage medium. The motion detection method comprises: acquiring a video stream of a scene area; detecting a motion direction of a target object in the scene area based on a plurality of image frames in the video stream; detecting a skeletal key point of the target object in the image frames; and determining motion information of the target object according to a geometric relationship between target key points corresponding to the motion direction in the detected skeletal key points.

Description

Technical Field

[0001] This disclosure relates to the field of image detection technology, and in particular to a motion detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of artificial intelligence technology, the types of image and video detection are increasing, and the effects are becoming better and better. In particular, applying detection technology to the field of security protection can prevent users from being in danger. Taking the vehicle cabin scenario as an example, image processing technology can be used to detect the safety of the cabin environment and personnel. By collecting images and videos inside the vehicle while it is in motion or parked, it is possible to detect whether people inside the vehicle are performing dangerous actions, thereby improving driving and passenger safety. However, the detection of human actions in related technologies is prone to false positives, resulting in a poor user experience. Summary of the Invention

[0003] This disclosure provides a motion detection method, apparatus, device, and storage medium to address the deficiencies in related technologies.

[0004] According to a first aspect of the present disclosure, an action detection method is provided, comprising:

[0005] Obtain the video stream of the scene area;

[0006] Based on multiple image frames in the video stream, the motion direction of the target object within the scene area is detected;

[0007] Detect the skeletal key points of the target object in the image frame;

[0008] Based on the geometric relationship between the target key points in the detected skeletal key points and the target key points corresponding to the direction of movement, the motion information of the target object is determined.

[0009] In one embodiment, detecting the motion direction of a target object within the scene region based on multiple image frames in the video stream includes:

[0010] Extract the features of the target object within each of the multiple image frames;

[0011] Based on the characteristics of the target object within each of the plurality of image frames, the optical flow information of the target object in the video stream is determined;

[0012] Based on the optical flow information of the target object in the video stream, the motion direction of the target object within the scene area is determined.

[0013] In one embodiment, each action direction corresponds to a target action;

[0014] The step of determining the motion information of the target object based on the geometric relationship between the target key points corresponding to the motion direction among the detected skeletal key points includes:

[0015] If the geometric relationship between the detected skeletal key points and the target key points corresponding to the action direction satisfies the first preset condition corresponding to the action direction, it is determined that the target object has a target action corresponding to the action direction.

[0016] Otherwise, it is determined that the target object does not have a target action corresponding to the action direction.

[0017] In one embodiment, the skeletal key points include: left shoulder key point, right shoulder key point, left wrist key point, right wrist key point, left elbow key point, right elbow key point, left ear key point, and right ear key point.

[0018] The step of determining the motion information of the target object based on the geometric relationship between the target key points corresponding to the motion direction among the detected skeletal key points includes:

[0019] When the movement direction is left, if the tangent of the angle between the first target vector from the right shoulder key point to the left shoulder key point and the standard vector to the right is positive and the absolute value of the tangent is greater than a first threshold, it is determined that the target object has a leftward leaning movement; and / or,

[0020] When the movement direction is to the right, if the tangent of the angle between the first target vector from the right shoulder key point to the left shoulder key point and the standard vector to the right is negative, and the absolute value of the tangent is greater than a first threshold, it is determined that the target object has a rightward leaning movement; and / or,

[0021] When the action direction is upward, the target object is determined to be making a chest-covering motion in response to either the first or second scenario below; the first scenario includes: the angle between the second and third target vectors is greater than a second threshold, the left wrist key point is lower than the right shoulder key point, and the vertical distance between the left wrist key point and the right shoulder key point is greater than a third threshold, wherein the second target vector is the vector from the left elbow key point to the left wrist key point, and the third target vector is the vector from the left elbow key point to the left shoulder key point; the second scenario includes: the angle between the fourth and fifth target vectors is greater than a second threshold, the right wrist key point is lower than the left shoulder key point, and the vertical distance between the right wrist key point and the left shoulder key point is greater than a third threshold; wherein the fourth target vector is the vector from the right elbow key point to the right wrist key point, and the fifth target vector is the vector from the right elbow key point to the right shoulder key point; and / or,

[0022] When the direction of the movement is downward, in response to the left ear key point being lower than the left shoulder key point and the vertical distance between the left ear key point and the left shoulder key point being greater than the fourth threshold, and / or in response to the right ear key point being lower than the right shoulder key point and the vertical distance between the right ear key point and the right shoulder key point being greater than the fourth threshold, it is determined that the target object has a forward leaning and lying down movement.

[0023] In one embodiment, detecting the motion direction of a target object within the scene region based on multiple image frames in the video stream includes:

[0024] Detect whether the target object within the video stream is moving;

[0025] When the target object in the video stream is moving, the direction of motion of the target object in the scene area is detected based on multiple image frames in the video stream.

[0026] In one embodiment, determining the motion information of the target object based on the geometric relationship between the detected skeletal keypoints and the target keypoints corresponding to the motion direction includes:

[0027] Based on the geometric relationship between the target key points corresponding to the action direction detected in the skeletal key points in each of the multiple image frames in the video stream, the action information of the target object in the corresponding image frame is determined.

[0028] The motion information of the target object in the multiple image frames is smoothed to obtain the motion information of the target object determined based on the multiple image frames.

[0029] In one embodiment, it also includes:

[0030] During the process of acquiring the video stream of the scene area, the latest preset number of image frames in the video stream are cached in real time.

[0031] The step of smoothing the motion information of the target object in the multiple image frames to obtain the motion information of the target object determined based on the multiple image frames includes:

[0032] The motion information of the target object in the cached preset number of image frames is smoothed to obtain the motion information of the target object determined based on the cached preset number of image frames.

[0033] In one embodiment, the real-time caching of the latest preset number of image frames includes:

[0034] Detect whether there is preset key information of the target object in each image frame of the video stream, wherein the preset key information includes at least one of face, at least part of body and skeletal key points;

[0035] The latest preset number of image frames in the video stream containing preset key information of the target object are cached in real time.

[0036] In one embodiment, the scene area includes a vehicle cabin scene area;

[0037] The acquisition of the video stream of the scene area includes:

[0038] When the vehicle doors are locked and / or the vehicle speed reaches a preset speed threshold, acquire the video stream of the scene area.

[0039] In one embodiment, it also includes:

[0040] Detect multiple objects in the video stream;

[0041] The target object is determined from the plurality of objects based on the location of each object within the vehicle cabin and / or the facial information of each object.

[0042] In one embodiment, it also includes:

[0043] When the action information of the target object indicates that the target object has a target action, an alarm message is sent to the service platform.

[0044] According to a second aspect of the present disclosure, an action detection device is provided, comprising:

[0045] The acquisition module is used to acquire the video stream of the scene area;

[0046] The direction module is used to detect the motion direction of a target object within the scene area based on multiple image frames in the video stream;

[0047] The detection module is used to detect the skeletal key points of the target object in the image frame;

[0048] The determination module is used to determine the motion information of the target object based on the geometric relationship between the target key points corresponding to the motion direction among the detected skeletal key points.

[0049] In one embodiment, the direction module is specifically used for:

[0050] Extract the features of the target object within each of the multiple image frames;

[0051] Based on the characteristics of the target object within each of the plurality of image frames, the optical flow information of the target object in the video stream is determined;

[0052] Based on the optical flow information of the target object in the video stream, the motion direction of the target object within the scene area is determined.

[0053] In one embodiment, each action direction corresponds to a target action;

[0054] The determining module is specifically used for:

[0055] If the geometric relationship between the detected skeletal key points and the target key points corresponding to the action direction satisfies the first preset condition corresponding to the action direction, it is determined that the target object has a target action corresponding to the action direction.

[0056] Otherwise, it is determined that the target object does not have a target action corresponding to the action direction.

[0057] In one embodiment, the skeletal key points include: left shoulder key point, right shoulder key point, left wrist key point, right wrist key point, left elbow key point, right elbow key point, left ear key point, and right ear key point.

[0058] The determining module is specifically used for:

[0059] When the movement direction is left, if the tangent of the angle between the first target vector from the right shoulder key point to the left shoulder key point and the standard vector to the right is positive and the absolute value of the tangent is greater than a first threshold, it is determined that the target object has a leftward leaning movement; and / or,

[0060] When the movement direction is to the right, if the tangent of the angle between the first target vector from the right shoulder key point to the left shoulder key point and the standard vector to the right is negative, and the absolute value of the tangent is greater than a first threshold, it is determined that the target object has a rightward leaning movement; and / or,

[0061] When the action direction is upward, the target object is determined to be making a chest-covering motion in response to either the first or second scenario below; the first scenario includes: the angle between the second and third target vectors is greater than a second threshold, the left wrist key point is lower than the right shoulder key point, and the vertical distance between the left wrist key point and the right shoulder key point is greater than a third threshold, wherein the second target vector is the vector from the left elbow key point to the left wrist key point, and the third target vector is the vector from the left elbow key point to the left shoulder key point; the second scenario includes: the angle between the fourth and fifth target vectors is greater than a second threshold, the right wrist key point is lower than the left shoulder key point, and the vertical distance between the right wrist key point and the left shoulder key point is greater than a third threshold; wherein the fourth target vector is the vector from the right elbow key point to the right wrist key point, and the fifth target vector is the vector from the right elbow key point to the right shoulder key point; and / or,

[0062] When the direction of the movement is downward, in response to the left ear key point being lower than the left shoulder key point and the vertical distance between the left ear key point and the left shoulder key point being greater than the fourth threshold, and / or in response to the right ear key point being lower than the right shoulder key point and the vertical distance between the right ear key point and the right shoulder key point being greater than the fourth threshold, it is determined that the target object has a forward leaning and lying down movement.

[0063] In one embodiment, the direction module is specifically used for:

[0064] Detect whether the target object within the video stream is moving;

[0065] When the target object in the video stream is moving, the direction of motion of the target object in the scene area is detected based on multiple image frames in the video stream.

[0066] In one embodiment, the determining module is specifically used for:

[0067] Based on the geometric relationship between the target key points corresponding to the action direction detected in the skeletal key points in each of the multiple image frames in the video stream, the action information of the target object in the corresponding image frame is determined.

[0068] The motion information of the target object in the multiple image frames is smoothed to obtain the motion information of the target object determined based on the multiple image frames.

[0069] In one embodiment, a caching module is also included, for:

[0070] During the process of acquiring the video stream of the scene area, the latest preset number of image frames in the video stream are cached in real time.

[0071] The determining module is used to smooth the motion information of the target object in the multiple image frames to obtain the motion information of the target object determined based on the multiple image frames. Specifically, it is used to:

[0072] The motion information of the target object in the cached preset number of image frames is smoothed to obtain the motion information of the target object determined based on the cached preset number of image frames.

[0073] In one embodiment, the caching module is specifically used for:

[0074] Detect whether there is preset key information of the target object in each image frame of the video stream, wherein the preset key information includes at least one of face, at least part of body and skeletal key points;

[0075] The latest preset number of image frames in the video stream containing preset key information of the target object are cached in real time.

[0076] In one embodiment, the scene area includes a vehicle cabin scene area;

[0077] The acquisition module is specifically used for:

[0078] When the vehicle doors are locked and / or the vehicle speed reaches a preset speed threshold, acquire the video stream of the scene area.

[0079] In one embodiment, a target module is also included, for:

[0080] Detect multiple objects in the video stream;

[0081] The target object is determined from the plurality of objects based on the location of each object within the vehicle cabin and / or the facial information of each object.

[0082] In one embodiment, an alarm module is also included, for:

[0083] When the action information of the target object indicates that the target object has a target action, an alarm message is sent to the service platform.

[0084] According to a third aspect of the present disclosure, an electronic device is provided, the device including a memory and a processor, the memory being configured to store computer instructions executable on the processor, and the processor being configured to implement the method described in the first aspect when executing the computer instructions.

[0085] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0086] As described in the above embodiments, by acquiring a video stream of a scene area, the movement direction of a target object within the scene area can be detected based on multiple image frames in the video stream. Then, the skeletal key points of the target object in the image frames can be detected. Finally, the movement information of the target object can be determined based on the geometric relationship between the target key points in the skeletal key points and the movement direction. Since the geometric relationship between the target key points is relatively objective and accurate, it can accurately detect whether the target object is performing a dangerous movement. Furthermore, by introducing the movement direction detection result before judging the movement information to specifically analyze the geometric relationship between the target key points, the temporal and spatial complexity of movement detection is reduced, making movement detection more targeted and further improving the accuracy of detection. If this detection method is applied to a vehicle, it can accurately detect whether the driver and passengers inside the vehicle are in danger, thereby improving passenger safety and enhancing the user experience.

[0087] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0088] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0089] Figure 1 This is a flowchart illustrating an embodiment of the motion detection method of this disclosure;

[0090] Figure 2 This is a schematic diagram of the structure of key skeletal points shown in an embodiment of the present disclosure;

[0091] Figure 3 This is a complete flowchart of a motion detection method in a vehicle driving scenario, as shown in an embodiment of this disclosure;

[0092] Figure 4 This is a schematic diagram of the motion detection device shown in an embodiment of the present disclosure;

[0093] Figure 5 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this disclosure. Detailed Implementation

[0094] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0095] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0096] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0097] Firstly, at least one embodiment of this disclosure provides an action detection method, please refer to the appendix. Figure 1 It illustrates the process of the method, including steps S101 to S103.

[0098] This method can be used to detect actions of target objects within a video stream of a scene area. For example, it can detect whether a target object in the video stream is performing a target action, which can be a dangerous action. In other words, this method can be used to detect whether a target object in the video stream of a scene area is performing a dangerous action. This method can be applied to scenarios such as vehicle movement, where it can detect whether the driver or passengers are performing dangerous actions. Dangerous actions can be predefined; for example, dangerous actions can include leaning to the left, leaning to the right, clutching the chest, or leaning forward and falling down.

[0099] A video stream can be video recorded by an image acquisition device for a specific area of ​​a scene. For example, in a scenario where a vehicle is in motion, the scene area could be the vehicle cabin. The video stream for this scene area could be video captured by cameras installed inside the vehicle cabin; this video stream could be video captured from inside the cabin for the driver or video captured from inside the cabin for the passengers.

[0100] Alternatively, this method can be executed by electronic devices such as terminal devices or servers. Terminal devices can include user equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistant (PDA) handheld devices, computing devices, in-vehicle devices, wearable devices, etc. This method can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the method can be executed by a server, such as a local server or a cloud server. In a vehicle-moving scenario, this method can be executed by an Artificial Intelligence Emergency Call system, which connects to cameras inside the vehicle cabin to acquire video streams of the cabin scene captured by the cameras.

[0101] In step S101, the video stream of the scene area is acquired.

[0102] The video stream of the scene area can be video recorded by an image acquisition device, such as a mobile phone, camera, or webcam. The video stream of the scene area contains a target object, which refers to a person whose actions need to be detected in the video stream. The area of ​​the video stream other than the target object is the background area and other objects (which may not even exist). For example, in a vehicle driving scene, the target object could be the driver or a specific passenger. Therefore, after acquiring the video stream of the scene area, multiple objects in the video stream can be detected. Then, based on the position of each object in the vehicle cabin and / or the facial information of each object, the target object can be determined from among the multiple objects. For example, the object in the driver's seat in the vehicle cabin can be determined as the target object, i.e., the driver can be determined as the target object. Alternatively, the object whose facial information matches pre-recorded reference facial features can be determined as the target object, i.e., the vehicle owner, registered driver, or other specific individuals can be determined as the target object. The operations targeting the target object in the following steps can be performed based on the target object determined in this step.

[0103] It is understood that there can be one or more target objects in the video stream. When there are multiple target objects in the video stream, each target object can be processed sequentially according to the method provided in this embodiment, or each target object can be processed simultaneously according to the method provided in this embodiment. For example, in the video of a vehicle cabin scene, the target object may include one or more of multiple target objects such as the driver, the front passenger, and the rear seat passengers.

[0104] In one possible embodiment, when the scene area is a vehicle cabin, the activation conditions for this step can be preset. For example, the activation conditions could be that the vehicle doors are locked and / or the vehicle speed reaches a preset speed threshold. Then, the video stream of the scene area can be acquired only when the vehicle meets the above activation conditions, i.e., when the vehicle doors are locked and / or the vehicle speed reaches the preset speed threshold. This allows the video stream to be acquired only in scenarios where the vehicle has a hazard detection requirement, making the detection method more targeted, saving computing power and memory, and reducing power consumption.

[0105] In step S102, the motion direction of the target object within the scene area is detected based on multiple image frames in the video stream.

[0106] The video stream contains a preset number of image frames, such as 2, 4, or 5 frames. For example, during the acquisition of the video stream of a scene area, the latest preset number of image frames in the video stream can be cached in real time. For instance, each image frame in the video stream can be used as a cache object; or image frames can be extracted from the video stream at certain intervals as cache objects; or image frames can be extracted from the video stream according to certain caching conditions. For example, it can be detected whether each image frame in the video stream contains preset key information about the target object, where the preset key information includes at least one of a face, at least part of the body, and skeletal key points. Part of the body can be a part such as the left shoulder, right shoulder, left ear, or right ear. Then, the latest preset number of image frames containing the preset key information about the target object in the video stream can be cached in real time. Since the cached image frames contain the preset key information, the success rate of obtaining the required skeletal key points when performing target object action detection based on the cached image frames can be improved. During the image frame caching stage, images in the video stream are coarsely filtered based on the aforementioned key information. This reduces the time and memory resources required for detecting missing key skeletal points during action detection, thus improving action detection efficiency.

[0107] The target object's movement direction can be left, right, up, down, etc. In one possible embodiment, the features of the target object within each of the multiple image frames can be extracted first. For example, feature points of the target object can be extracted based on the fundamental principle that the brightness of the same target remains unchanged when it moves between different frames. Then, based on the features of the target object within each of the multiple image frames, the optical flow information of the target object in the video stream is determined, whereby the optical flow information can characterize the target's movement between different frames. Finally, based on the optical flow information of the target object in the video stream, the movement direction of the target object within the scene area is determined. For example, the LucasKanade algorithm can be used to calculate the movement direction of the target object from relative stillness to making a movement.

[0108] Optionally, triggering conditions can be set for this step to enable targeted execution. For example, during the acquisition of the video stream of the scene area, it can be detected whether the target object within the video stream is moving; and if the target object is moving, this step is then executed, i.e., based on multiple image frames in the video stream, the movement direction of the target object within the scene area is detected. This improves the effectiveness and targeting of this step.

[0109] In step S103, the skeletal key points of the target object in the image frame are detected.

[0110] A pre-trained neural network can be used to process image frames to obtain the skeletal keypoints of the target object. Skeletal keypoints represent the joints in the human skeletal structure, and these joints can be used to draw a skeletal diagram of the human body. For example, the neural network can detect skeletal keypoints and generate a skeletal diagram based on these keypoints, such as... Figure 2 As shown, from Figure 2 As can be seen, the skeletal key points include: nose key point 0, left eye key point 1, right eye key point 2, left ear key point 3, right ear key point 4, left shoulder key point 5, right shoulder key point 6, left elbow key point 7, right elbow key point 8, left wrist key point 9, right wrist key point 10, left hip joint key point 11, right hip joint key point 12, left knee key point 13, right knee key point 14, left ankle key point 15, and right ankle key point 16. It is important to note that the video stream is recorded with the camera facing directly at or at a certain angle towards the target object. Therefore, the target object in the image frame is a mirror image of the target object in the real scene; that is, the left side of the target object in the real scene is the right side of the target object in the image frame, and vice versa. The origin of the coordinate system within the image frame can be at its upper left corner. Moving horizontally to the right can be the positive direction of the horizontal axis (e.g., the x-axis), and moving vertically downwards can be the positive direction of the vertical axis (e.g., the y-axis).

[0111] The step of detecting skeletal keypoints of the target object involves identifying the skeletal keypoints of the portion of the target object appearing in the image frame. For example, if the driver's upper body is visible in the image frame, this step only detects the skeletal keypoints of the upper body. In other words, this step detects the skeletal keypoints of the target object, and can detect... Figure 2 All the key points shown, or those that can be detected. Figure 2 Some of the key points shown are shown.

[0112] The detected skeletal key points can be represented by their coordinate positions in the image frame, and the skeletal key points can also be marked at their corresponding positions in the image frame.

[0113] In step S104, the motion information of the target object is determined based on the geometric relationship between the target key points in the skeletal key points and the target key points corresponding to the motion direction.

[0114] The target object's action information can be whether the target object has a target action or not. The target action can be a dangerous action to be detected, etc., and the dangerous actions to be detected can be preset. Each action direction can correspond to one or more preset target actions, and each target action has multiple target key points. In each target action, the multiple target key points satisfy corresponding geometric relationship constraints. Therefore, each target action can be preset with a first preset condition, which can be set for the geometric relationship satisfied between the target key points corresponding to the target action. Thus, when determining the target object's action information, for each target action within the action direction detected in step S102, it can be determined whether the target key points corresponding to the target action in the skeletal key points of the target object detected in step S103 satisfy the first preset condition corresponding to the target action. If they are satisfied, the target object has the target action; otherwise, the target object does not have the target action.

[0115] For example, each action direction corresponds to a target action, so target key points and a first preset condition can be set for that action direction. Then, when determining the action information of the target object, if the geometric relationship between the detected skeletal key points and the target key points corresponding to the action direction satisfies the first preset condition corresponding to the action direction, it is determined that the target object has a target action corresponding to the action direction; otherwise (i.e., the geometric relationship between the target key points does not satisfy the corresponding first preset condition), it is determined that the target object does not have a target action corresponding to the action direction. Narrowing the detection range of the target action based on the action direction further saves energy and memory, improving detection efficiency; moreover, targeted detection of target key points according to the action direction makes action detection more targeted, further improving detection accuracy.

[0116] In one possible embodiment, the target actions corresponding to the four action directions of left, right, up and down are preset as leaning the body to the left (corresponding to the direction "left"), leaning the body to the right (corresponding to the direction "right"), covering the chest (corresponding to the direction "up") and leaning forward and lying down (corresponding to the direction "down"), and a corresponding target key point and a first preset condition are set for each target action (i.e. each action direction).

[0117] The target key points corresponding to the leftward tilt of the body can be set as the right shoulder key points and the left shoulder key points. The vector formed by the line connecting the right shoulder key points to the left shoulder key points is called the first target vector. The vector pointing horizontally to the right (i.e., parallel to the horizontal edge of the image to be detected and pointing to the right) is called the standard vector. The corresponding first preset condition is set as follows: the tangent of the angle between the target vector and the standard vector is positive, and the absolute value of the tangent is greater than a first threshold (e.g., the first threshold is 0.4). Figure 2 Taking the right shoulder keypoint 6 and left shoulder keypoint 5 as examples, the first preset condition corresponding to the leftward body tilt can be expressed as tan(vec(6,5))>0.4. That is, when the direction of movement is to the left, if the tangent of the angle between the first target vector from the right shoulder keypoint to the left shoulder keypoint and the standard vector to the right is positive and the absolute value of the tangent is greater than the first threshold, it is determined that the target object has a leftward body tilt.

[0118] The target key points corresponding to the body tilting to the right can be set as the right shoulder key point and the left shoulder key point. The vector formed by the line connecting the right shoulder key point and the left shoulder key point is called the target vector. The vector pointing horizontally to the right (i.e., parallel to the horizontal edge of the image to be detected and pointing to the right) is called the standard vector. The corresponding first preset condition is set as follows: the tangent of the angle between the target vector and the standard vector is negative, and the absolute value of the tangent is greater than a first threshold (e.g., the first threshold is 0.4). Figure 2Taking the right shoulder keypoint 6 and left shoulder keypoint 5 as examples, the first preset condition corresponding to the rightward body tilt can be expressed as tan(vec(6,5)) < -0.4. That is, when the direction of movement is to the right, if the tangent of the angle between the first target vector from the right shoulder keypoint to the left shoulder keypoint and the standard vector to the right is negative, and the absolute value of the tangent is greater than the first threshold, it is determined that the target object has a rightward body tilt.

[0119] The target key points corresponding to covering the chest can be set as the left elbow key point, left wrist key point, left shoulder key point, right elbow key point, right wrist key point, and right shoulder key point. The vector formed by the line connecting the left elbow key point to the left wrist key point is called the second target vector; the vector formed by the line connecting the left elbow key point to the left shoulder key point is called the third target vector; the vector formed by the line connecting the right elbow key point to the right wrist key point is called the fourth target vector; and the vector formed by the line connecting the right elbow key point to the right shoulder key point is called the fifth target vector. The second and third target vectors are then compared... The cosine of the angle between the two targets is called the first cosine value, and the cosine of the angle between the fourth and fifth target vectors is called the second cosine value. The corresponding first preset condition is set as follows: the first cosine value is greater than the second threshold (e.g., the second threshold is 0.2), and the vertical distance between the left wrist key point and the right shoulder key point is greater than the third threshold (e.g., the third threshold is 100); and / or, the second cosine value is greater than the second threshold (e.g., the second threshold is 0.2), and the vertical distance between the right wrist key point and the left shoulder key point is greater than the third threshold (e.g., the third threshold is 100). Figure 2Taking the left elbow key point 7, left wrist key point 9, left shoulder key point 5, right elbow key point 8, right wrist key point 10, and right shoulder key point 6 as examples, the first preset condition corresponding to covering the chest can be expressed as cos(vec(7,5), vec(7,9)) > 0.2 and y(9) - y(6) > 100, and / or, the first preset condition can be expressed as cos(vec(8,6), vec(8,10)) > 0.2 and y(10) - y(5) > 100. In other words, when the direction of the action is upward, the target object is determined to be making a chest-covering action in response to either the first or second scenario. The first scenario includes: the angle between the second and third target vectors is greater than a second threshold, the left wrist key point is lower than the right shoulder key point, and the vertical distance between the left wrist key point and the right shoulder key point is greater than a third threshold, wherein the second target vector is the vector from the left elbow key point to the left wrist key point, and the third target vector is the vector from the left elbow key point to the left shoulder key point. The second scenario includes: the angle between the fourth and fifth target vectors is greater than a second threshold, the right wrist key point is lower than the left shoulder key point, and the vertical distance between the right wrist key point and the left shoulder key point is greater than a third threshold, wherein the fourth target vector is the vector from the right elbow key point to the right wrist key point, and the fifth target vector is the vector from the right elbow key point to the right shoulder key point.

[0120] The target key points corresponding to the forward-leaning prone position can be set as the left ear key point, left shoulder key point, right ear key point, and right shoulder key point. The corresponding first preset condition is set as follows: the vertical distance between the left ear key point and the left shoulder key point is greater than a fourth threshold (e.g., the fourth threshold is 50), and / or, the vertical distance between the right ear key point and the right shoulder key point is greater than a fourth threshold (e.g., the fourth threshold is 50). Figure 2 Taking the left ear key point 3, left shoulder key point 5, right ear key point 4, and right shoulder key point 6 shown as examples, the first preset condition corresponding to leaning forward and lying down can be expressed as y(3)-y(5)>50, and / or y(4)-y(6)>50. That is to say, when the direction of the movement is downward, in response to the left ear key point being lower than the left shoulder key point and the vertical distance between the left ear key point and the left shoulder key point being greater than the fourth threshold, and / or in response to the right ear key point being lower than the right shoulder key point and the vertical distance between the right ear key point and the right shoulder key point being greater than the fourth threshold, it is determined that the target object has a leaning forward and lying down movement.

[0121] It is understandable that when the action information of the target object indicates that the target object is performing a target action, an alarm message can be sent to the service platform. For example, if the target action is a dangerous action that needs to be detected, an alarm message can be sent to the service platform when the target object performs a dangerous action. In a vehicle driving scenario, the service platform can be a vehicle operation service platform, such as a ride-hailing service platform, or it can be a medical platform. After receiving the alarm message, the service platform can take rescue measures, thereby improving the efficiency of rescue for drivers in danger, and thus improving the rescue effect and better protecting the lives of people in the vehicle. Furthermore, it is understandable that as the video stream of the area scene is recorded, the action information of the target object can be determined in real time or at a certain frequency. Therefore, within a preset time period, if the action information continuously indicates that the target object is performing a target action, an alarm message can be sent to the service platform, thereby reducing the possibility of false alarm messages being sent due to fluctuations in the action information detection results.

[0122] As described in the above embodiments, by acquiring a video stream of a scene area, the movement direction of a target object within the scene area can be detected based on multiple image frames in the video stream. Then, the skeletal key points of the target object in the image frames can be detected. Finally, the movement information of the target object can be determined based on the geometric relationship between the skeletal key points and the target key points corresponding to the movement direction. Since the geometric relationship between the target key points is relatively objective and accurate, it can accurately detect whether the target object is performing a dangerous action. Moreover, it detects target key points specifically according to the movement direction, thus making the action detection more targeted and further improving the accuracy of the detection. If this detection method is applied to a vehicle, it can accurately detect whether the driver and passengers inside the vehicle are in danger, thereby improving passenger safety and enhancing the user experience.

[0123] In some embodiments of this disclosure, appendix Figure 1 The step S103 shown can detect the skeletal key points of a target object in one of the multiple image frames of the video stream, that is, detect the skeletal key points of the target object in one of the multiple image frames in step S102. The specific image frames to be detected can be preset, such as the first frame, the last frame, or intermediate frames. For example, when the latest preset number of image frames in the video stream are cached in real time, the skeletal key points of the target object in one of the latest preset number of image frames can be detected in step S103.

[0124] Based on this, attached Figure 1In step S104 shown, the motion information of the target object can be determined directly based on the geometric relationship between the target key points corresponding to the motion direction among the detected skeletal key points. For the specific determination method, please refer to the detailed description of step S104 in the above embodiment.

[0125] In other embodiments of this disclosure, appendix Figure 1 The illustrated step S103 can detect the skeletal key points of the target object in each of the multiple image frames in the video stream, that is, detect the skeletal key points of the target object in each of the multiple image frames in step S102. For example, in the case of caching the latest preset number of image frames in the video stream in real time, the skeletal key points of the target object in each of the latest preset number of image frames can be detected in step S103.

[0126] Based on this, attached Figure 1 In step S104, the motion information of the target object in the corresponding image frame can be determined first based on the geometric relationship between the target key points corresponding to the action direction detected in the skeletal key points of each image frame in the video stream. Then, the motion information of the target object in the multiple image frames is smoothed to obtain the motion information of the target object determined based on the multiple image frames. For example, when the latest preset number of image frames in the video stream are cached in real time, the motion information of the target object in the cached preset number of image frames is smoothed to obtain the motion information of the target object determined based on the cached preset number of image frames.

[0127] The motion information detected in each image frame can be input into a smoothing queue. Then, a smoothing window is set, the size of which can be the number of multiple image frames in the video stream, such as a preset number of image frames in a cache. The smoothing window is then moved as the smoothing queue is updated, and after each movement of the smoothing window, the motion information of the target object determined based on the multiple image frames is determined according to the multiple motion information in the smoothing window. This achieves smoothing processing of motion information and improves the effectiveness and stability of motion information.

[0128] If the motion information detected in each of multiple image frames is the same, then it can be determined as the motion information of the target object determined based on the multiple image frames. For example, if the motion information detected in each of the five image frames is leaning to the left, then the leaning-to-the-left motion can be determined as the motion information of the target object determined based on the aforementioned five image frames.

[0129] If the motion information detected in multiple image frames is inconsistent, the motion information with the highest frequency is taken as the motion information of the target object determined based on the multiple image frames. For example, if the motion information detected in 4 out of 5 image frames is "leaning to the left," and the motion information detected in 1 image frame is "no target motion," then "leaning to the left" can be determined as the motion information of the target object determined based on the above 5 image frames. Optionally, if the motion information detection result with the highest frequency in the motion information detection results of multiple image frames within the smoothing window does not exceed a preset proportion (e.g., 50%), the detection results within the smoothing window can be discarded to further improve the reliability of the motion detection results.

[0130] It should be noted that, in order to accurately determine the motion information of the target object based on the multiple image frames, the number of multiple image frames can be set to an odd number such as 3, 5, 7, etc. If it is set to an even number and multiple motion information with the same quantity appear, then the latest motion information is taken as the motion information of the target object determined based on the multiple image frames.

[0131] Please refer to the appendix. Figure 3 This example illustrates the complete flow of a motion detection method in a vehicle driving scenario. From Figure 3 As can be seen from the process, firstly, step S11 is executed, and if the vehicle status meets the activation conditions of the method, the emergency call function in the cabin is activated; then step S12 is executed, and the camera inside the vehicle collects passenger information in the cabin, that is, the video stream of the passengers in the cabin; then step S13 is executed, and more than two of the most recent images are saved to the cache; then step S14 is executed, and the direction of motion is determined using the optical flow method; then step S15 is executed, and motion detection is performed in the space corresponding to the direction of motion (that is, the range of the target action corresponding to the direction of motion), that is, to detect whether the target object has a target action corresponding to the direction of motion; then step S16 is executed, and the multiple detection results obtained in step S15 are smoothed; finally, step S17 is executed, and if a dangerous action is detected and the dangerous action continues for a period of time, a distress signal is issued based on the smoothing result of S15.

[0132] The motion detection method provided in this embodiment combines motion geometry features, optical flow detection processing, and some objective factors to more accurately and comprehensively assess the current state of a passenger's motion information. Simultaneously, it employs smoothing algorithms and caching to effectively handle abrupt and fluctuating results, providing important reference data for car rental companies and traffic regulatory departments. This allows for targeted customization of safety plans and operational management, ultimately improving the life, health, and safety of passengers in the vehicle.

[0133] According to a second aspect of the embodiments of this disclosure, an action detection device is provided. Please refer to the appendix. Figure 4The device includes:

[0134] Acquisition module 401 is used to acquire the video stream of the scene area;

[0135] Direction module 402 is used to detect the motion direction of a target object within the scene area based on multiple image frames in the video stream;

[0136] Detection module 403 is used to detect the skeletal key points of the target object in the image frame;

[0137] The determination module 404 is used to determine the motion information of the target object based on the geometric relationship between the target key points corresponding to the motion direction among the detected skeletal key points.

[0138] In some embodiments of this disclosure, the direction module is specifically used for:

[0139] Extract the features of the target object within each of the multiple image frames;

[0140] Based on the characteristics of the target object within each of the plurality of image frames, the optical flow information of the target object in the video stream is determined;

[0141] Based on the optical flow information of the target object in the video stream, the motion direction of the target object within the scene area is determined.

[0142] In some embodiments of this disclosure, each action direction corresponds to a target action;

[0143] The determining module is specifically used for:

[0144] If the geometric relationship between the detected skeletal key points and the target key points corresponding to the action direction satisfies the first preset condition corresponding to the action direction, it is determined that the target object has a target action corresponding to the action direction.

[0145] Otherwise, it is determined that the target object does not have a target action corresponding to the action direction.

[0146] In some embodiments of this disclosure, the skeletal key points include: left shoulder key point, right shoulder key point, left wrist key point, right wrist key point, left elbow key point, right elbow key point, left ear key point, and right ear key point.

[0147] The determining module is specifically used for:

[0148] When the movement direction is left, if the tangent of the angle between the first target vector from the right shoulder key point to the left shoulder key point and the standard vector to the right is positive and the absolute value of the tangent is greater than a first threshold, it is determined that the target object has a leftward leaning movement; and / or,

[0149] When the movement direction is to the right, if the tangent of the angle between the first target vector from the right shoulder key point to the left shoulder key point and the standard vector to the right is negative, and the absolute value of the tangent is greater than a first threshold, it is determined that the target object has a rightward leaning movement; and / or,

[0150] When the action direction is upward, the target object is determined to be making a chest-covering motion in response to either the first or second scenario below; the first scenario includes: the angle between the second and third target vectors is greater than a second threshold, the left wrist key point is lower than the right shoulder key point, and the vertical distance between the left wrist key point and the right shoulder key point is greater than a third threshold, wherein the second target vector is the vector from the left elbow key point to the left wrist key point, and the third target vector is the vector from the left elbow key point to the left shoulder key point; the second scenario includes: the angle between the fourth and fifth target vectors is greater than a second threshold, the right wrist key point is lower than the left shoulder key point, and the vertical distance between the right wrist key point and the left shoulder key point is greater than a third threshold; wherein the fourth target vector is the vector from the right elbow key point to the right wrist key point, and the fifth target vector is the vector from the right elbow key point to the right shoulder key point; and / or,

[0151] When the direction of the movement is downward, in response to the left ear key point being lower than the left shoulder key point and the vertical distance between the left ear key point and the left shoulder key point being greater than the fourth threshold, and / or in response to the right ear key point being lower than the right shoulder key point and the vertical distance between the right ear key point and the right shoulder key point being greater than the fourth threshold, it is determined that the target object has a forward leaning and lying down movement.

[0152] In some embodiments of this disclosure, the direction module is specifically used for:

[0153] Detect whether the target object within the video stream is moving;

[0154] When the target object in the video stream is moving, the direction of motion of the target object in the scene area is detected based on multiple image frames in the video stream.

[0155] In some embodiments of this disclosure, the determining module is specifically used for:

[0156] Based on the geometric relationship between the target key points corresponding to the action direction detected in the skeletal key points in each of the multiple image frames in the video stream, the action information of the target object in the corresponding image frame is determined.

[0157] The motion information of the target object in the multiple image frames is smoothed to obtain the motion information of the target object determined based on the multiple image frames.

[0158] In some embodiments of this disclosure, a caching module is also included for:

[0159] During the process of acquiring the video stream of the scene area, the latest preset number of image frames in the video stream are cached in real time.

[0160] The determining module is used to smooth the motion information of the target object in the multiple image frames to obtain the motion information of the target object determined based on the multiple image frames. Specifically, it is used to:

[0161] The motion information of the target object in the cached preset number of image frames is smoothed to obtain the motion information of the target object determined based on the cached preset number of image frames.

[0162] In some embodiments of this disclosure, the caching module is specifically used for:

[0163] Detect whether there is preset key information of the target object in each image frame of the video stream, wherein the preset key information includes at least one of face, at least part of body and skeletal key points;

[0164] The latest preset number of image frames in the video stream containing preset key information of the target object are cached in real time.

[0165] In some embodiments of this disclosure, the scene area includes a vehicle cabin scene area;

[0166] The acquisition module is specifically used for:

[0167] When the vehicle doors are locked and / or the vehicle speed reaches a preset speed threshold, acquire the video stream of the scene area.

[0168] In some embodiments of this disclosure, a target module is also included, for:

[0169] Detect multiple objects in the video stream;

[0170] The target object is determined from the plurality of objects based on the location of each object within the vehicle cabin and / or the facial information of each object.

[0171] In some embodiments of this disclosure, an alarm module is also included, for:

[0172] When the action information of the target object indicates that the target object has a target action, an alarm message is sent to the service platform.

[0173] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the third aspect of the embodiments concerning the method, and will not be elaborated upon here.

[0174] Thirdly, at least one embodiment of this disclosure provides an apparatus, please refer to the appendix. Figure 5 The diagram illustrates the structure of the device, which includes a memory and a processor. The memory stores computer instructions that can run on the processor, and the processor detects actions based on the method described in any of the first aspects when executing the computer instructions.

[0175] Fourthly, at least one embodiment of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the first aspects.

[0176] In this disclosure, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "multiple" refers to two or more unless otherwise expressly defined.

[0177] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0178] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A motion detection method, characterized in that, include: Obtain the video stream of the scene area; Based on multiple image frames in the video stream, the motion direction of the target object within the scene area is detected; Detect the skeletal key points of the target object in the image frame, wherein the skeletal key points include the left shoulder key point and the right shoulder key point; Based on the geometric relationship between the target key points in the detected skeletal key points and the target key points corresponding to the direction of movement, the motion information of the target object is determined; The step of determining the motion information of the target object based on the geometric relationship between the target key points corresponding to the motion direction among the detected skeletal key points includes: When the movement direction is left, if the tangent of the angle between the first target vector from the right shoulder key point to the left shoulder key point and the standard vector to the right is positive and the absolute value of the tangent is greater than a first threshold, it is determined that the target object has a leftward leaning movement; and / or, When the direction of the movement is to the right, if the tangent of the angle between the first target vector from the right shoulder key point to the left shoulder key point and the standard vector to the right is negative, and the absolute value of the tangent is greater than the first threshold, it is determined that the target object has a rightward body tilting movement.

2. The motion detection method according to claim 1, characterized in that, The step of detecting the motion direction of a target object within the scene region based on multiple image frames in the video stream includes: Extract the features of the target object within each of the multiple image frames; Based on the characteristics of the target object within each of the plurality of image frames, the optical flow information of the target object in the video stream is determined; Based on the optical flow information of the target object in the video stream, the motion direction of the target object within the scene area is determined.

3. The motion detection method according to claim 1, characterized in that, Each direction of action corresponds to a target action; The step of determining the motion information of the target object based on the geometric relationship between the target key points corresponding to the motion direction among the detected skeletal key points includes: If the geometric relationship between the detected skeletal key points and the target key points corresponding to the action direction satisfies the first preset condition corresponding to the action direction, it is determined that the target object has a target action corresponding to the action direction. Otherwise, it is determined that the target object does not have a target action corresponding to the action direction.

4. The motion detection method according to claim 1 or 3, characterized in that, The key skeletal points include: key points of the left wrist, right wrist, left elbow, right elbow, left ear, and right ear. The step of determining the motion information of the target object based on the geometric relationship between the target key points corresponding to the motion direction among the detected skeletal key points includes: When the action direction is upward, the target object is determined to be making a chest-covering motion in response to either the first or second scenario below; the first scenario includes: the angle between the second and third target vectors is greater than a second threshold, the left wrist key point is lower than the right shoulder key point, and the vertical distance between the left wrist key point and the right shoulder key point is greater than a third threshold, wherein the second target vector is the vector from the left elbow key point to the left wrist key point, and the third target vector is the vector from the left elbow key point to the left shoulder key point; the second scenario includes: the angle between the fourth and fifth target vectors is greater than a second threshold, the right wrist key point is lower than the left shoulder key point, and the vertical distance between the right wrist key point and the left shoulder key point is greater than a third threshold; wherein the fourth target vector is the vector from the right elbow key point to the right wrist key point, and the fifth target vector is the vector from the right elbow key point to the right shoulder key point; and / or, When the direction of the movement is downward, in response to the left ear key point being lower than the left shoulder key point and the vertical distance between the left ear key point and the left shoulder key point being greater than the fourth threshold, and / or in response to the right ear key point being lower than the right shoulder key point and the vertical distance between the right ear key point and the right shoulder key point being greater than the fourth threshold, it is determined that the target object has a forward leaning and lying down movement.

5. The motion detection method according to any one of claims 1 to 4, characterized in that, The step of detecting the motion direction of a target object within the scene region based on multiple image frames in the video stream includes: Detect whether the target object within the video stream is moving; When the target object in the video stream is moving, the direction of motion of the target object in the scene area is detected based on multiple image frames in the video stream.

6. The motion detection method according to claim 1, 3, or 4, characterized in that, The step of determining the motion information of the target object based on the geometric relationship between the target key points corresponding to the motion direction among the detected skeletal key points includes: Based on the geometric relationship between the target key points corresponding to the action direction detected in the skeletal key points in each of the multiple image frames in the video stream, the action information of the target object in the corresponding image frame is determined. The motion information of the target object in the multiple image frames is smoothed to obtain the motion information of the target object determined based on the multiple image frames.

7. The motion detection method according to claim 6, characterized in that, Also includes: During the process of acquiring the video stream of the scene area, the latest preset number of image frames in the video stream are cached in real time. The step of smoothing the motion information of the target object in the multiple image frames to obtain the motion information of the target object determined based on the multiple image frames includes: The motion information of the target object in the cached preset number of image frames is smoothed to obtain the motion information of the target object determined based on the cached preset number of image frames.

8. The motion detection method according to claim 7, characterized in that, The real-time cache of the latest preset number of image frames includes: Detect whether there is preset key information of the target object in each image frame of the video stream, wherein the preset key information includes at least one of face, at least part of body and skeletal key points; The latest preset number of image frames in the video stream containing preset key information of the target object are cached in real time.

9. The motion detection method according to any one of claims 1 to 8, characterized in that, The scene area includes the vehicle cabin scene area; The acquisition of the video stream of the scene area includes: When the vehicle doors are locked and / or the vehicle speed reaches a preset speed threshold, acquire the video stream of the scene area.

10. The motion detection method according to claim 9, characterized in that, Also includes: Detect multiple objects in the video stream; The target object is determined from the plurality of objects based on the location of each object within the vehicle cabin and / or the facial information of each object.

11. The motion detection method according to any one of claims 1 to 10, characterized in that, Also includes: When the action information of the target object indicates that the target object has a target action, an alarm message is sent to the service platform.

12. A motion detection device, characterized in that, include: The acquisition module is used to acquire the video stream of the scene area; The direction module is used to detect the motion direction of a target object within the scene area based on multiple image frames in the video stream; The detection module is used to detect the skeletal key points of the target object in the image frame, wherein the skeletal key points include the left shoulder key point and the right shoulder key point; The determination module is used to determine the motion information of the target object based on the geometric relationship between the target key points corresponding to the motion direction among the detected skeletal key points; The determining module is used for: When the movement direction is left, if the tangent of the angle between the first target vector from the right shoulder key point to the left shoulder key point and the standard vector to the right is positive and the absolute value of the tangent is greater than a first threshold, it is determined that the target object has a leftward leaning movement; and / or, When the direction of the movement is to the right, if the tangent of the angle between the first target vector from the right shoulder key point to the left shoulder key point and the standard vector to the right is negative, and the absolute value of the tangent is greater than the first threshold, it is determined that the target object has a rightward body tilting movement.

13. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store computer instructions executable on the processor, and the processor being used to implement the method of any one of claims 1 to 11 when executing the computer instructions.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method described in any one of claims 1 to 11.

Images