Motion detection method, apparatus, electronic device and storage medium

Abstract

The present disclosure relates to a motion detection method, an apparatus, an electronic device, and a storage medium, the motion detection method including: acquiring an image of a target object; detecting skeletal keypoints on the image of the target object to obtain detection information of skeletal keypoints of the target object; detecting facial keypoints on the image of the target object to obtain detection information of facial keypoints of the target object; acquiring an estimation result of motion information of the target object based on a geometric relationship between target keypoints in the detection information of skeletal keypoints; and verifying the estimation result based on the detection information of the facial keypoints to obtain a motion detection result of the target object.

Description

[Technical field]

[0001] <CROSS REFERENCE TO RELATED APPLICATIONS> This disclosure claims priority to a Chinese patent application bearing application number 202210346614.X, filed with the China National Intellectual Property Office on March 31, 2022, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the technical field of image detection, and in particular to a motion detection method, apparatus, electronic device and storage medium. [Background technology]

[0003] With the development of artificial intelligence technology, the types of image and video detection are becoming more and more numerous, and the effects are becoming more and more excellent. In particular, detection technology can be applied to the field of safety protection to help users avoid danger. Taking the scene of a vehicle cabin as an example, image processing technology can be used to detect the vehicle cabin environment and human safety. By collecting images and videos of the vehicle interior during the driving process or parking state, it can be detected whether the person in the vehicle is performing dangerous actions, thereby improving driving and riding safety. However, in the conventional technology, human action detection is prone to erroneous judgment, resulting in a poor user experience. Summary of the Invention [Means for solving the problem]

[0004] The present disclosure provides a motion detection method, apparatus, device and storage medium to address the deficiencies of the prior art.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a motion detection method, the motion detection method comprising: Obtaining an image of a target object; Performing skeleton keypoint detection on the image of the target object to obtain skeleton keypoint detection information of the target object; performing face keypoint detection on the image of the target object to obtain face keypoint detection information of the target object; Obtaining an estimation result of motion information of the target object based on a geometric relationship between object keypoints in the detection information of the skeleton keypoints; and verifying the estimation result based on the detection information of the face key points to obtain a motion detection result of the target object.

[0006] In one embodiment, obtaining an estimation result of motion information of the target object based on a geometric relationship between target keypoints in the detection information of the skeleton keypoints includes: identifying the estimated result of the target object's motion information as a motion of the target object when a geometric relationship between the target keypoints in the detection information of the skeleton keypoints satisfies a first preset condition corresponding to a motion of the target object; Verifying the estimation result based on detection information of the face key points to obtain a motion detection result of the target object, If the inference result is that a target movement is present, identifying the detection result of the target object as the presence of the target movement in response to a target face key point corresponding to the target movement in the face key point detection information satisfying a second predetermined condition corresponding to the target movement.

[0007] In one embodiment, identifying the target object detection result as the presence of the target motion in response to a target face keypoint corresponding to the target motion in the face keypoint detection information satisfying a second preset condition corresponding to the target motion includes: If the target motion is a left or right tilt of the body, identifying the target object motion detection result as indicating the presence of a left or right tilt of the body in response to the positional relationship between a plurality of face key points of the first target corresponding to the left or right tilt of the body in the detection information of the face key points satisfying a second preset condition corresponding to the left or right tilt of the body.

[0008] In one embodiment, the plurality of first object's face key points include a left outer eye key point and a right outer eye key point; When the motion of the target is a leftward or rightward tilt of the body, specifying the motion detection result of the target target as a motion of the leftward or rightward tilt of the body in response to a positional relationship between a plurality of face key points of the first target corresponding to the leftward or rightward tilt of the body in the detection information of the face key points satisfying a second preset condition corresponding to the leftward or rightward tilt of the body, When the motion of the target is a left lean of the body, the tangent value of the angle between a first target vector from a key point on the outer right eye of the target target to a key point on the outer left eye of the target target and a standard vector in a horizontal rightward direction is a positive number, and the absolute value of the tangent value is greater than a first threshold value, and accordingly, the motion detection result of the target target is determined as the presence of a left lean of the body; or If the target motion is a right body tilt, the tangent value of the angle between a first target vector from a key point on the outer right eye of the target target to a key point on the outer left eye of the target target and a standard vector in a horizontal rightward direction is negative and the absolute value of the tangent value is greater than a first threshold value, and accordingly, the motion detection result of the target object is determined to indicate the presence of a right body tilt motion.

[0009] In one embodiment, identifying the target object detection result as the presence of the target motion in response to a target face keypoint corresponding to the target motion in the face keypoint detection information satisfying a second preset condition corresponding to the target motion includes: If the motion of the target is leaning forward prone, the method includes identifying the motion detection result of the target target as indicating that the motion of leaning forward prone is present in response to the absence of a facial key point of a second target corresponding to the leaning forward prone in the detection information of the facial key points.

[0010] In one embodiment, the method further comprises: Identifying a missing face key point of the second subject corresponding to the forward-leaning prone position when it is determined that the number of face content key points including face key points other than face edge key points detected based on the detection information of the face key points does not exceed a preset number; or When it is determined that the number of facial key points detected based on the detection information of the facial key points does not exceed a predetermined number, identifying a missing facial key point of the second subject corresponding to the forward leaning prone position.

[0011] In one embodiment, acquiring an image of the target object comprises: sequentially obtaining cached frame images of the target object from a video; After verifying the estimation result based on the detection information of the face key points and obtaining the motion detection result of the target object, the method further comprises: acquiring a motion detection result of the target object obtained based on a preset number of latest frame images of the target object; Identifying the motion detection result that occurs most frequently among the motion detection results of the target object obtained based on a latest preset number of frame images of the target object as the current motion of the target object.

[0012] In one embodiment, before sequentially obtaining each cached frame image of a target object from the video, further comprising: Detecting the target object based on each frame image in the video or tracking the target object; identifying a plurality of images including the target object based on the detection or tracking result of the target object; and adding each frame image, which contains predetermined key information including at least one of the key points of the face, at least a portion of the body, and the skeleton of the target object, to a cache to be used as an image of the target object.

[0013] In one embodiment, obtaining an estimation result of motion information of the target object based on a geometric relationship between target keypoints in the detection information of the skeleton keypoints includes: detecting a direction of motion of the target object based on a plurality of frame images of the target object in the cache; and obtaining an estimation result of motion information of the target object based on a geometric relationship between object keypoints corresponding to the movement directions in the detection information of the skeletal keypoints.

[0014] In one embodiment, the image of the target object includes an image of a target object within a vehicle interior; Acquiring an image of the target object includes: capturing an image of the target object when the vehicle doors are locked and / or the vehicle speed reaches a pre-set speed threshold.

[0015] In one embodiment, further comprising: acquiring an image of an interior of a vehicle and detecting a plurality of objects in the image; and identifying the target object from the plurality of objects based on a position within the vehicle cabin of each of the plurality of objects and / or facial information of each of the plurality of objects.

[0016] In one embodiment, further comprising: and sending alarm information to a service platform when the target object motion detection result is that a target motion exists in the target object.

[0017] According to a second aspect of the embodiment of the present disclosure, there is provided a motion detection device, the motion detection device comprising: an acquisition module for acquiring an image of a target object; A first detection module for detecting skeleton key points on the image of the target object to obtain detection information of skeleton key points of the target object; A second detection module for detecting face key points on the image of the target object to obtain face key point detection information of the target object; an estimation module for obtaining an estimation result of motion information of the target object based on a geometric relationship between target keypoints in the detection information of the skeleton keypoints; and a verification module for verifying the estimation result based on the detection information of the face key points to obtain a motion detection result of the target object.

[0018] In one embodiment, the estimation module specifically: and identifying the estimated result of the motion information of the target object as a motion of the target object when a geometric relationship between the target key points in the detection information of the skeleton key points satisfies a first preset condition corresponding to a motion of the target object; Specifically, the verification module: If the inference result is that a target movement is present, the detection result of the target target is identified as the presence of the target movement in response to a target face key point corresponding to the target movement in the face key point detection information satisfying a second predetermined condition corresponding to the target movement.

[0019] In one embodiment, the verification module, when identifying the detection result of the target object as having a target motion according to the target face keypoints corresponding to the target motion in the face keypoint detection information satisfying a second preset condition corresponding to the target motion, specifically: When the target movement is a left or right tilt of the body, the positional relationship between a plurality of face key points of the first target corresponding to the left or right tilt of the body in the detection information of the face key points satisfies a second preset condition corresponding to the left or right tilt of the body, thereby identifying the movement detection result of the target target as including a left or right tilt of the body.

[0020] In one embodiment, the plurality of first object's face key points include a left outer eye key point and a right outer eye key point; When the motion of the target is a left leaning of the body or a right leaning of the body, the verification module, in response to a positional relationship between a plurality of face key points of the first target corresponding to the left leaning of the body or the right leaning of the body in the detection information of the face key points satisfying a second preset condition corresponding to the left leaning of the body or the right leaning of the body, specifies the motion detection result of the target target as including a left leaning motion of the body or a right leaning motion of the body, specifically, If the motion of the target is a left lean of the body, the tangent value of the angle between a first target vector from a key point on the outer right eye of the target target to a key point on the outer left eye of the target target and a standard vector in a horizontal rightward direction is a positive number, and the absolute value of the tangent value is greater than a first threshold value, and / or, When the target motion is a right tilt of the body, the tangent value of the angle between a first target vector from a key point on the outer right eye of the target target to a key point on the outer left eye of the target target and a standard vector in a horizontal rightward direction is negative, and the absolute value of the tangent value is greater than a first threshold value, thereby identifying the motion detection result of the target target as the presence of a right tilt of the body.

[0021] In one embodiment, the verification module is for identifying the detection result of the target object as having a target motion according to a target face key point corresponding to the target motion in the face key point detection information satisfying a second preset condition corresponding to the target motion, specifically: When the motion of the target is leaning forward prone, the motion detection result of the target target is identified as including the motion of leaning forward prone depending on the absence of a facial key point of a second target corresponding to the motion of leaning forward prone in the detection information of the facial key points.

[0022] In one embodiment, the method further includes a determination module, the determination module comprising: When it is determined that the number of face content key points including face key points other than face edge key points detected based on the face key point detection information does not exceed a preset number, it determines the absence of face key points of the second object corresponding to the forward leaning prone position, or When it is determined that the number of facial key points detected based on the detection information of the facial key points does not exceed a predetermined number, the method identifies a missing facial key point of the second subject corresponding to the forward leaning prone position.

[0023] In one embodiment, the acquisition module specifically: For sequentially obtaining cached frame images of the target object from a video.

[0024] The apparatus further includes a smoothing module, the smoothing module comprising: Verify the estimation result based on the detection information of the face key points, and obtain a motion detection result of the target object. Then, obtain the motion detection result of the target object obtained based on a latest predetermined number of frame images of the target object; The motion detection result that occurs most frequently among the motion detection results of the target object obtained based on the latest preset number of frame images of the target object is identified as the current motion of the target object.

[0025] In one embodiment, the apparatus further comprises a mitigation module, the mitigation module comprising: Before sequentially acquiring each of the cached frame images of the target object from the video, identify a plurality of images including the target object based on the detection or tracking result of the target object; The purpose is to add each frame image containing predetermined key information including at least one of the face, at least a portion of the body, and skeletal key points of the target object among the multiple images containing the target object to a cache to make it an image of the target object.

[0026] In one embodiment, the estimation module specifically: Detecting a direction of motion of the target object based on a plurality of frame images of the target object in the cache; The method is to obtain an estimation result of motion information of the target object based on a geometric relationship between object key points corresponding to the movement direction in the detection information of the skeleton key points.

[0027] In one embodiment, the image of the target object includes an image of a target object within a vehicle interior; The acquisition module specifically includes: and for capturing an image of the target object when the vehicle doors are locked and / or the vehicle speed reaches a pre-set speed threshold.

[0028] In one embodiment, the object module further comprises: acquiring an image of an interior of a vehicle; and detecting a plurality of objects in the image; The target object is identified from the plurality of objects based on a position of each of the plurality of objects within the vehicle cabin and / or facial information of each of the plurality of objects.

[0029] In one embodiment, the system further includes an alarm module, the alarm module comprising: and sending alarm information to a service platform when the target object motion detection result is that a target motion exists in the target object.

[0030] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, the device comprising a memory and a processor, the memory for storing computer instructions executable by the processor, and for performing a method as described in the first aspect when the processor executes the computer instructions.

[0031] According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium having a computer program stored therein, the computer program being executed by a processor to perform the method according to the first aspect.

[0032] As can be seen from the above embodiment, the image of the target object is obtained, and the detection information of the skeleton keypoints of the target object can be obtained by detecting the skeleton keypoints of the image of the target object, or the detection information of the face keypoints of the target object can be obtained by detecting the face keypoints of the image of the target object, and thus the estimation result of the motion information of the target object can be obtained based on the geometric relationship between the target keypoints in the detection information of the skeleton keypoints, and the estimation result can be verified based on the detection information of the face keypoints to obtain the motion detection result of the target object. The geometric relationship between the target keypoints can objectively and accurately detect whether the target object has made a dangerous motion, and the detection information of the face keypoints can further verify the estimation result of the motion information obtained based on the target keypoints, thereby further improving the accuracy of the motion detection. When the detection method is applied in a vehicle, it can accurately detect whether a danger occurs to the driver and passengers in the vehicle, thereby improving the safety of the ride and improving the user's experience.

[0033] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. [Brief description of the drawings]

[0035] [Figure 1] 1 shows a flowchart of a motion detection method according to an embodiment of the present disclosure. [Diagram 2]FIG. 2 shows a schematic diagram of the structure of key points of the framework of one embodiment of the present disclosure. [Diagram 3] FIG. 2 shows a schematic diagram of face keypoint configuration according to an embodiment of the present disclosure. [Figure 4] 1 shows a complete flowchart of a method for detecting motion in a vehicle driving scene according to one embodiment of the present disclosure. [Diagram 5] 1 shows a schematic diagram of a configuration of a motion detection device according to an embodiment of the present disclosure. [Figure 6] 1 shows a schematic diagram of a configuration of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] Exemplary embodiments will now be described in detail herein, examples of which are illustrated in the accompanying drawings. Where the following description refers to the drawings, like numerals in different drawings represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as described in the appended claims.

[0037] The terms used in this disclosure are for the purpose of describing particular embodiments only and are not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "the," and "said" are intended to include the plural forms unless the context clearly indicates otherwise. It is also to be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0038] Terms such as first, second, third, etc. may be used in this disclosure to describe various information, but it will be understood that the information should not be limited to these terms. These terms are used only to distinguish the same type of information from each other. For example, the first information can be called the second information, and similarly, the second information can be called the first information, without departing from the scope of this disclosure. Depending on the context, the word "when" used in this specification can be interpreted as "at the time" or "when" or "depending on a particular".

[0039] As a first aspect, at least one embodiment of the present disclosure provides a motion detection method, and referring to FIG. 1, FIG. 1 shows a flow of the method, including steps S101 to S103.

[0040] The method can also be used to perform motion detection on a target object in an image of the target object. For example, whether a target motion occurs on the target object in the image of the target object, and the target motion may be a dangerous motion, that is, the method can be used to detect whether a dangerous motion occurs on the target object in the image of the target object. The method can be applied to a scene such as a vehicle driving, and can detect whether a dangerous motion occurs on the driver or passenger in the scene. The dangerous motion may be predefined, and for example, the dangerous motion may be a left lean of the body, a right lean of the body, a chest cover with the hands, or a forward leaning face down.

[0041] The image of the target object may be an image taken by an image collecting device or a frame image in a video recorded by an image collecting device. For example, in a vehicle scene, the image of the target object may be an image of the target object in the vehicle cabin, such as an image collected by a surveillance camera pre-installed in the vehicle cabin, or an image frame in a video, and the image of the target object may be an image collected of the driver in the vehicle cabin, or an image frame in a video, or an image collected of a passenger in the vehicle cabin, or an image frame in a video.

[0042] The method may also be performed by an electronic device such as a terminal device or a server, and the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a mobile phone, a cordless phone, a personal digital assistant (PDA) handheld device, a computing device, an in-vehicle device, a wearable device, etc., and the method may be realized in a manner in which a processor calls a computer-readable instruction stored in a memory. Or, the method may be performed by a server, and the server may be a local server, a cloud server, etc. In a vehicle driving scene, the method may be performed by an artificial intelligence emergency call system, and the system is connected to a surveillance camera in the vehicle cabin, and thus can obtain a video stream of a scene area in the vehicle cabin collected by the surveillance camera.

[0043] In step S101, an image of a target object is acquired.

[0044] Also, the image of the target object may be an image taken by an image collecting device or an image frame in a video recorded by an image collecting device, where the image collecting device may be an electronic device with an image collecting function such as a mobile phone, a camera, a security camera, etc. As the name implies, the target object has a target object in the image of the target object, the target object being a person whose action needs to be detected in the image, and the area of ​​the image other than the target object is a background area or other objects (other objects may not exist). For example, the target object in a vehicle scene may be a driver or a specific passenger. Thus, after obtaining a video stream of a scene area, multiple objects in the video stream are detected, and a target object can be identified from the multiple objects based on the position of each object in the multiple objects in the vehicle interior and / or the facial information of each object in the multiple objects. Exemplarily, an object on the driver's seat in the vehicle interior can be identified as the target object, i.e., the driver can be identified as the target object, or an object whose facial information matches the features of a pre-registered reference face can be identified as the target object, i.e., a specific person such as the owner of the vehicle, a registered driver, etc. can be identified as the target object. In the following steps, operations on the target objects can be performed based on the target objects identified in this step.

[0045] As can be understood, the target object in the image of the target object may be one or more. When there are multiple target objects in the image of the target object, each target object can be processed in sequence according to the method provided by the embodiment, or each target object can be processed simultaneously according to the method provided by the embodiment. For example, the image of the target object may be an image collected for a scene in a vehicle interior, and the target object may include multiple target objects such as a driver, a passenger, etc.

[0046] When the image of the target object is an image frame in a video, a plurality of images can be cached from the video first, and the image of the target object in each cached frame from the video can be obtained in sequence. When caching images from a video, each frame image can be obtained and cached sequentially from the video, or image frames can be extracted and cached from the video at regular intervals, or image frames can be extracted and cached from the video according to certain cache conditions, for example, the target object can be detected or tracked based on each frame image in the video, and a plurality of images including the target object can be identified based on the detection / tracking result of the target object, and each frame image in which preset key information of the target object exists in the plurality of images including the target object is added to the cache as the image of the target object, and the preset key information includes at least one of the key points of the face, at least a part of the body, and the skeleton, and the part of the body can be a part such as the left shoulder, the right shoulder, the left ear, the right ear, etc. In this way, the image frames in the cache contain key information of the target object, which is helpful for locating effective skeletal key points in subsequent motion detection based on the cached image frames, and reduces the time and resources consumed in detecting skeletal key points of the target object in motion detection. As can be seen, the number of cached images can be set, and even if any of the above methods are used to cache images from a video, the cached images need to be kept up to date, and the number of cached images must meet the set number, i.e., as the video is updated, the cached images are also updated.

[0047] In one possible embodiment, in a scene of a vehicle traveling, the target object is a driver or a passenger, and the activation condition of the step can be preset, for example, the activation condition can be that the door is locked and / or the speed of the vehicle reaches a preset speed threshold, etc., and when the vehicle meets the above activation condition, i.e., when the door of the vehicle is locked and / or the speed of the vehicle reaches a preset speed threshold, the image of the target object can be acquired. In this way, the image of the target object can be acquired and detected in a scene where the vehicle has a need for danger detection, so that the detection method has a directionality and reduces the waste of computing power, memory and power consumption.

[0048] In step S102, detection of skeleton key points is performed on the image of the target object to obtain detection information of skeleton key points of the target object.

[0049] A pre-trained neural network is used to detect the image of the target object, and thus the skeletal keypoints of the target object can be obtained. The skeletal keypoints can indicate the joint parts in the skeletal structure of the human body, and the skeletal structure diagram of the human body can be drawn by these joint parts. For example, the neural network can detect the obtained skeletal keypoints and the skeletal structure diagram drawn by the skeletal keypoints, as shown in Figure 2. As can be seen from Figure 2, the skeletal keypoints include: nose keypoint 0, left eye keypoint 1, right eye keypoint 2, left ear keypoint 3, right ear keypoint 4, left shoulder keypoint 5, right shoulder keypoint 6, left elbow keypoint 7, right elbow keypoint 8, left wrist keypoint 9, right wrist keypoint 10, left hip keypoint 11, right hip keypoint 12, left knee keypoint 13, right knee keypoint 14, left ankle keypoint 15, and right ankle keypoint 16. The image of the target object is obtained by facing the target object directly or by taking a photograph toward the target object at a certain angle. Therefore, the target object in the image of the target object and the target object in the actual scene are mirror images, i.e., the left side of the target object in the actual scene is the right side of the target object in the image of the target object and the right side of the target object in the actual scene is the left side of the target object in the image of the target object. The origin of the coordinate system in the image of the target object may be in the upper left corner, or may be along the horizontal side toward the right in the positive direction of the horizontal axis (e.g., the x-axis), or along the vertical side toward the bottom in the positive direction of the vertical axis (e.g., the y-axis).

[0050] In addition, the skeleton key points of the target object can be detected, and the skeleton key points of the target object included in the part where the target object appears in the image of the target object can be detected, for example, if the upper body of a driver appears in the image of the target object, only the skeleton key points of the upper body can be detected in this step. In other words, in this step, the skeleton key points of the target object can be detected, and all the key points shown in Figure 2 can be detected, or some of the key points shown in Figure 2 can be detected.

[0051] The detected skeletal keypoints may be indicated by their coordinate locations in the image of the target object, and the skeletal keypoints may be marked at corresponding locations in the image of the target object.

[0052] In step S103, face keypoint detection is performed on the image of the target object to obtain face keypoint detection information of the target object.

[0053] A pre-trained neural network is used to detect the image of the target object, thereby obtaining facial key points of the target object. The neural network may be a different neural network from the neural network for detecting skeleton key points described above. The facial key points may be key points of each region of the face, and these key points can draw the facial contour and important regions. For example, the neural network can detect the obtained facial key points and the facial contour obtained by the facial key points, as shown in FIG. 3, and as can be seen from FIG. 3, the facial key points include facial contour key points 0-32, upper edge contour of the right eyebrow 33-37, upper edge contour of the left eyebrow 38-42, midline of the nose 43-46, lower edge contour of the nose 47-51, outer edge contour of the nose 80-83, outer edge contour of the lips 84-95, inner edge contour of the lips 97-103, center point of the right eye 104, and center point of the left eye 105. It should be noted that the image of the target object is taken facing the target object, and therefore the target object in the image of the target object and the target object in the actual scene are mirror images, i.e., the left side of the target object in the actual scene is the right side of the target object in the image of the target object, and the right side of the target object in the actual scene is the left side of the target object in the image of the target object.

[0054] Also, the facial key points of the target object can be detected, and the facial key points included in the part where the target object's face appears in the image of the target object can be detected. In other words, the result obtained by detecting the facial key points of the target object in this step can be all the key points shown in Fig. 3, or some of the key points shown in Fig. 3, or the facial key points shown in Fig. 3 cannot be detected.

[0055] In some embodiments, the detection of face key points can be directly performed for the target object in the image, or the face detection can be performed for the target object in the image first, and then the image of the face region can be divided according to the face detection result, and the face key points can be detected for the image of the face region, and in this process, the face detection can be assisted according to the features of other regions such as the body of the target object, so that even if some key information of the face is missing, the position of the face can be located, and the detection result of the face key points can be obtained.

[0056] The detected facial keypoints may be indicated by coordinate locations in the image of the target object, and the facial keypoints may be marked at corresponding locations on the image of the target object.

[0057] In step S104, an estimation result of motion information of the target object is determined based on the geometric relationships between object keypoints in the detection information of the skeleton keypoints.

[0058] In addition, the estimation result of the motion information of the target object may be the presence or absence of the target motion in the target object, and the target motion may be a dangerous motion that needs to be detected, etc. The target motion may be preset, and may be one or more. Each target motion has a corresponding target key point, and the geometric relationship between the corresponding multiple target key points in each target motion satisfies a certain preset condition. Thus, a first preset condition for each target object can be preset, and the first preset condition can be set for the geometric relationship between the target key points corresponding to the target motion, so that when identifying the motion information of the target object, for each target motion, it can be determined whether the geometric relationship between the target key points corresponding to the target motion satisfies the first preset condition corresponding to the target motion, and if so, it is identified that the target motion exists in the target object, and if not, it is identified that the target motion does not exist in the target object.

[0059] In one possible embodiment, four target motions, i.e., leaning the body left, leaning the body right, covering the chest with hands, and leaning forward and prone, are preset, and target key points and a first preset condition corresponding to each target motion are set.

[0060] The target key points corresponding to the left lean of the body can be set to the right shoulder key point and the left shoulder key point, the vector formed by the connecting line from the right shoulder key point to the left shoulder key point is called the first target vector, the vector in the horizontal rightward direction (i.e., parallel to the horizontal side of the image of the target object and oriented rightward) is called the standard vector, and the corresponding first preset condition is set as follows: the tangent value of the angle between the first target vector and the standard vector is a positive number, and the absolute value of the tangent value is greater than a first threshold (e.g., the first threshold is 0.4). Taking the right shoulder key point 6 and the left shoulder key point 5 shown in FIG. 2 as an example, the first preset condition corresponding to the left lean of the body can be expressed as tan(vec(6,5))>0.4. In other words, if the tangent value of the angle between the first target vector from the right shoulder key point to the left shoulder key point and the standard vector in the horizontal rightward direction is a positive number and the absolute value of the tangent value is greater than a first threshold value, the estimation result is identified as indicating that the target object has a leftward body tilt motion.

[0061] The target key points corresponding to the right lean of the body can be set to the right shoulder key point and the left shoulder key point, the vector formed by the connecting line from the right shoulder key point to the left shoulder key point is called the target vector, the vector in the horizontal rightward direction (i.e., parallel to the horizontal side of the image of the target object and oriented rightward) is called the standard vector, and the corresponding first preset condition is set as follows: the tangent value of the angle between the target vector and the standard vector is a negative number, and the absolute value of the tangent value is greater than a first threshold (e.g., the first threshold is 0.4). Taking the right shoulder key point 6 and the left shoulder key point 5 shown in FIG. 2 as an example, the first preset condition corresponding to the right lean of the body can be expressed as tan(vec(6,5))<-0.4. In other words, if the tangent value of the angle between the first target vector from the right shoulder key point to the left shoulder key point and the standard vector in the horizontal rightward direction is negative and the absolute value of the tangent value is greater than a first threshold value, the estimation result is determined to indicate that the target object has a rightward body tilt motion.

[0062] The target key points corresponding to the hand covering the chest can be set as a left elbow key point, a left wrist key point, a left shoulder key point, a right elbow key point, a right wrist key point, and a right shoulder key point, and a vector formed by a connecting line from the left elbow key point to the left wrist key point is called a second target vector, a vector formed by a connecting line from the left elbow key point to the left shoulder key point is called a third target vector, a vector formed by a connecting line from the right elbow key point to the right wrist key point is called a fourth target vector, and a vector formed by a connecting line from the right elbow key point to the right shoulder key point is called a fifth target vector, a cosine value of an angle between the second target vector and the third target vector is called a first cosine value, and a cosine value of an angle between the fourth target vector and the fifth target vector is called a second cosine value, and a corresponding first preset condition is set as a condition in which the first cosine value is greater than or equal to a second threshold. (e.g., the second threshold is 0.2), and the vertical distance between the left wrist keypoint and the right shoulder keypoint is greater than a third threshold (e.g., the third threshold is 100), and / or the second cosine value is greater than the second threshold (e.g., the second threshold is 0.2), and the vertical distance between the right wrist keypoint and the left shoulder keypoint is greater than a third threshold (e.g., the third threshold is 100), and the left elbow keypoint 7 and the left wrist keypoint 8 are set as shown in FIG. Taking key point 9, left shoulder key point 5, right elbow key point 8, right wrist key point 10, and right shoulder key point 6 as an example, the first preset condition corresponding to covering the chest with the hand can be expressed as cos(vec(7,5), vec(7,9))>0.2 and y(9)-y(6)>100, and / or the first preset condition can be expressed as cos(vec(8,6), vec(8,10))>0.2 and y(10)-y(5)>100. That is, the estimation result is identified as the presence of a covering action of the chest with the hand of the target object according to the following first or second situation.The first situation includes that the angle between the second target vector and the third target vector is greater than a second threshold, the left wrist keypoint is lower than the right shoulder keypoint, and the vertical distance between the left wrist keypoint and the right shoulder keypoint is greater than a third threshold, the second target vector is a vector from the left elbow keypoint to the left wrist keypoint, and the third target vector is a vector from the left elbow keypoint to the left shoulder keypoint. The second situation includes that the angle between the fourth target vector and the fifth target vector is greater than a second threshold, the right wrist keypoint is lower than the left shoulder keypoint, and the vertical distance between the right wrist keypoint and the left shoulder keypoint is greater than a third threshold, the fourth target vector is a vector from the right elbow keypoint to the right wrist keypoint, and the fifth target vector is a vector from the right elbow keypoint to the right shoulder keypoint.

[0063] The target key points corresponding to the forward leaning prone position can be set to the left ear key point, the left shoulder key point, the right ear key point, and the right shoulder key point, and the corresponding first preset condition can be set to the vertical distance between the left ear key point and the left shoulder key point being greater than a fourth threshold (e.g., the fourth threshold is 50) and / or the vertical distance between the right ear key point and the right shoulder key point being greater than a fourth threshold (e.g., the fourth threshold is 50). Taking the left ear key point 3, the left shoulder key point 5, the right ear key point 4, and the right shoulder key point 6 shown in FIG. 2 as examples, the first preset condition corresponding to the forward leaning prone position can be expressed as y(3)-y(5)>50 and / or y(4)-y(6)>50. In other words, the estimation result is identified as indicating the presence of a forward leaning prone motion in the target subject when the left ear keypoint is lower than the left shoulder keypoint and the vertical distance between the left ear keypoint and the left shoulder keypoint is greater than a fourth threshold value, and / or when the right ear keypoint is lower than the right shoulder keypoint and the vertical distance between the right ear keypoint and the right shoulder keypoint is greater than a fourth threshold value.

[0064] In step S105, the estimation result is verified based on the detection information of the face key points to obtain the target object motion detection result.

[0065] Also, in each target motion, the face assumes a corresponding pose, for example, the face is deflected toward a certain direction, or at least a part of the face region is occluded. The estimation result of the motion information acquired in step S104 can be further verified by detecting the geometric constraint relationship or relative position relationship satisfied between the specific target face key points in the corresponding face pose. Specifically, one or more target face key points associated with each motion can be set, and each target motion can further be preset with a second preset condition, and the second preset condition can be set with respect to the geometric constraint relationship or relative position relationship satisfied between the target face key points corresponding to the target motion. Thus, when the estimation result is that the target motion exists, the detection result of the target object is identified as the presence of the target motion according to the target face key points corresponding to the target motion in the detection information of the face key points satisfying the second preset condition corresponding to the target motion. That is, when both the two dimensions of the skeleton key points and the face key points identify the target motion as existing in the target object, the target motion is identified as existing in the target object. This further improves the accuracy of motion detection and avoids erroneous detection of motion information based on skeletal keypoints.

[0066] In one possible embodiment, the motion of the target is a left tilt of the body or a right tilt of the body, i.e., the estimated result of the motion information of the target target identified in step S104 is that there is a left tilt of the body or a right tilt of the body. In this case, a second preset condition corresponding to the left tilt of the body or the right tilt of the body can be identified based on the positional relationship between the face key points of the first target corresponding to the left tilt of the body or the right tilt of the body, and the motion detection result of the target target is identified as the presence of the left tilt of the body or the right tilt of the body in response to the positional relationship between the face key points of the first target corresponding to the left tilt of the body or the right tilt of the body in the detection information of the face key points satisfying the second preset condition corresponding to the left tilt of the body or the right tilt of the body.

[0067] For example, the key points of the face of the first target related to the left inclination of the body can be set as the right eye outer key point (i.e., key point 36 shown in FIG. 3) and the left eye outer key point (i.e., key point 45 shown in FIG. 3), the vector formed by the connecting line from the right eye outer key point to the left eye outer key point is called the sixth target vector, and the vector in the horizontal rightward direction (i.e., parallel to the horizontal side of the image of the target target and oriented rightward) is called the standard vector. Furthermore, the second preset condition corresponding to the left inclination of the body can be set to an integer tangent value of the angle between the sixth target vector and the standard vector, and the absolute value of the tangent value is greater than a fifth threshold (e.g., the fifth threshold is 0.4), and the second preset condition corresponding to the left inclination of the body can be expressed as tan(vec(36,45))>0.4, taking the right eye outer key point 36 and the left eye outer key point 45 shown in FIG. 3 as an example. In other words, if the target object's movement is a left leaning of the body, the tangent value of the angle between a first target vector from a key point on the outer right eye of the target object to a key point on the outer left eye of the target object and a standard vector in a horizontal rightward direction is a positive number, and the absolute value of the tangent value is greater than a first threshold value, and therefore the movement detection result of the target object is identified as the presence of a left leaning of the body.

[0068] The key points of the first target face related to the right tilt of the body can be set as the right eye outer key point (i.e., key point 36 shown in FIG. 3) and the left eye outer key point (i.e., key point 45 shown in FIG. 3), and the vector formed by the connecting line from the right eye outer key point to the left eye outer key point is called the sixth target vector, and the vector in the horizontal rightward direction (i.e., parallel to the horizontal side of the image of the target target and oriented rightward) is called the standard vector. Furthermore, the second preset condition corresponding to the right tilt of the body can be set as the tangent value of the angle between the sixth target vector and the standard vector being a negative number, and the absolute value of the tangent value being greater than a fifth threshold (e.g., the fifth threshold is 0.4). Taking the right eye outer key point 36 and the left eye outer key point 45 shown in FIG. 3 as an example, the second preset condition corresponding to the left tilt of the body can be expressed as tan(vec(36,45))<-0.4. In other words, if the target motion is a right body tilt, the tangent value of the angle between a first target vector from a key point on the outer right eye of the target target to a key point on the outer left eye of the target target and a standard vector in a horizontal rightward direction is negative, and the absolute value of the tangent value is greater than a first threshold value, then the motion detection result of the target target is identified as a right body tilt motion.

[0069] In one possible embodiment, the target motion is a prone motion, i.e., the estimated result of the motion information of the target target identified in step S104 is that a prone motion exists. In this case, a second preset condition corresponding to prone motion can be identified based on whether a facial key point of the target is missing, and if the motion of the target is prone motion, the motion detection result of the target target is identified as the presence of the prone motion according to the missing facial key points of a second target corresponding to the prone motion in the detection information of the facial key points.

[0070] In addition, the facial key points of the second object corresponding to the forward leaning prone position can be set to include eye key points, nose key points, and mouth key points. Furthermore, the second preset condition corresponding to the forward leaning prone position can be set to that none of the eye key points, nose key points, and mouth key points appear in the detection information of the facial key points.

[0071] Furthermore, the facial key points of the second object corresponding to the leaning forward prone position can be set as facial content key points, and the facial content key points include facial key points other than the facial edge key points. Furthermore, the second preset condition corresponding to the leaning forward prone position can be set as the number of facial content key points does not exceed a preset number. In other words, when it is determined that the number of facial content key points detected based on the detection information of the facial key points does not exceed a preset number, the missing facial key points of the second object corresponding to the leaning forward prone position are determined.

[0072] Also, the facial key points of the second object corresponding to the forward-leaning prone position can be set as all facial key points. Furthermore, the second preset condition corresponding to the forward-leaning prone position can be set as the number of facial key points not exceeding a preset number. In other words, when it is determined that the number of facial key points detected based on the detection information of the facial key points does not exceed the preset number, the missing facial key points of the second object corresponding to the forward-leaning prone position are determined.

[0073] The above-mentioned multiple methods of identifying missing key points of the second object's face can be adapted to accurately detect the forward leaning prone motion when the face takes on multiple different poses in real situations, thereby realizing more flexible and more accurate verification of the motion detection results.

[0074] As can be seen, when the motion detection result of the target object is that the target object has a target motion, alarm information can be sent to the service platform. For example, the target motion is a dangerous motion that needs to be detected, and when the target object has a dangerous motion, alarm information can be sent to the service platform. In a vehicle driving scenario, the service platform can be a service platform that operates vehicles, such as an internet ride-hailing service platform. The service platform can be a medical platform, and after the service platform receives the alarm information, emergency treatment measures can be taken, thereby improving the emergency efficiency of the driver who has encountered a danger, further improving the emergency effect, and better protecting the life safety of the person inside the vehicle. Also, as can be seen, when the image of the target object is an image frame in a video, and the image of the target object is updated with the update of the video recording, within a preset period, the motion detection result is continuously that the target object has a target motion, and alarm information can be sent to the service platform, thereby avoiding erroneous transmission of alarm information due to fluctuations in motion information.

[0075] As can be seen from the above embodiment, the image of the target object is obtained, and the skeleton keypoint detection is performed on the image of the target object to obtain the skeleton keypoint detection information of the target object, and the face keypoint detection is performed on the image of the target object to obtain the face keypoint detection information of the target object, so that the estimation result of the motion information of the target object is obtained based on the geometric relationship between the target keypoints in the skeleton keypoint detection information, and finally, the estimation result is verified based on the face keypoint detection information to obtain the motion detection result of the target object. Because the geometric relationship between the target keypoints is objective and accurate, it can accurately detect whether the target object has made a dangerous motion, and the estimation result of the motion information obtained by the target keypoints can be further verified by the face keypoint detection information, further improving the accuracy of motion detection. When the detection method is applied in a vehicle, it can accurately detect whether a danger occurs to the driver and passengers in the vehicle, thereby improving the safety of the ride and improving the user's experience.

[0076] In some embodiments of the present disclosure, the target object image is an image frame in a video. A number of images can be cached from the video first, and then the target object image of each cached frame can be obtained from the video in sequence. The specific manner of caching images has already been described in detail in step S101, and will not be repeated here.

[0077] Based on the above acquisition manner of the image of the target object, step S104 can be performed in the following manner, that is, the estimation result of the motion information of the target object can be determined based on the geometric relationship between the target keypoints in the detection information of the skeleton keypoints.

[0078] First, detect the motion direction of the target object based on the images of the target object of the multiple frames in the cache. First, the features of the target object in each image frame can be extracted in the multiple image frames, and for example, the feature points of the target object are extracted based on the basic principle that the brightness of the same target does not change when it moves between different frames; then, in the multiple image frames, determine optical flow information of the target object in the video stream based on the features of the target object in each image frame, and the optical flow information can indicate that the target moves between different frames; and finally, determine the motion direction of the target object in the scene area based on the optical flow information of the target object in the video stream. For example, the LucasKanade algorithm is used to determine the motion direction of the target object from being relatively still to moving.

[0079] Then, obtain an estimation result of the motion information of the target object based on the geometric relationship between the target key points corresponding to the motion direction in the detection information of the skeletal key points. Each motion direction can correspond to one or more preset target motions, and the corresponding multiple skeletal key points in each target motion satisfy a certain geometric constraint relationship. Therefore, for each target motion, a corresponding multiple target key points can be set, and a first preset condition can be preset for each target motion, and the first preset condition can be set for the geometric constraint relationship between the target key points corresponding to the target motion. Therefore, when identifying the motion information of the target object, for each target motion in the motion direction, it can be determined whether the target key points corresponding to the target motion satisfy the first preset condition corresponding to the target motion, and if so, the target object has the motion of the target object, and if not, the target object does not have the motion of the target object.

[0080] For example, each motion direction corresponds to the motion of one target, and a target key point and a first preset condition corresponding to the motion direction can be set. Thus, after the motion direction is identified, it can be determined whether the target motion corresponding to the motion direction exists in the target target only based on the target key point corresponding to the motion direction and the first preset condition corresponding to the motion direction. This method narrows the detection range of the target motion based on the motion direction, thereby further saving energy consumption and memory and improving detection efficiency. Furthermore, the target key point is detected based on the motion direction, so that the motion detection has directionality and further improves the detection accuracy. Specifically, the target motions corresponding to the four motion directions, which are left, right, and up, are preset to body left tilt, body right tilt, chest covering with hands, and leaning forward and prone, and the corresponding target key point and the first preset condition can be preset for each target motion (i.e., each motion direction). The target key point and the first preset condition corresponding to each target motion have already been described in detail in step S104, so they will not be repeated here.

[0081] In some embodiments of the present disclosure, the target image is an image frame in a video, and a number of images are first cached from the video, and then the target image of each frame cached in the video is obtained in sequence. The specific manner of caching images has been described in detail in step S101, and will not be repeated here.

[0082] After performing step S104 based on the above-mentioned acquisition method of the image of the target object, the motion detection results of the target object obtained based on the latest preset number of frame images of the target object are acquired, and the motion detection result with the highest occurrence frequency among the motion detection results of the target object obtained based on the latest preset number of frame images of the target object is identified as the current motion of the target object. Also, the preset number of frames may be two frames, four frames, five frames, etc.

[0083] In addition, the motion detection results obtained by detecting each frame image of the target object are input into a smoothed queue, and a smoothed window is set, the size of which is the above-mentioned preset number, and the smoothed window is moved along with the update of the smoothed queue, and after the smoothed window is moved each time, a current motion is determined according to multiple motion detection results within the smoothed window, thereby realizing smoothing processing of the motion detection result and improving the effectiveness and stability of the current motion.

[0084] If the motion detection results detected based on the target object image of each frame in a preset number of frames of target object images are the same, it can be identified as the current motion. For example, if the motion detection results detected based on the target object image of each frame in five frames of target object images are all a leftward tilt of the body, it can be identified as the current motion.

[0085] If a preset number of motion detection results detected based on a preset number of frames of target object images do not match, the motion detection result that appears most frequently, i.e., the motion detection result with the highest number of occurrences, is determined as the current motion. For example, if a motion detection result detected based on four frames of target object images in five frames is a leftward tilt of the body, and a motion detection result detected based on one frame of target object image is no target motion, the leftward tilt of the body can be determined as the current motion. Optionally, if the number of motion detection results with the highest number of occurrences in the motion detection results of multiple frame images in a smooth window does not exceed a preset percentage (e.g., 50%) of the length of the smooth window (i.e., the total number of image frames in the smooth window), the motion detection result of the smooth window can be discarded. At this time, the smooth window is moved by a preset step size (i.e., the number of image frames) to determine the motion detection result corresponding to the new smooth window.

[0086] In order to accurately identify the current motion, the preset number can be set to an odd number such as 3, 5, 7, etc. If the preset number is set to an even number and multiple motion detection results with the same number appear, the most recent motion detection result among them is determined to be the current motion.

[0087] As can be seen, when the current motion of the target object is a target motion, alarm information can be sent to the service platform, which is more accurate and reliable than the mode of sending alarm information when the target object motion detection result is that a target motion exists, and can avoid sending erroneous alarm information.

[0088] Referring to FIG. 4, it shows a complete flow of a motion detection method in a scene of vehicle driving as an example. As can be seen from FIG. 4, first, execute step S11, and if the vehicle state meets the start condition of the method, start the vehicle cabin emergency system, then execute step S12, and the monitoring camera in the vehicle collects the vehicle cabin passenger information, i.e., the image of the passenger in the vehicle cabin, and then execute step S13, and obtain the motion information estimation result, i.e., whether the target object has a motion of the target corresponding to the movement direction, based on the motion detection of the skeleton key points. Then execute step S14, and use the face key points to verify the estimation result of step S13 and obtain the motion detection result. Then execute step S15, and smooth the multiple detection results obtained in step S14, and finally execute step S16, and detect the dangerous motion based on the smoothing result of S15, and send the emergency signal if it continues for a certain time.

[0089] The motion detection method provided in this embodiment combines motion detection based on skeleton key points and motion verification based on face key points, which can evaluate passenger motion information more accurately and all-round, and at the same time use smoothing algorithm and cache processing to effectively handle some jumps and fluctuations in the results, provide important reference data for rental car companies and traffic supervision and management departments, and directional customize safety plans and operation management to ensure the life, health and safety of passengers in the vehicle.

[0090] According to a second aspect of the present disclosure, a motion detection device is provided. Referring to FIG. 5 , the device includes: an acquisition module 501, a first detection module 502, a second detection module 503, an estimation module 504 and a verification module 505; The acquisition module 501 is for acquiring an image of a target object; The first detection module 502 is for detecting skeleton key points on the image of the target object to obtain detection information of skeleton key points of the target object; The second detection module 503 is for detecting face key points on the image of the target object to obtain detection information of the face key points of the target object; The estimation module 504 is for obtaining an estimation result of motion information of the target object based on a geometric relationship between target key points in the detection information of the skeleton key points; A verification module 505 is for verifying the estimation result based on the detection information of the face key points, and obtaining a motion detection result of the target object.

[0091] In some embodiments of the present disclosure, the estimation module specifically comprises: If the geometric relationship between the target keypoints in the detection information of the skeletal keypoints satisfies a first preset condition corresponding to a target motion, the estimation result of the target object's motion information identifies the target motion as existing.

[0092] Specifically, the verification module: If the estimation result is that a target movement is present, a second predetermined condition corresponding to the target movement is satisfied according to the target facial key points corresponding to the target movement in the facial key point detection information, and the detection result of the target target is identified as that the target movement is present.

[0093] In some embodiments of the present disclosure, the verification module satisfies a second preset condition corresponding to the target's motion according to the target's facial key points corresponding to the target's motion in the facial key point detection information, and identifying that the target target's detection result is the presence of the target's motion is specifically: When the target movement is a left or right tilt of the body, the positional relationship between a plurality of first target's facial key points corresponding to the left or right tilt of the body in the detection information of the facial key points satisfies a second preset condition corresponding to the left or right tilt of the body, thereby identifying that the target target's movement detection result indicates the presence of a left or right tilt of the body.

[0094] In some embodiments of the present disclosure, the plurality of first object's face key points are left eye outer key points and right eye outer key points;

[0095] The verification module is for, when the motion of the target is a leftward or rightward tilt of the body, identifying the motion detection result of the target target as including a leftward or rightward tilt of the body in response to a positional relationship between a plurality of face key points of the first target corresponding to the leftward or rightward tilt of the body in the detection information of the face key points satisfying a second preset condition corresponding to the leftward or rightward tilt of the body, and specifically, If the motion of the target is a left lean of the body, the tangent value of the angle between a first target vector from a key point on the outer right eye of the target target to a key point on the outer left eye of the target target and a standard vector in a horizontal rightward direction is a positive number, and the absolute value of the tangent value is greater than a first threshold value, and / or, When the target motion is a right tilt of the body, the tangent value of the angle between a first target vector from a key point on the outer right eye of the target target to a key point on the outer left eye of the target target and a standard vector in a horizontal rightward direction is negative, and the absolute value of the tangent value is greater than a first threshold value, thereby identifying the motion detection result of the target target as the presence of a right tilt of the body.

[0096] In some embodiments of the present disclosure, when the verification module identifies the detection result of the target object as having a target motion according to the target face keypoints corresponding to the target motion in the face keypoint detection information satisfying a second preset condition corresponding to the target motion, specifically: When the motion of the target is leaning forward prone, the motion detection result of the target target is identified as including the motion of leaning forward prone depending on the absence of a facial key point of a second target corresponding to the motion of leaning forward prone in the detection information of the facial key points.

[0097] In some embodiments of the present disclosure, the method further includes a determination module, the determination module comprising: When it is determined that the number of face content key points including face key points other than face edge key points detected based on the face key point detection information does not exceed a preset number, it determines that the second object's face key points corresponding to the forward leaning prone position are missing, or When it is determined that the number of facial key points detected based on the detection information of the facial key points does not exceed a preset number, a missing facial key point of the second subject corresponding to the forward leaning prone position is determined.

[0098] In some embodiments of the present disclosure, the acquisition module specifically includes: For sequentially obtaining cached frame images of the target object from a video. The apparatus further includes a smoothing module, the smoothing module comprising: After verifying the estimation result based on the detection information of the face key points to obtain a motion detection result of the target object, the motion detection result of the target object obtained based on a latest predetermined number of frame images of the target object is obtained; The motion detection result that occurs most frequently among the motion detection results of the target object obtained based on the latest preset number of frame images of the target object is identified as the current motion of the target object.

[0099] In some embodiments of the present disclosure, the apparatus further comprises a mitigation module, the mitigation module for: Before sequentially acquiring each frame image of the cached target object from the video, detect the target object based on each frame image in the video or track the target object, and identify a plurality of images including the target object based on the detection result or tracking result of the target object; Each frame image in which preset key information including at least one of the key points of the face, at least a part of the body, and the skeleton of the target object is present in the plurality of images including the target object is added to a cache and regarded as an image of the target object.

[0100] In some embodiments of the present disclosure, the estimation module specifically comprises: Detecting a direction of motion of the target object based on a plurality of frames of images of the target object in the cache; The method is to obtain an estimation result of motion information of the target object based on a geometric relationship between object key points corresponding to the movement direction in the detection information of the skeleton key points.

[0101] In some embodiments of the present disclosure, the image of the target object includes an image of the target object within a vehicle interior. The acquisition module specifically includes: To capture an image of the target object when the vehicle doors are locked and / or the vehicle speed reaches a pre-set speed threshold.

[0102] In some embodiments of the present disclosure, the method further includes a target module, the target module comprising: acquiring an image of an interior of a vehicle; and detecting a plurality of objects in the image; The target object is identified from the plurality of objects based on a position of each of the plurality of objects in the vehicle interior and / or facial information of each of the plurality of objects.

[0103] In some embodiments of the present disclosure, the device further includes an alarm module, the alarm module comprising: and sending alarm information to a service platform when the target object motion detection result is that a target motion exists in the target object.

[0104] The specific manner in which each module of the apparatus in the above embodiment performs operations has already been described in detail in the embodiment of the method in the third aspect, so it will not be described in detail here.

[0105] In a third aspect, at least one embodiment of the present disclosure provides an apparatus, and referring to FIG. 6, it shows a configuration of the apparatus, the apparatus including a memory and a processor, the memory for storing computer instructions executable by the processor, and the processor for detecting an operation based on any one of the methods according to the first aspect when executing the computer instructions.

[0106] As a fourth aspect, at least one embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored therein, the computer program being executed by a processor to perform a method as recited in any one of the first aspects.

[0107] In this disclosure, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying a relative importance. The term "plurality" means two or more, unless otherwise specified.

[0108] Other embodiments of the present disclosure will be readily suggested to those skilled in the art from consideration of this specification and practice of the disclosure disclosed herein. This disclosure follows the general principles of this disclosure and includes common general knowledge or practice in the art that is not disclosed by this disclosure. It is intended to cover any modifications, uses, or adaptations of this disclosure. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0109] It should be understood that the present disclosure is not limited to the exact configuration described above and illustrated in the accompanying drawings, and various modifications and changes may be made thereto without departing from the scope of the present disclosure, which is limited only by the scope of the appended claims.

Claims

1. 1. A motion detection method, comprising: Obtaining an image of a target object; Performing skeleton keypoint detection on the image of the target object to obtain skeleton keypoint detection information of the target object; performing face keypoint detection on the image of the target object to obtain face keypoint detection information of the target object; Obtaining an estimation result of motion information of the target object based on a geometric relationship between object keypoints in the detection information of the skeleton keypoints; verifying the estimation result based on detection information of the face key points to obtain a motion detection result of the target object.

2. Obtaining an estimation result of the motion information of the target object based on the geometric relationship between the target keypoints in the detection information of the skeleton keypoints includes: identifying the estimation result of the motion information of the target object as the presence of the motion of the target when the geometric relationship between the target keypoints in the detection information of the skeleton keypoints satisfies a first preset condition corresponding to a motion of the target; Verifying the estimation result based on detection information of the face key points to obtain a motion detection result of the target object, 2. The method of claim 1, further comprising: if the estimation result is that a target motion is present, identifying the detection result of the target target as the presence of the target motion in response to a target face key point corresponding to the target motion in the face key point detection information satisfying a second predetermined condition corresponding to the target motion.

3. 3. The method of claim 2, wherein identifying the detection result of the target object as indicating the presence of the target movement in response to the target's facial key points corresponding to the target movement in the detection information of the facial key points satisfying a second preset condition corresponding to the target movement includes: when the target movement is a left leaning of the body or a right leaning of the body, identifying the target object's movement detection result as indicating the presence of a left leaning of the body or a right leaning of the body in response to a positional relationship between a plurality of first target's facial key points corresponding to the left leaning of the body or the right leaning of the body in the detection information of the facial key points satisfying a second preset condition corresponding to the left leaning of the body or the right leaning of the body.

4. the plurality of first object face key points include a left eye outer key point and a right eye outer key point; When the motion of the target is a leftward or rightward tilt of the body, specifying the motion detection result of the target target as a motion of the leftward or rightward tilt of the body in response to a positional relationship between a plurality of face key points of the first target corresponding to the leftward or rightward tilt of the body in the detection information of the face key points satisfying a second preset condition corresponding to the leftward or rightward tilt of the body, If the motion of the target is a left lean of the body, the tangent value of the angle between a first target vector from a key point on the outer right eye of the target target to a key point on the outer left eye of the target target and a standard vector in a horizontal rightward direction is a positive number, and the absolute value of the tangent value is greater than a first threshold value, the motion detection result of the target target is determined to indicate the presence of a left lean of the body; or 4. The method for detecting a movement of the target object as described in claim 3, further comprising: when the movement of the target object is a right tilt of the body, identifying the movement detection result of the target object as the presence of a right tilt of the body in response to a tangent value of an angle between a first target vector from a key point on the outer right eye of the target object to a key point on the outer left eye of the target object and a standard vector in a horizontal rightward direction being a negative number and an absolute value of the tangent value being greater than a first threshold value.

5. Identifying the detection result of the target object as a presence of the target motion in response to a target face key point corresponding to the target motion in the detection information of the face key point satisfying a second preset condition corresponding to the target motion, 3. The movement detection method according to claim 2, further comprising: when the movement of the target is leaning forward prone, identifying the movement detection result of the target object as including the movement of leaning forward prone in accordance with the absence of a facial key point of a second target corresponding to the leaning forward prone in the detection information of the facial key points.

6. The method further comprises: When it is determined that the number of face content key points including face key points other than face edge key points detected based on the face key point detection information does not exceed a preset number, determining that a face key point of the second object corresponding to the forward leaning prone position is missing; or 6. The movement detection method according to claim 5, further comprising: when it is determined that the number of facial key points detected based on the detection information of the facial key points does not exceed a predetermined number, determining that a missing facial key point of the second object corresponding to the forward leaning prone position is present.

7. obtaining images of the target object includes sequentially obtaining cached frame images of the target object from a video; After verifying the estimation result based on the detection information of the face key points and obtaining the motion detection result of the target object, the method further comprises: acquiring a motion detection result of the target object obtained based on a preset number of latest frame images of the target object; and identifying, as a current motion of the target object, a motion detection result that appears most frequently among the motion detection results of the target object obtained based on a latest preset number of frame images of the target object.

8. before sequentially obtaining each cached frame image of a target object from the video, further Detecting the target object based on each frame image in the video or tracking the target object; identifying a plurality of images including the target object based on the detection or tracking result of the target object; and adding each frame image, among the plurality of images including the target object, in which preset key information including at least one of key points of a face, at least a part of a body, and a skeleton of the target object is present, to a cache to set the frame image as an image of the target object.

9. Obtaining an estimation result of motion information of the target object based on a geometric relationship between target keypoints in the detection information of the skeleton keypoints, detecting a direction of motion of the target object based on a plurality of frame images of the target object in the cache; and obtaining an estimation result of motion information of the target object based on a geometric relationship between object key points corresponding to the movement direction in the detection information of the skeletal key points.

10. the image of the target object includes an image of the target object within the vehicle interior; Acquiring an image of the target object includes:

10. The method of claim 1, further comprising: capturing an image of the target object when a vehicle door is locked and / or when the vehicle speed reaches a preset speed threshold.

11. The method further comprises: acquiring an image of an interior of a vehicle and detecting a plurality of objects in the image; and identifying the target object from the plurality of objects based on a position of each of the plurality of objects in the vehicle interior and / or face information of each of the plurality of objects.

12. The method further comprises: The motion detection method according to any one of claims 1 to 11, further comprising: sending alarm information to a service platform when the motion detection result of the target object is that a target motion exists in the target object.

13. A motion detection device, comprising: an acquisition module for acquiring an image of a target object; a first detection module for performing skeleton keypoint detection on the image of the target object to obtain skeleton keypoint detection information of the target object; a second detection module for detecting face key points on the image of the target object to obtain face key point detection information of the target object; an estimation module for obtaining an estimation result of motion information of the target object based on a geometric relationship between target keypoints in the detection information of the skeleton keypoints; a verification module for verifying the estimation result based on detection information of the face key points to obtain a motion detection result of the target object.

14. 1. An electronic device comprising: A memory and a processor are included.

13. An electronic device, characterized in that the memory is for storing computer instructions executable by a processor, the execution of which, when executed by the processor, performs the method according to any one of claims 1 to 12.

15. 1. A computer-readable storage medium, comprising: A computer readable storage medium having a computer program stored thereon, the computer program being adapted to perform the method according to any one of claims 1 to 12 when executed by a processor.

Images