A method and apparatus for detecting

Abstract

The application provides a detection method. The method analyzes continuous multiple frames of images collected by an electronic device, determines a target dynamic object capable of displacement in each frame of image and multiple key points of the target dynamic object, the key points can represent the characteristics of the target dynamic object, and analyzes based on the multiple key points to determine whether displacement occurs in the process of collecting the continuous multiple frames of images, the displacement is caused by relative movement between the target dynamic object and the electronic device, and the relative movement can be caused by movement of the electronic device, movement of the target dynamic object, or movement of both the electronic device and the target dynamic object, and is not limited to detecting only shaking of the electronic device, thereby improving the detection accuracy.

Description

Technical Field

[0001] This application relates to the field of information processing, and more specifically, to a detection method and apparatus. Background Technology

[0002] Using electronic devices while in motion can harm a user's eye health. This "movement" can include three scenarios: first, the electronic device is moving while the user remains stationary; second, the electronic device is stationary while the user is moving; and third, both the electronic device and the user are shaking, but at different frequencies and amplitudes. All three scenarios can harm a user's eye health; in other words, when the electronic device and the user are not relatively stationary, it can cause eye damage.

[0003] Currently, mobile devices use their built-in accelerometers to detect acceleration and then determine if the device is shaking. However, this method can only detect device shaking and cannot detect user-induced shaking. Summary of the Invention

[0004] In view of this, this application provides a detection method, as follows:

[0005] Obtain at least two frames of images continuously captured by the electronic device within the current time period;

[0006] The at least two frames of images are identified to determine the target dynamic object in each frame of the image and at least two key points of the target dynamic object, wherein the dynamic object represents an object that can undergo displacement, and the key points represent the features of the target dynamic object.

[0007] Based on the at least two frames of images, at least two key points of the target dynamic object are analyzed to obtain the analysis results;

[0008] Based on the analysis results, it is determined whether displacement has occurred between the target dynamic object and the electronic device.

[0009] This application also provides a detection device, as follows:

[0010] The acquisition module is used to acquire at least two frames of images continuously captured by the electronic device within the current time period;

[0011] The recognition module is used to recognize the at least two frames of images, determine the target dynamic object in each frame of image and at least two key points of the target dynamic object, wherein the dynamic object represents an object that can undergo displacement, and the key points represent the features of the target dynamic object;

[0012] The analysis module is used to analyze at least two key points of the target dynamic object based on the at least two frames of images to obtain analysis results; and based on the analysis results, to determine whether displacement has occurred between the target dynamic object and the electronic device.

[0013] As can be seen from the above technical solution, this application provides a detection method, including: obtaining at least two frames of images continuously acquired by an electronic device within a current time period; identifying the at least two frames of images to determine a target dynamic object and at least two key points of the target dynamic object in each frame of images, wherein the dynamic object represents an object capable of displacement, and the key points represent features of the target dynamic object; analyzing the at least two key points of the target dynamic object based on the at least two frames of images to obtain analysis results; and determining whether displacement has occurred between the target dynamic object and the electronic device based on the analysis results. In this solution, by analyzing multiple consecutive frames of images acquired by the electronic device, a target dynamic object capable of displacement in the images and multiple key points of the target dynamic object are identified, and analysis is performed based on these multiple key points to determine whether displacement has occurred during the acquisition of the multiple consecutive frames of images. This displacement is generated by relative movement between the target moving object and the electronic device, and this relative movement can be caused by the movement of the electronic device, the movement of the target dynamic object, or even the movement of both the electronic device and the target dynamic object. It is not limited to detecting only the shaking of the electronic device, thus improving the accuracy of detection. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0015] Figure 1 A flowchart of an embodiment 1 of a control method provided in this application;

[0016] Figure 2 A flowchart of an embodiment 2 of the control method provided in this application;

[0017] Figure 3 A flowchart of embodiment 3 of the control method provided in this application;

[0018] Figure 4 A flowchart of embodiment 4 of the control method provided in this application;

[0019] Figure 5 A flowchart of embodiment 5 of the control method provided in this application;

[0020] Figure 6 A flowchart of embodiment 6 of the control method provided in this application;

[0021] Figure 7 A flowchart of embodiment 7 of the control method provided in this application;

[0022] Figure 8 A flowchart of embodiment 8 of the control method provided in this application;

[0023] Figure 9 This is a schematic diagram of a control device embodiment provided in this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] like Figure 1 The diagram shown is a flowchart of an embodiment 1 of a control method provided in this application. The method is applied to an electronic device and includes the following steps:

[0026] Step S101: Obtain at least two frames of images continuously captured by the electronic device within the current time period;

[0027] Among them, electronic devices are equipped with image acquisition structures, such as cameras, which can acquire images from their image acquisition areas to obtain images, which are RGB (red-green-blue) images.

[0028] Specifically, during the operation of an electronic device, the image acquisition structure acquires images and obtains multiple frames of images. These multiple frames of images are images continuously acquired by the image acquisition structure within the current time period.

[0029] In practice, the time period can be a fixed duration set in advance, or it can be set according to the actual application scenario, such as 3 seconds, 5 seconds, 15 seconds, etc. However, this application does not limit the length of the time period.

[0030] In practice, the number of frames in the image can be 10 frames, 30 frames, etc., and this application does not impose any restrictions on the number of frames.

[0031] Step S102: Identify the at least two frames of images to determine the target dynamic object in each frame of images and at least two key points of the target dynamic object;

[0032] The dynamic object represents an object capable of displacement, and the key point represents the characteristics of the target dynamic object.

[0033] One method involves identifying the target dynamic object and its key points for each frame of an image based on a pre-trained neural network model; another method involves comparing each frame of an image based on a pre-defined dynamic object and its key points to obtain the target dynamic object and its key points.

[0034] Specifically, in this application, only dynamic objects in the image are identified, while static objects (such as tables, walls, etc.) in the image are ignored.

[0035] In practice, dynamic objects can be people, faces, airplanes, animals, etc. Any object that can move can be considered a dynamic object.

[0036] It should be noted that in the current scenario, the target dynamic object can be moving or stationary; this application identifies the dynamic object.

[0037] The key point is a feature that characterizes the target dynamic object. For example, if the target dynamic object is a human face, the key point could be the eyes, ears, nose, facial contour points, etc.; if the target dynamic object is an airplane, the key point could be the airplane's wings, contour points, etc. This application does not restrict which structure of the dynamic object the key point represents.

[0038] Step S103: Based on the at least two frames of images, analyze at least two key points of the target dynamic object to obtain the analysis results;

[0039] Specifically, the analysis is performed based on multiple key points of the target dynamic object in at least two frames of images to determine the situation of the target dynamic object in the image and obtain the analysis results.

[0040] Here, the key point is a point that characterizes the features of the target dynamic image, and the distribution of each feature in the target dynamic object is specified. By analyzing each key point, the situation of each feature in the target dynamic image can be obtained, that is, the situation of the target dynamic object as a whole can be obtained.

[0041] Step S104: Based on the analysis results, determine whether displacement has occurred between the target dynamic object and the electronic device.

[0042] The analysis result characterizes whether displacement occurs between the target dynamic object and the electronic device, and this displacement is generated by the relative movement between the target moving object and the electronic device.

[0043] It should be noted that this relative movement can be caused by the movement of the electronic device, the movement of the target dynamic object, or even the movement of both the electronic device and the target dynamic object.

[0044] In summary, this embodiment provides a detection method comprising: acquiring at least two consecutive frames of images acquired by an electronic device within a current time period; identifying the at least two frames of images to determine a target dynamic object and at least two key points of the target dynamic object in each frame, wherein the dynamic object represents an object capable of displacement, and the key points represent features of the target dynamic object; analyzing the at least two key points of the target dynamic object based on the at least two frames of images to obtain analysis results; and determining, based on the analysis results, whether displacement has occurred between the target dynamic object and the electronic device. In this solution, by analyzing multiple consecutive frames of images acquired by the electronic device, a target dynamic object capable of displacement and multiple key points of the target dynamic object are identified in the images. Based on these multiple key points, it is determined whether displacement has occurred during the acquisition of the multiple consecutive frames of images. This displacement is generated by relative movement between the target moving object and the electronic device. This relative movement can be caused by movement of the electronic device, movement of the target dynamic object, or even movement of both the electronic device and the target dynamic object. It is not limited to detecting only the shaking of the electronic device, thus improving the accuracy of detection.

[0045] like Figure 2 The diagram shown is a flowchart of a control method embodiment 2 provided in this application. The method includes the following steps:

[0046] Step S201: Obtain at least two frames of images continuously captured by the electronic device within the current time period;

[0047] Step S202: Identify the at least two frames of images to determine the target dynamic object in each frame of images and at least two key points of the target dynamic object;

[0048] Steps S201-202 are the same as steps S101-102 in Example 1, and will not be described again in this example.

[0049] Step S203: Based on the at least two frames of images, analyze at least two key points of the target dynamic object to obtain the coordinate sequence of each key point;

[0050] Specifically, the key points identified in the multi-frame images are analyzed sequentially to obtain the coordinate sequence of each key point.

[0051] Each keypoint corresponds to a coordinate sequence, which is the set of coordinates of that keypoint in each frame of the multi-frame image.

[0052] For example, there are N frames of images, where N is an integer greater than 1, and the coordinate sequence of a key point has N values.

[0053] The process of obtaining the key point coordinate sequence will be described in detail in subsequent embodiments, but will not be described in detail in this embodiment.

[0054] Step S204: Analyze the coordinate sequence of the at least two key points to obtain the coordinate changes of the target object in three dimensions, and obtain the analysis results;

[0055] Specifically, since the key points are known, when displacement occurs between the target dynamic object and the electronic device, the relative displacement can be at least one in the XYZ three-dimensional direction.

[0056] Among them, a pre-trained LSTM (Long Short-Term Memory) model or a preset calculation formula can be used to analyze and process the coordinate sequence of the multiple key points to obtain the coordinate changes of the target object in the three-dimensional direction, that is, to obtain the relative changes between the target dynamic object and the electronic device in the three-dimensional direction.

[0057] Specifically, based on the coordinate changes of key points on the corresponding two-dimensional plane of the image, the coordinate changes of the target dynamic object on that two-dimensional plane, as well as its changes in the direction perpendicular to that two-dimensional plane, are analyzed and determined. This process will be described in detail in subsequent embodiments.

[0058] Step S205: Based on the analysis results, determine whether displacement has occurred between the target dynamic object and the electronic device.

[0059] Step S205 is the same as step S103 in Example 1, and will not be described again in this example.

[0060] In summary, the detection method provided in this embodiment includes: analyzing at least two key points of the target dynamic object based on at least two frames of images to obtain analysis results, including: analyzing at least two key points of the target dynamic object based on at least two frames of images to obtain a coordinate sequence for each key point; and analyzing the coordinate sequence of the at least two key points to obtain the coordinate changes of the target dynamic object in the three-dimensional direction. In this scheme, the key points in each frame of images are analyzed to obtain the coordinate sequence of the key points, and the coordinate changes of the target object in the three-dimensional direction are analyzed based on the coordinate changes of the key points in each frame of images. In this scheme, the relative movement between the target dynamic object and the electronic device in the three-dimensional direction is obtained based on the coordinate change analysis in the two-dimensional image.

[0061] like Figure 3 The diagram shown is a flowchart of embodiment 3 of a control method provided in this application. The method includes the following steps:

[0062] Step S301: Obtain at least two frames of images continuously captured by the electronic device within the current time period;

[0063] Step S302: Identify the at least two frames of images to determine the target dynamic object in each frame of images and at least two key points of the target dynamic object;

[0064] Step S303: Based on the at least two frames of images, analyze at least two key points of the target dynamic object to obtain the coordinate sequence of each key point;

[0065] Steps S301-303 are the same as steps S201-203 in Example 2, and will not be described again in this example.

[0066] Step S304: Based on the analysis of the coordinate sequence of the at least two key points, determine the coordinate changes of each key point in the first and second directions;

[0067] Specifically, the coordinate sequence of the multiple key points is analyzed to determine that each key point in the multi-frame image changes, including changes in the first direction and the second direction.

[0068] The plane containing the first and second directions is the same as the two-dimensional plane of the image.

[0069] For example, in an image, the direction from bottom to top is the positive Y-axis direction, and the direction from left to right is the positive X-axis direction. This first direction can be the X-axis direction, and correspondingly, the second direction is the Y-axis direction. Then, in each frame of the image, the bottom left corner of the image can be taken as the origin (0, 0) of the XY plane.

[0070] For example, the coordinate sequence of key point A is: {(1,2),(1,3),(1,4),(1,3),……(1,2),(1,3)}. Based on the comparison of the Y-axis coordinates, it can be determined that the coordinates have changed, while the X-axis has not changed.

[0071] For example, the coordinate sequence of key point B is: {(2,2),(3,2),(4,2),(5,2),……(5,2),(3,2)}. Based on the comparison of the X-axis coordinates, it can be determined that the coordinates have changed, while the Y-axis has not changed.

[0072] For example, the coordinate sequence of key point C is: {(1,2),(2,3),(3,4),(2,3),……(1,2),(1,3)}. Based on the comparison of the X and Y axis coordinates, it can be determined that the coordinates have all changed.

[0073] Step S305: Based on the coordinate changes of the at least two key points in the first and second directions, determine the coordinate changes of the target dynamic object in at least one of the first, second, and third directions, and obtain the analysis results;

[0074] The first direction, the second direction, and the third direction are perpendicular to each other.

[0075] In practice, for the coordinate sequence of these multiple key points, the changes can be analyzed and judged by taking the difference, variance, or average value.

[0076] If keypoint A changes only its Y-axis coordinate, then the representation changes only in the second direction; if keypoint B changes only its X-axis coordinate, then the representation changes only in the first direction.

[0077] If the XY coordinates of key point C change, it indicates that the first direction and the second direction, or the first direction and the third direction, or the second direction and the third direction, or both the first direction, the second direction and the third direction have changed.

[0078] Specifically, when the magnitude of the change in the X and Y axes is the same, it can be determined that the key point C moves on the plane formed by the first direction and the second direction, specifically in the region between the first direction and the second direction, such as the first direction deviating from the second direction by 50°, etc. This application does not restrict the specific direction of the oblique movement.

[0079] In this application, the determination of whether a third-party coordinate change occurs can be based on the principle of scaling transformation. This will be explained in detail in subsequent embodiments, and will not be elaborated upon in this embodiment.

[0080] In practice, the judgment can be made by calculating the coordinate sequence of each key point through methods such as taking the difference, variance, and average.

[0081] Step S306: Based on the analysis results, determine whether displacement has occurred between the target dynamic object and the electronic device.

[0082] Step S306 is the same as step S205 in Example 2, and will not be described again in this example.

[0083] In summary, the detection method provided in this embodiment analyzes the coordinate sequence of at least two key points to obtain the coordinate changes of the target object in three dimensions. This includes: determining the coordinate changes of each key point in a first and second direction based on the analysis of the coordinate sequence of the at least two key points; and determining the coordinate changes of the target dynamic object in at least one of the first, second, and third directions based on the coordinate changes of the at least two key points in the first and second directions; wherein the first, second, and third directions are mutually perpendicular. In this scheme, the analysis of the coordinate sequence of each key point can determine the coordinate changes of each key point in the first and second directions, and further analysis based on the coordinate changes of each key point in the first and second directions determines whether the target dynamic object has changed in three mutually perpendicular directions. This achieves the process of obtaining the relative movement of the target dynamic object and the electronic device in three dimensions based on coordinate change analysis in a two-dimensional image.

[0084] like Figure 4 The diagram shown is a flowchart of embodiment 4 of a control method provided in this application. The method includes the following steps:

[0085] Step S401: Obtain at least two frames of images continuously captured by the electronic device within the current time period;

[0086] Step S402: Identify the at least two frames of images to determine the target dynamic object in each frame of images and at least two key points of the target dynamic object;

[0087] Step S403: Based on the at least two frames of images, analyze at least two key points of the target dynamic object to obtain the coordinate sequence of each key point;

[0088] Step S404: Based on the analysis of the coordinate sequence of the at least two key points, determine the coordinate changes of each key point in the first and second directions;

[0089] Steps S401-404 are the same as steps S301-304 in Example 3, and will not be described again in this example.

[0090] Step S405: Obtain the distance between the at least two key points in each of the at least two frames of images;

[0091] In practice, if the coordinates of the key point change only in the first or second direction, it indicates that the target dynamic object has not moved in the third direction. However, if the coordinates of the key point change in both the first and second directions, it is necessary to further determine whether the target dynamic object has changed in the third direction.

[0092] The third direction refers to the direction that is closer to or farther from the screen.

[0093] It should be noted that in this embodiment, the determination of whether a third-party coordinate change occurs is based on the principle of scaling transformation.

[0094] Specifically, the distance change between two key points in multiple consecutive frames of images represents the change in the distance of the corresponding features of the key point in the image. The relative positional relationship between the key points is fixed. The change in distance represents the change in the size of the target dynamic object in the image. When the image is captured, the size of the captured image is fixed. The change in the size of the target dynamic object in the captured multiple frames of images is due to the change in the distance between the target dynamic object and the camera. Therefore, the distance change between two key points in multiple consecutive frames of images is due to the target dynamic object moving closer to or further away from the camera (i.e., moving in a third direction).

[0095] Specifically, select any two key points and calculate the distance based on their coordinates.

[0096] For example, the coordinates of keypoint A and keypoint B in the first frame are (1,2) and (2,2) respectively; in the second frame, they are (1,3) and (3,2) respectively; in the third frame, they are (1,4) and (4,2) respectively; and in the fourth frame, they are (1,3) and (5,2) respectively...

[0097] In the first frame image, the distance between keypoints A and B is 1, and in the second frame image, the distance between keypoints A and B is... The distance between key points A and B in the third frame image is The distance between key points A and B in the fourth frame image is ...

[0098] Of course, in practice, to improve the accuracy of the judgment, when there are more than two key points, the key points can be combined in pairs to calculate the distance.

[0099] Step S406: Based on the distance change between the two key points in the at least two frames of images, determine the coordinate change of the target dynamic object in a third direction, and obtain the analysis result;

[0100] The principle of scaling transformation is used to determine whether a third-party coordinate change occurs.

[0101] Specifically, if the distance between any two key points in the multi-frame image changes—for example, if they all gradually increase, or all gradually decrease, or all gradually increase and then gradually decrease, or all gradually decrease and then gradually increase—

[0102] It should be noted that during the shooting process, when the target moving object moves closer to the camera, its image becomes larger in the image, and the distance between the two key points increases accordingly; when the target moving object moves away from the camera, its image becomes smaller in the image, and the distance between the two key points decreases accordingly. The movement of the target moving object relative to the camera is a change in the third direction. Therefore, it is only necessary to determine the coordinate change of the target moving object in the third direction based on the change in the distance between the two key points.

[0103] Step S407: Based on the analysis results, determine whether displacement has occurred between the target dynamic object and the electronic device.

[0104] Step S407 is the same as step S306 in Example 3, and will not be described again in this example.

[0105] In summary, the detection method provided in this embodiment determines the coordinate change of the target dynamic object in a third direction based on the coordinate changes of the at least two key points in a first and second direction. This includes: obtaining the distance between the at least two key points in each of the at least two frames of images; and determining the coordinate change of the target dynamic object in a third direction based on the distance change between the two key points in the at least two frames of images. In this solution, the distance change between two key points in multiple consecutive frames of images is due to the target dynamic object moving closer to or further away from the camera (i.e., moving in a third direction). Therefore, in this solution, it is only necessary to determine that the distance between the two key points in each frame of images has changed to determine that the target dynamic object has undergone a coordinate change in a third direction.

[0106] like Figure 5 The diagram shown is a flowchart of embodiment 5 of a control method provided in this application. The method includes the following steps:

[0107] Step S501: Obtain at least two frames of images continuously captured by the electronic device within the current time period;

[0108] Step S502: Identify the at least two frames of images to determine the target dynamic object in each frame of images and at least two key points of the target dynamic object;

[0109] Steps S501-502 are the same as steps S201-202 in Example 2, and will not be described again in this example.

[0110] Step S503: Determine the coordinates of each key point in each frame of the image;

[0111] Specifically, the coordinates of each key point in each frame of the image are determined.

[0112] In practice, a fixed position of the image can be used as the origin, such as the lower left corner or the upper left corner. This application does not restrict the position of the origin.

[0113] For example, if there are 30 frames of images, and each frame contains 5 key points, then the 30 frames are analyzed sequentially to obtain the coordinates of the 5 key points in each frame.

[0114] Correspondingly, in each frame of the image, the coordinates of 5 key points are obtained sequentially, resulting in 30 sets of coordinates for 5 key points.

[0115] In practice, there are several ways to determine the coordinates of key points in each frame of an image, such as using a heatmap, a preset mathematical calculation formula, or a regression method.

[0116] Heatmaps can use color variations to reflect data information in two-dimensional matrices or tables, and can intuitively represent the magnitude of data values ​​with defined color intensities.

[0117] In practice, a pre-trained neural network model sequentially identifies features in each frame of the image, generating a heatmap for each feature. Each pixel in the heatmap corresponds to a confidence level, which represents the confidence level in the identified feature. A higher confidence level indicates a greater probability that the pixel represents that feature, while a lower confidence level indicates a lower probability. The coordinates of the point with the highest confidence level in the heatmap are then selected as the coordinates of the corresponding keypoint.

[0118] For example, 30 consecutively acquired images, each containing 256×256 pixels, are used to identify the eyes of a face. A trained neural network model is then used to identify the eyes in these 30 images, resulting in 30 heatmaps. Each pixel in each heatmap corresponds to a confidence level, with a value range of (0,1). The coordinates of the point with the highest confidence level are selected as the keypoint coordinates for the corresponding image.

[0119] In practice, based on the above process, the coordinates of each key point whose coordinates are to be determined are determined sequentially.

[0120] Alternatively, regression analysis or mathematical formulas can be used to determine the coordinates of each key point in each frame of an image.

[0121] The regression method will be explained in detail in the subsequent embodiment 6, and will not be described in detail in this embodiment.

[0122] In practice, a pre-defined processing model can be used, which employs a neural network model. Through pre-learning and training, the aforementioned methods for determining the coordinates of key points in each frame of an image can be implemented in the processing model.

[0123] Step S504: Collect the coordinates of each key point in the at least two frames of images to obtain the coordinate sequence of each key point;

[0124] Specifically, by sequentially taking the coordinates of the same key point in each image according to the image order, the coordinate sequence of that key point is obtained.

[0125] For example, in N frames of images, the coordinates of key point A are (1, 2) in the first frame, (1, 3) in the second frame, (1, 4) in the third frame, (1, 3) in the fourth frame, ..., (1, 2) in the (N-1)th frame, and (1, 3) in the Nth frame. The set of coordinates of key point A yields the coordinate sequence of key point A as: {(1, 2), (1, 3), (1, 4), (1, 3), ..., (1, 2), (1, 3)}.

[0126] Correspondingly, if each frame of the image includes 3 key points, then the coordinate sequence of the 3 key points is obtained in sequence.

[0127] Step S505: Analyze the coordinate sequence of the at least two key points to obtain the coordinate changes of the target object in three dimensions, and obtain the analysis results;

[0128] Step S506: Based on the analysis results, determine whether displacement has occurred between the target dynamic object and the electronic device.

[0129] Steps S505-506 are the same as steps S204-205 in Example 2, and will not be described again in this example.

[0130] In summary, the detection method provided in this embodiment analyzes at least two key points of the target dynamic object based on at least two frames of images to obtain a coordinate sequence for each key point. This includes: determining the coordinates of each key point in each frame of images; and collecting the coordinates of each key point in the at least two frames of images to obtain a coordinate sequence for each key point. In this scheme, the coordinates of each key point in each frame of images are first determined, and then the coordinates of identical key points are collected to obtain a coordinate sequence for that key point, thus achieving the determination of the coordinate sequence of each key point in at least two frames of images.

[0131] like Figure 6 The diagram shown is a flowchart of embodiment 6 of a control method provided in this application. The method includes the following steps:

[0132] Step S601: Obtain at least two frames of images continuously captured by the electronic device within the current time period;

[0133] Step S602: Identify the at least two frames of images to determine the target dynamic object in each frame of images and at least two key points of the target dynamic object;

[0134] Steps S601-602 are the same as steps S501-502 in Example 5, and will not be described again in this example.

[0135] Step S603: Obtain the offset of each key point in each frame of the image from the center point of the target region, and determine the coordinates of the center point in each frame of the image based on the specified location;

[0136] The target dynamic object is located within the target area, and the specified position is a fixed position serving as the origin.

[0137] In this embodiment, the coordinates of key points can be determined based on regression analysis or mathematical calculation formulas.

[0138] In each frame of the image, after a target dynamic object is identified, a target region is selected based on the target dynamic object. The target dynamic object is located within the target region. When the position of the target dynamic object changes in the image, the target region moves accordingly to follow the position change of the target dynamic object.

[0139] In practice, the target area can be a square or a rectangle, and this application does not impose any restrictions on the specific shape of the target area.

[0140] Specifically, a two-dimensional coordinate system is set in each frame of the image, and a fixed position in the image is selected as the origin, such as the lower left corner or the upper left corner of the image.

[0141] After determining the target area, the coordinates of the center point of the target area are determined.

[0142] Specifically, a neural network is pre-trained to perform regression processing. This neural network is used to obtain the offset between the center point of the target region and the key point in each frame of the image.

[0143] The mathematical calculation formula can be a pre-set mathematical calculation formula, based on which the offset between the key point and the center point is calculated.

[0144] Since the relative positions of each key point in each target dynamic object are determined, such as the left eye being to the right of the left ear and the nose being to the lower right of the left eye, and the relative positional relationship between the center point of the target area and the key points is also determined.

[0145] As the target moving object moves closer to the camera during the shooting process, its image becomes larger in the image, and the distance between the two key points increases accordingly. Conversely, as the target moving object moves further away from the camera during the shooting process, its image becomes smaller in the image, and the distance between the two key points decreases accordingly. Therefore, the overall magnification or reduction of the target moving object's image in the image corresponds proportionally to the offset between the key points and the center point.

[0146] Specifically, after analyzing the first frame and determining the offset between the keypoints and the center point, the second frame can be analyzed based on the area of ​​the target region. In the first frame, the target region area is 5, and the offset between the keypoints and the center point (O... x O y The values ​​of ) are (4,4). In the second frame image, the area of ​​the target region is calculated to be 1, so the ratio of the target region area in the first frame image to the target region area in the second frame image is 1:5. Correspondingly, the offset is reduced proportionally to obtain the offset (O) between the key point and the center point in the second frame image. x O y The values ​​of ) are (0.8, 0.8).

[0147] By using mathematical calculation formulas, only the offset of the first frame image needs to be identified. For the remaining frames, the offset can be calculated proportionally based on the area of ​​the target region.

[0148] Step S604: Based on the coordinates of the center point and the offset, obtain the coordinates of each key point of the target dynamic object;

[0149] Given the coordinates of the center point and the offset between the target dynamic object and the center point, the coordinates of the key point can be obtained by performing mathematical calculations based on these two information.

[0150] For example, the center point (C x C y) The offset (O) between the key point and the center point was detected. x O y Based on the above results, the key point coordinates (C1, C2, ...) of the dynamic target / dynamic object are obtained. x +O x C y +O y Each keypoint in each frame of this multi-frame image can be calculated using the method described above.

[0151] In practice, the coordinates of all key points can be obtained by sequentially identifying each frame of the multi-frame image.

[0152] Step S605: Collect the coordinates of each key point in the at least two frames of images to obtain the coordinate sequence of each key point;

[0153] Step S606: Analyze the coordinate sequence of the at least two key points to obtain the coordinate changes of the target object in three dimensions, and obtain the analysis results;

[0154] Step S607: Based on the analysis results, determine whether displacement has occurred between the target dynamic object and the electronic device.

[0155] Steps S605-607 are the same as steps S504-506 in Example 5, and will not be described again in this example.

[0156] In summary, the detection method provided in this embodiment, which determines the coordinates of each key point in each frame of an image, includes: obtaining the offset of each key point in each frame of an image from the center point of a target region, and determining the coordinates of the center point in each frame of an image based on a specified position; wherein the target dynamic object is located within the target region, and the specified position is a fixed position serving as the origin; and obtaining the coordinates of each key point of the target dynamic object based on the coordinates of the center point and the offset. In this scheme, a target region is determined in each frame of an image, the offset of each key point in the same image from the center point of the target region is identified, and the coordinates of the key point are calculated based on the coordinates of the center point and the offset, thereby realizing the determination of the coordinates of each key point in each frame of an image.

[0157] like Figure 7 The diagram shown is a flowchart of embodiment 7 of a control method provided in this application. The method includes the following steps:

[0158] Step S701: Obtain at least two frames of images continuously captured by the electronic device within the current time period;

[0159] Step S702: Identify the at least two frames of images to determine the target dynamic object in each frame of images and at least two key points of the target dynamic object;

[0160] Steps S701-702 are the same as steps S601-602 in Example 6, and will not be described again in this example.

[0161] Step S703: Determine the target region where the target dynamic object is located based on the target dynamic object;

[0162] In this process, after the target dynamic object is obtained through analysis, a target region is determined based on the area where the target dynamic object is located, and the target dynamic object is located within the target region.

[0163] The position and size of the target area change with the position and size of the target dynamic object.

[0164] For example, when the position of the target dynamic object changes in the image, the target region moves accordingly following the position change of the target dynamic object.

[0165] For example, when the area of ​​the target dynamic object changes in the image, the area of ​​the target region changes accordingly.

[0166] In practice, the target area can be a square or a rectangle, and this application does not impose any restrictions on the specific shape of the target area.

[0167] Step S704: Obtain the offset of each key point in each frame image from the center point of the target region, and determine the coordinates of the center point in each frame image based on the specified location;

[0168] Step S705: Based on the coordinates of the center point and the offset, obtain the coordinates of each key point of the target dynamic object;

[0169] Step S706: Collect the coordinates of each key point in the at least two frames of images to obtain the coordinate sequence of each key point;

[0170] Step S707: Analyze the coordinate sequence of the at least two key points to obtain the coordinate changes of the target object in three dimensions, and obtain the analysis results;

[0171] Step S708: Based on the analysis results, determine whether displacement has occurred between the target dynamic object and the electronic device.

[0172] Steps S704-708 are the same as steps S603-607 in Example 6, and will not be described again in this example.

[0173] In summary, the detection method provided in this embodiment further includes: determining the target region where the target dynamic object is located based on the target dynamic object. In this solution, a target region is determined based on the target dynamic object so that subsequent processing can be performed in conjunction with this target region.

[0174] like Figure 8 The diagram shown is a flowchart of embodiment 8 of a control method provided in this application. The method includes the following steps:

[0175] Step S801: Obtain at least two frames of images continuously captured by the electronic device within the current time period;

[0176] Step S802: Identify the at least two frames of images to determine the target dynamic object in each frame of images and at least two key points of the target dynamic object;

[0177] Steps S801-802 are the same as steps S601-602 in Example 6, and will not be described again in this example.

[0178] Step S803: Obtain the coordinate sequence of at least one specified point in the target region in the at least two frames of images, wherein the specified point is located at the edge of the target region;

[0179] In practice, analysis and processing can be carried out by combining specified points in the target area with key points of the target dynamic object.

[0180] Specifically, the specified point can be any point on the edge of the target area, and the specified point is different from the keypoint of the target dynamic object.

[0181] For example, if the target area is rectangular, the specified point can be any point on the border of the rectangular area, a point on the side, or a vertex where the sides intersect.

[0182] Step S804: Obtain the offset of each key point in each frame of the image from the center point of the target region, and determine the coordinates of the center point in each frame of the image based on the specified location;

[0183] Step S805: Based on the coordinates of the center point and the offset, obtain the coordinates of each key point of the target dynamic object;

[0184] Step S806: Collect the coordinates of each key point in the at least two frames of images to obtain the coordinate sequence of each key point;

[0185] Steps S804-806 are the same as steps S603-605 in Example 6, and will not be described again in this example.

[0186] Step S807: Based on the analysis of the coordinate sequences of the at least two key points and the coordinate sequences of the at least one designated point, determine the coordinate changes of each key point and the designated point in the first and second directions;

[0187] Step S808: Based on the coordinate changes of each key point and the specified point in the first and second directions, determine the coordinate changes of the target dynamic object in at least one of the first, second, and third directions, and obtain the analysis results;

[0188] The first direction, the second direction, and the third direction are perpendicular to each other.

[0189] In the process of determining whether the target dynamic object has undergone coordinate changes in at least one of the three directions based on key points and designated points, the designated point can be regarded as a special key point, which enriches the data basis for analysis. The explanation of this step S807-808 refers to steps S304-305 in embodiment 3. Its specific implementation is similar to the process in embodiments 3-7. The specific implementation process can be explained by referring to embodiments 3-7. It will not be described in detail in this embodiment.

[0190] Step S809: Based on the analysis results, determine whether displacement has occurred between the target dynamic object and the electronic device.

[0191] Step S809 is the same as step S607 in Example 6, and will not be described again in this example.

[0192] In summary, this embodiment provides a detection method that obtains the coordinate sequence of at least one designated point in a target region within at least two frames of images. The designated point is located at the edge of the target region. The method analyzes the coordinate sequence of the at least two key points to obtain the coordinate changes of the target object in three dimensions. This includes: determining the coordinate changes of each key point and the designated point in a first and second direction based on the analysis of the coordinate sequences of the at least two key points and the at least one designated point; and determining the coordinate changes of the target dynamic object in at least one of the first, second, and third directions based on the coordinate changes of each key point and the designated point in the first and second directions. The first, second, and third directions are mutually perpendicular. In this solution, at least one designated point in the target region is added and analyzed in combination with multiple key points of the target dynamic object to determine whether the target dynamic object has changed in three mutually perpendicular directions. This achieves the process of obtaining the relative movement of the target dynamic object and the electronic device in three dimensions based on coordinate change analysis in a two-dimensional image.

[0193] Corresponding to the detection method embodiment provided in this application above, this application also provides an apparatus embodiment for applying the detection method.

[0194] like Figure 9 The diagram shown is a structural schematic of an embodiment 1 of a detection device provided in this application. The device includes the following components: an acquisition module 901, an identification module 902, and an analysis module 903.

[0195] The acquisition module 901 is used to acquire at least two frames of images continuously collected by the electronic device within the current time period;

[0196] The recognition module 902 is used to recognize the at least two frames of images, and determine the target dynamic object in each frame of the image and at least two key points of the target dynamic object, wherein the dynamic object represents an object that can undergo displacement, and the key points represent the features of the target dynamic object.

[0197] The analysis module 903 is used to analyze at least two key points of the target dynamic object based on the at least two frames of images to obtain analysis results; and based on the analysis results, to determine whether displacement has occurred between the target dynamic object and the electronic device.

[0198] Optionally, this analysis module is used for:

[0199] Based on the at least two frames of images, at least two key points of the target dynamic object are analyzed to obtain the coordinate sequence of each key point;

[0200] By analyzing the coordinate sequence of the at least two key points, the coordinate changes of the target dynamic object in three dimensions are obtained.

[0201] Optionally, this analysis module is used for:

[0202] Based on the analysis of the coordinate sequence of the at least two key points, determine the coordinate changes of each key point in the first and second directions;

[0203] Based on the coordinate changes of the at least two key points in the first and second directions, determine the coordinate changes of the target dynamic object in at least one of the first, second, and third directions;

[0204] The first direction, the second direction, and the third direction are perpendicular to each other.

[0205] Optionally, this analysis module is used for:

[0206] Obtain the distance between at least two key points in each of the at least two frames of images;

[0207] Based on the distance change between the two key points in the at least two frames of images, the coordinate change of the target dynamic object in a third direction is determined.

[0208] Optionally, this analysis module is used for:

[0209] Determine the coordinates of each key point in each frame of the image;

[0210] The coordinate set of each key point in the at least two frames of images is used to obtain the coordinate sequence of each key point.

[0211] Optionally, this analysis module is used for:

[0212] The offset of each key point in each frame of the image from the center point of the target region is obtained, and the coordinates of the center point in each frame of the image are determined based on a specified position; wherein, the target dynamic object is located within the target region, and the specified position is a fixed position serving as the origin;

[0213] Based on the coordinates of the center point and the offset, the coordinates of each key point of the target dynamic object are obtained.

[0214] Optionally, after identifying the target dynamic object in each of the at least two frames of images, the analysis module is used to:

[0215] The target region where the target dynamic object is located is determined based on the target dynamic object.

[0216] Optionally, based on at least two frames of images, at least two key points of the target dynamic object are analyzed. Before obtaining the analysis results, the analysis module is used to:

[0217] Obtain a sequence of coordinates of at least one designated point in the target region in the at least two frames of images, wherein the designated point is located at the edge of the target region.

[0218] Optionally, the analysis module is used for:

[0219] Based on the analysis of the coordinate sequences of the at least two key points and the coordinate sequences of the at least one designated point, the coordinate changes of each key point and the designated point in the first and second directions are determined.

[0220] Based on the coordinate changes of each key point and the specified point in the first and second directions, determine the coordinate changes of the target dynamic object in at least one of the first, second, and third directions;

[0221] The first direction, the second direction, and the third direction are perpendicular to each other.

[0222] For an explanation of the functions of each component of the device, please refer to the method embodiment; this embodiment will not elaborate further.

[0223] In summary, this embodiment provides a detection device comprising: an acquisition module for acquiring at least two consecutive frames of images captured by an electronic device within a current time period; an identification module for identifying the at least two frames of images, determining a target dynamic object in each frame and at least two key points of the target dynamic object, wherein the dynamic object represents an object capable of displacement, and the key points represent features of the target dynamic object; and an analysis module for analyzing the at least two key points of the target dynamic object based on the at least two frames of images to obtain analysis results; and determining, based on the analysis results, whether displacement has occurred between the target dynamic object and the electronic device. In this solution, by analyzing multiple consecutive frames of images captured by the electronic device, a target dynamic object capable of displacement in the images and multiple key points of the target dynamic object are identified. Based on these multiple key points, analysis is performed to determine whether displacement has occurred during the acquisition of the multiple consecutive frames of images. This displacement is generated by relative movement between the target moving object and the electronic device. This relative movement can be caused by movement of the electronic device, movement of the target dynamic object, or even movement of both the electronic device and the target dynamic object. It is not limited to detecting only the shaking of the electronic device, thus improving the accuracy of detection.

[0224] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The apparatus provided in the embodiments is described simply because it corresponds to the method provided in the embodiments; relevant parts can be found in the method section.

[0225] The above description of the provided embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features provided herein.

Claims

1. A detection method, comprising: Obtain at least two frames of images continuously captured by the electronic device within the current time period; The at least two frames of images are identified to determine the target dynamic object in each frame of the image and at least two key points of the target dynamic object, wherein the dynamic object represents an object that can undergo displacement, and the key points represent the features of the target dynamic object. Based on the at least two frames of images, at least two key points of the target dynamic object are analyzed to obtain the coordinate sequence of each key point; Based on the analysis of the coordinate sequence of the at least two key points, determine the coordinate changes of each key point in the first and second directions; Obtain the distance between at least two key points in each of the at least two frames of images; Based on the distance change between the two key points in the at least two frames of images, the coordinate change of the target dynamic object in a third direction is determined; the first direction, the second direction, and the third direction are perpendicular to each other; Based on the coordinate changes of the target dynamic object in three dimensions, it is determined whether displacement has occurred between the target dynamic object and the electronic device.

2. The method of claim 1, further comprising: Based on the coordinate changes of the at least two key points in the first and second directions, the coordinate changes of the target dynamic object in at least one of the first and second directions are determined.

3. The method according to claim 1, wherein analyzing at least two key points of the target dynamic object based on the at least two frames of images to obtain a coordinate sequence for each key point includes: Determine the coordinates of each key point in each frame of the image; The coordinate set of each key point in the at least two frames of images is used to obtain the coordinate sequence of each key point.

4. The method according to claim 3, wherein determining the coordinates of each key point in each frame of the image comprises: The offset of each key point in each frame of the image from the center point of the target region is obtained, and the coordinates of the center point in each frame of the image are determined based on a specified position; wherein, the target dynamic object is located within the target region, and the specified position is a fixed position serving as the origin; Based on the coordinates of the center point and the offset, the coordinates of each key point of the target dynamic object are obtained.

5. The method according to claim 4, after identifying the target dynamic object in each of the at least two frames of images, further comprising: The target region where the target dynamic object is located is determined based on the target dynamic object.

6. The method according to claim 4, further comprising, before obtaining the analysis results by analyzing at least two key points of the target dynamic object based on at least two frames of images: Obtain a sequence of coordinates of at least one designated point in the target region in the at least two frames of images, wherein the designated point is located at the edge of the target region.

7. The method according to claim 6, wherein determining the coordinate changes of the target dynamic object in the three-dimensional direction includes: Based on the analysis of the coordinate sequences of the at least two key points and the coordinate sequences of the at least one designated point, the coordinate changes of each key point and the designated point in the first and second directions are determined. Based on the coordinate changes of each key point and the specified point in the first and second directions, determine the coordinate changes of the target dynamic object in at least one of the first, second, and third directions; The first direction, the second direction, and the third direction are perpendicular to each other.

8. A detection device, comprising: The acquisition module is used to acquire at least two frames of images continuously captured by the electronic device within the current time period; The recognition module is used to recognize the at least two frames of images, determine the target dynamic object in each frame of image and at least two key points of the target dynamic object, wherein the dynamic object represents an object that can undergo displacement, and the key points represent the features of the target dynamic object; The analysis module is used to analyze at least two key points of the target dynamic object based on the at least two frames of images to obtain a coordinate sequence for each key point; based on the analysis of the coordinate sequences of the at least two key points, determine the coordinate changes of each key point in a first direction and a second direction; obtain the distance between the at least two key points in each frame of the at least two frames of images; based on the distance changes between the two key points in the at least two frames of images, determine the coordinate changes of the target dynamic object in a third direction; the first direction, the second direction, and the third direction are mutually perpendicular; based on the coordinate changes of the target dynamic object in three dimensions, determine whether displacement has occurred between the target dynamic object and the electronic device.

Images