A remote security management and control method and system based on image recognition

Abstract

The application discloses a remote security control method and system based on image recognition, relates to the technical field of image recognition and intelligent security, and responds to a lock opening trigger event, collects continuous image frames in a door opening process to obtain a door opening process image sequence, performs image recognition, extracts door body movement reference features, determines the view angle change parameters of each image frame relative to a reference frame, maps target positions in each image frame to a unified door safety area coordinate system to obtain registered target position data, performs human body target detection and target correlation on the door opening process image sequence, generates the movement trajectories of authorized user targets and unauthorized targets, determines the distance change and movement direction of the unauthorized targets relative to the door gap area and the authorized user targets, generates a risk identification result, and sends a remote security control instruction. Through view angle registration, trajectory analysis and risk linkage control, the application realizes more accurate and timely remote security early warning and control.

Description

Technical Field

[0001] This application relates to the fields of image recognition and intelligent security technology, and in particular to a remote security control method and system based on image recognition. Background Technology

[0002] With the development of smart locks, video peepholes, remote access control, and home security equipment, image recognition-based remote security control technology is widely used in scenarios such as resident identification, visitor recognition, remote unlocking, and abnormal behavior alarms. Existing technologies typically use smart lock cameras to capture images of the area in front of the door, detect faces, bodies, or hand movements in the images, and combine local or remote recognition results to determine whether to allow unlocking or send an alarm. However, in actual use, smart lock cameras are often installed on the surface of the door or lock body, and the camera's viewing angle is not fixed. When an authorized user unlocks the door and pushes it, the camera rotates or translates with the door, causing a significant positional drift of the human target in the continuous images.

[0003] Under the aforementioned dynamic perspective, traditional target detection, tracking, or behavior recognition methods based on fixed surveillance cameras are prone to problems such as target loss, trajectory jumps, and incorrect personnel identification. Especially in scenarios involving tailgating, unauthorized individuals often approach the door crack, follow authorized users closely, or use the door to obstruct entry within a short time after it opens. This process is brief, involves significant mutual obstruction of targets, and drastic changes in the camera's perspective. If judgment is based solely on single-frame face recognition or ordinary human detection, it is easy to miss tailgating targets or misjudge normal behaviors such as neighbors passing by, accompanying family members, or delivery personnel leaving as abnormal behavior.

[0004] Therefore, how to establish a stable spatial coordinate relationship at the doorway based on the motion perspective image during the opening process of a smart lock, and how to continuously track authorized users and unauthorized targets even when the image coordinates drift, is an urgent problem to be solved in remote security management. Summary of the Invention

[0005] To address the aforementioned shortcomings, this application provides a remote security control method and system based on image recognition, which solves the problems of image coordinate drift, unstable target trajectory, and difficulty in accurately identifying tailgating risks caused by the movement of smart lock cameras with the door. This method extracts image reference features such as the door frame, door gaps, floor texture, and fixed points on the wall during the door opening process to estimate the camera's viewing angle changes. It then maps the human target positions in different frames to a unified doorway security area coordinate system. Finally, it identifies tailgating risks based on the relative trajectory relationship between authorized users and unauthorized targets, thereby generating corresponding remote security control commands.

[0006] Firstly, this application provides a remote security control method based on image recognition, including:

[0007] In response to the unlocking trigger event, continuous image frames are acquired during the door opening process to obtain an image sequence of the door opening process;

[0008] Image recognition is performed on the image sequence of the door opening process to extract door motion reference features, which include at least one of the following: door frame edge, door seam line, threshold line, ground texture or wall fixed point;

[0009] Based on the positional changes of the door motion reference features in different image frames, the perspective change parameters of each image frame relative to the reference frame are determined.

[0010] Based on the aforementioned viewpoint change parameters, the target positions in each image frame are mapped to a unified doorway safety area coordinate system to obtain registered target position data.

[0011] Human target detection and target association are performed on the image sequence of the door opening process, and the motion trajectories of authorized user targets and unauthorized targets in the coordinate system of the doorway security area are generated by combining the registered target position data;

[0012] Based on the motion trajectory, the distance change and motion direction of the unauthorized target relative to the door gap area and the authorized user target are determined, and a risk identification result is generated;

[0013] Based on the risk identification results, remote security control instructions are generated and sent.

[0014] Optionally, in response to the unlocking trigger event, acquiring continuous image frames during the door opening process to obtain an image sequence of the door opening process includes:

[0015] In response to the detection of an unlock trigger event, the trigger time is obtained;

[0016] The image acquisition start time is determined based on the trigger time, and continuous image frames during the door opening process are acquired at a preset frame rate within a preset acquisition duration.

[0017] The consecutive image frames are sorted according to the acquisition time to obtain the door opening process image sequence.

[0018] Optionally, after obtaining the image sequence of the door opening process, the method further includes:

[0019] Brightness equalization, distortion correction, and sharpness evaluation are performed on each image frame in the image sequence of the door opening process to obtain the image quality value of each image frame;

[0020] The image quality value is compared with a preset quality threshold, and image frames with image quality values ​​less than the preset quality threshold are removed to obtain a sequence of image frames to be identified.

[0021] The sequence of image frames to be identified is used as the image sequence for image recognition.

[0022] Optionally, the step of performing image recognition on the door opening process image sequence and extracting door movement reference features includes:

[0023] Edge detection is performed on the image frames in the image frame sequence to be identified to obtain a set of edge pixels;

[0024] Line fitting is performed on the edge pixel set to obtain candidate line features;

[0025] Based on the direction, length, and image position of the candidate line features, determine the door frame edge, door seam line, or threshold line from the candidate line features;

[0026] Corner detection and texture detection are performed on the image frames in the image frame sequence to be identified to obtain candidate fixed feature points;

[0027] Based on the matching relationship between the candidate fixed feature points in adjacent image frames, the ground texture or wall fixed points are determined.

[0028] Optionally, determining the viewpoint change parameters of each image frame relative to the reference frame based on the positional changes of the door motion reference features in different image frames includes:

[0029] Image frames that are selected from the image frame sequence to be identified before the door is opened or when the door opening angle is less than a preset angle are used as reference frames.

[0030] Determine the reference features in the reference frame, and determine the current reference features in the current image frame;

[0031] The reference feature is matched with the current reference feature to obtain a reference feature matching pair;

[0032] Based on the coordinate differences between the reference feature matching pair in the reference frame and the current image frame, the viewpoint change parameters of the current image frame relative to the reference frame are determined.

[0033] Optionally, the viewpoint change parameters include translation parameters, rotation parameters, scale parameters, and perspective parameters; determining the viewpoint change parameters of the current image frame relative to the reference frame based on the coordinate difference between the reference feature matching pair in the reference frame and the current image frame includes:

[0034] The translation parameters are determined based on the pixel coordinate difference between the reference frame and the current image frame using the same gate motion reference feature.

[0035] The rotation parameters are determined based on the direction difference between the door frame edge, door seam line, or threshold line in the reference frame and the current image frame.

[0036] The scale parameter is determined based on the distance change between at least two reference feature matching pairs;

[0037] The perspective parameters are determined based on the pixel coordinates of at least four reference feature matching pairs.

[0038] Optionally, the step of mapping the target position in each image frame to a unified doorway security area coordinate system based on the viewpoint change parameters to obtain registered target position data includes:

[0039] The door frame edge, door seam line, or threshold line in the reference frame are used as coordinate references to establish the coordinate system of the doorway safety area;

[0040] Mark the door gap area, threshold area, and waiting area outside the door in the coordinate system of the doorway safety area;

[0041] Based on the perspective change parameters corresponding to the current image frame, determine the coordinate transformation relationship from the current image frame to the coordinate system of the doorway safety area;

[0042] The target location is obtained as the center point of the target detection box, the target foot point, or the target contour point in the current image frame.

[0043] The target position is mapped according to the coordinate transformation relationship to obtain the registered target position data.

[0044] Optionally, the step of performing human target detection and target association on the door opening process image sequence, and combining the registered target position data to generate the motion trajectories of authorized user targets and unauthorized targets in the coordinate system of the doorway security area, includes:

[0045] Human detection is performed on each image frame in the door opening process image sequence to obtain human target detection results, which include target identifier, target detection box and target appearance features;

[0046] Based on the target appearance features of human target detection results in adjacent image frames and the registered target location data, human targets are associated to obtain target association results.

[0047] Based on the authentication information corresponding to the unlocking trigger event, the authorized user target is determined from the target association result, and human targets that are different from the authorized user target are determined as unauthorized targets;

[0048] Arrange the registered target position data corresponding to the authorized user target and the unauthorized target according to the acquisition time, and generate the motion trajectory of the authorized user target and the unauthorized target.

[0049] Optionally, based on the motion trajectory, determining the distance change and movement direction of the unauthorized target relative to the door gap area and the authorized user target, and generating a risk identification result, includes:

[0050] Based on the motion trajectory of the unauthorized target, calculate the first distance sequence between the unauthorized target and the door gap area at multiple acquisition times;

[0051] Based on the motion trajectory of the unauthorized target and the motion trajectory of the authorized user target, calculate a second distance sequence between the unauthorized target and the authorized user target at multiple acquisition times;

[0052] Based on the registered target position data of the unauthorized target at adjacent acquisition times, the movement direction of the unauthorized target is determined;

[0053] The first distance sequence, the second distance sequence, and the direction of movement are matched with preset risk determination conditions to obtain risk identification results.

[0054] Secondly, this application provides a remote security control system based on image recognition, comprising:

[0055] The image acquisition module is used to acquire continuous image frames during the door opening process in response to the unlocking trigger event, and obtain the door opening process image sequence;

[0056] The reference feature extraction module is used to perform image recognition on the door opening process image sequence and extract door motion reference features, which include at least one of the following: door frame edge, door seam line, threshold line, ground texture or wall fixed point;

[0057] The perspective change determination module is used to determine the perspective change parameters of each image frame relative to the reference frame based on the position change of the door motion reference feature in different image frames.

[0058] The image registration module is used to map the target position in each image frame to a unified doorway security area coordinate system based on the viewpoint change parameters, so as to obtain the registered target position data.

[0059] The target trajectory generation module is used to perform human target detection and target association on the door opening process image sequence, and combine the registered target position data to generate the motion trajectory of authorized user targets and unauthorized targets in the coordinate system of the doorway security area;

[0060] The risk identification module is used to determine the distance change and movement direction of the unauthorized target relative to the door gap area and the authorized user target based on the movement trajectory, and generate risk identification results;

[0061] The remote control module is used to generate and send remote security control commands based on the risk identification results.

[0062] Compared with the prior art, the beneficial effects of the present invention are:

[0063] This application acquires continuous image frames during the door opening process after the unlock is triggered. By extracting door movement reference features such as the door frame edge, door seam line, threshold line, floor texture, or fixed points on the wall, it determines the viewing angle change parameters between different image frames, thereby mapping the target position in each frame to a unified doorway safety area coordinate system. This reduces the impact of camera viewing angle changes on target position judgment during door opening, making the human target trajectory more stable and accurate.

[0064] This application combines human target detection, target association, and identity verification information to distinguish between authorized and unauthorized user targets. Based on the distance changes and movement directions of unauthorized targets relative to the door gap area and authorized user targets, it generates risk identification results, enabling more precise identification of abnormal entry behavior. Finally, based on the risk identification results, it generates and sends remote security control commands, allowing users or security platforms to receive timely image push notifications, alarm prompts, door lock hold information, and other control information, improving the accuracy, timeliness, and reliability of remote security control. Attached Figure Description

[0065] Figure 1 A flowchart illustrating a remote security control method based on image recognition, provided as an embodiment of this application;

[0066] Figure 2 A flowchart for generating risk identification results provided in this application embodiment;

[0067] Figure 3 This is a schematic diagram of a remote security control system based on image recognition, provided as an embodiment of this application. Detailed Implementation

[0068] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0069] like Figure 1 As shown in the flowchart, an embodiment of this application provides a remote security control method based on image recognition, including steps S101 to S107, wherein:

[0070] S101, in response to the unlocking trigger event, acquires continuous image frames during the door opening process to obtain an image sequence of the door opening process;

[0071] S102, perform image recognition on the door opening process image sequence and extract door motion reference features, the door motion reference features include at least one of the following: door frame edge, door seam line, threshold line, ground texture or wall fixed point;

[0072] S103, Based on the positional changes of the door motion reference features in different image frames, determine the perspective change parameters of each image frame relative to the reference frame;

[0073] S104, Based on the aforementioned viewpoint change parameters, the target positions in each image frame are mapped to a unified doorway safety area coordinate system to obtain registered target position data;

[0074] S105, perform human target detection and target association on the image sequence of the door opening process, and combine the registered target position data to generate the motion trajectory of authorized user target and unauthorized target in the coordinate system of the doorway security area;

[0075] S106, Based on the motion trajectory, determine the distance change and motion direction of the unauthorized target relative to the door gap area and the authorized user target, and generate a risk identification result;

[0076] S107, Based on the risk identification results, generate and send remote security control instructions.

[0077] Regarding the above S101:

[0078] As an optional implementation, in response to an unlocking trigger event, continuous image frames are acquired during the door opening process to obtain an image sequence of the door opening process, including:

[0079] In response to the detection of an unlock trigger event, the trigger time is obtained;

[0080] The image acquisition start time is determined based on the trigger time, and continuous image frames during the door opening process are acquired at a preset frame rate within a preset acquisition duration.

[0081] The consecutive image frames are sorted according to the acquisition time to obtain the door opening process image sequence.

[0082] In practical implementation, upon detecting an unlocking trigger event, the corresponding trigger time is first obtained. This trigger event can originate from the unlocking verification result, the lock body state change result, or the door opening state change result. For example, when fingerprint verification, facial recognition verification, or password verification is successful, the moment of successful verification can be determined as the trigger time; when the bolt retracts or the unlocking motor completes the unlocking action, the moment the bolt retraction signal is generated or the unlocking action is completed can be determined as the trigger time; when the door handle is pressed, rotated, or the door magnetic state changes, the moment the corresponding action signal or state change signal is generated can be determined as the trigger time. It should be noted that the trigger time should be recorded using the same time base as the acquisition time of each image frame, such as using a local clock timestamp or a synchronized standard timestamp.

[0083] Furthermore, the image acquisition start time is determined based on the trigger time. As one optional implementation, the image acquisition start time can be set to the trigger time, meaning image frame acquisition begins immediately after the unlocking trigger event is detected. Alternatively, the image acquisition start time can be set to a preset delay after the trigger time, such as 100 to 300 milliseconds after the trigger time. Regardless of the method used, the image acquisition start time is determined based on the trigger time to ensure that the acquired image frames cover the door opening process after the unlocking trigger.

[0084] After determining the image acquisition start time, continuous image frames are acquired during the door opening process at a preset frame rate within a preset acquisition duration. The preset acquisition duration can be determined based on the duration of the door opening action and the window of occurrence of risky behaviors. For example, for a typical entrance door, the time from unlocking to entering the door by an authorized user can be set to 5 to 8 seconds. The preset frame rate can be determined based on the temporal resolution required for subsequent target tracking, for example, set to 10 to 25 frames per second. The acquisition time corresponding to each acquired image frame is recorded. The acquisition time can be the start time of image exposure, the end time of image exposure, or the time of image frame generation, as long as a consistent rule is used throughout the implementation process.

[0085] After acquisition is completed within the preset acquisition time, multiple consecutive image frames are obtained. Since the order of image acquisition, buffering, reading, or processing may change, it is necessary to sort them according to the acquisition time corresponding to each image frame to obtain the door opening process image sequence. Specifically, the acquired consecutive image frames can be represented as a set of several image frames and their corresponding acquisition times, for example... ,in, Indicates the first Image frames, Indicates the first The acquisition time of each image frame This indicates the number of image frames acquired. The image frames are sorted from earliest to latest according to their acquisition time to obtain the image sequence of the door opening process. ,in, The corresponding collection time is no later than The corresponding collection time, The corresponding collection time is no later than The corresponding collection time follows the same pattern.

[0086] After obtaining the image sequence of the door opening process, brightness equalization, distortion correction, and sharpness evaluation are performed on each image frame in the sequence to obtain the image quality value of each image frame. For brightness equalization, the average brightness value of each image frame is first determined. If the image frame is an RGB image, it can be converted to a grayscale image, or converted to a color space that includes a brightness channel, such as YUV, HSV, or Lab, and the brightness channel can be extracted. The average brightness value can be obtained by averaging the brightness values ​​of all pixels in the brightness channel. If the brightness channel is represented by 8 bits, the brightness value range is 0 to 255. The preset low brightness threshold can be set to 50, 60, or 70, preferably 60; the preset high brightness threshold can be set to 200 or 210, preferably 200. When the average brightness value of an image frame is lower than the preset low brightness threshold, the image frame is considered to be too dark overall, and the brightness of the image frame can be enhanced; when the average brightness value is higher than the preset high brightness threshold, the image frame is considered to be at risk of being too bright or overexposed, and the brightness of the image frame can be suppressed; when the average brightness value is between the preset low brightness threshold and the preset high brightness threshold, the overall brightness of the image frame is considered to be within a suitable range for recognition, and no brightness adjustment or only slight local equalization can be performed.

[0087] Brightness equalization can be achieved through histogram equalization, adaptive histogram equalization, contrast-limited adaptive histogram equalization, or gamma correction. For low-light scenes in corridors, contrast-limited adaptive histogram equalization is preferred to enhance the local contrast of door frames, door gaps, thresholds, and human silhouettes, while avoiding excessive noise amplification. For image frames that are generally dark, gamma correction to enhance the brightness of dark areas can be used; for image frames that are generally too bright, brightness adjustment to suppress bright areas can be used. If the image frame is a color image, it is preferable to perform equalization only on the luminance channel before combining it with the original chroma channel to reduce color distortion.

[0088] For distortion correction, geometric correction is performed on the brightness-processed image frames based on pre-obtained camera imaging parameters. Camera imaging parameters may include horizontal focal length, vertical focal length, principal point x-coordinate, principal point y-coordinate, radial distortion parameters, and tangential distortion parameters. These parameters can be obtained through factory calibration or during installation and debugging using a checkerboard or dot calibration board. For example, for a 1280×720 resolution camera, the principal point x-coordinate is typically close to half the image width, and the principal point y-coordinate is typically close to half the image height; that is, the principal point x-coordinate can be close to 640, and the principal point y-coordinate can be close to 360. The horizontal and vertical focal lengths are obtained by calibrating the lens focal length and pixel size.

[0089] During distortion correction, for any pixel in the image, the horizontal pixel coordinates are subtracted from the principal point's horizontal coordinates, and then divided by the horizontal focal length to obtain the horizontal normalized coordinates. Similarly, the vertical pixel coordinates are subtracted from the principal point's vertical coordinates, and then divided by the vertical focal length to obtain the vertical normalized coordinates. This process transforms the pixel from the image pixel coordinate system to a normalized coordinate system centered on the camera's optical axis and measured by the focal length.

[0090] Subsequently, the mapping relationship between the pixel in the distorted image and the corrected image is determined based on the radial distortion model and the tangential distortion model. The radial distortion model describes the barrel or pincushion distortion of a pixel in lens imaging as its distance from the principal point increases. Specifically, the normalized distance from the pixel to the principal point is first determined based on the horizontal and vertical normalized coordinates, and then the horizontal and vertical normalized coordinates are radially scaled and corrected using one or more radial distortion parameters. The tangential distortion model describes the horizontal and vertical offsets caused when the lens is not perfectly centered or the lens plane is not perfectly parallel to the imaging plane. Specifically, the horizontal and vertical offsets of the pixel are determined based on the horizontal and vertical normalized coordinates and the tangential distortion parameters, and this offset is used to correct the pixel's position. After combining the radial and tangential distortion corrections, the corrected coordinate position is obtained.

[0091] For image sharpness evaluation, it can be performed on image frames after brightness equalization and distortion correction. As an optional implementation, the Laplacian operator can be used to evaluate image sharpness. Specifically, the image frame is converted into a grayscale image, and the Laplacian operator is applied to the grayscale image to obtain a response result reflecting changes in image edges and textures. Then, the dispersion of this response result is statistically analyzed; the greater the dispersion, the sharper the edges and textures in the image, and the higher the sharpness.

[0092] After obtaining the brightness equalization result, distortion correction result, and sharpness evaluation result, the image quality value is further determined. The image quality value can be jointly determined by the brightness evaluation value, distortion correction evaluation value, and sharpness evaluation value. The brightness evaluation value is used to indicate whether the image brightness is within a suitable range for recognition. As an optional implementation, 128 can be used as the ideal brightness value for an 8-bit brightness image, and the allowable brightness deviation can be set to 100. The closer the average brightness value of the image frame is to 128, the higher the brightness evaluation value; the farther the average brightness value deviates from 128, the lower the brightness evaluation value. For example, when the average brightness value is 128, the brightness evaluation value can be 1; when the average brightness value is 60, the brightness evaluation value can be approximately 0.32; when the average brightness value is 210, the brightness evaluation value can be approximately 0.18; if a negative value is obtained based on the deviation, the brightness evaluation value is set to 0.

[0093] The distortion correction evaluation value indicates whether the effective area of ​​the image after distortion correction is sufficient. As an optional implementation, the distortion correction evaluation value can be determined based on the proportion of the effective pixel area after distortion correction. During distortion correction, if a pixel in the corrected image can find a corresponding effective sampling position in the original image, then that pixel is a valid pixel; if the corresponding sampling position exceeds the boundary of the original image, then that pixel is an invalid pixel. The proportion of the number of valid pixels to the total number of pixels in the corrected image is used as the distortion correction evaluation value. For example, if the proportion of the effective pixel area after distortion correction is 93%, then the distortion correction evaluation value can be 0.93; if the proportion of the effective pixel area is only 72% due to large wide-angle distortion, then the distortion correction evaluation value can be 0.72.

[0094] The sharpness rating is used to indicate whether the edges and textures of an image frame are sharp. As an optional implementation, the Laplacian variance can be used as the base value for sharpness, and a sharpness normalization upper limit can be set. For example, at a resolution of 1280×720, the sharpness normalization upper limit can be set to 800 or 1000. When the Laplacian variance reaches or exceeds this normalization upper limit, the sharpness rating is 1; when the Laplacian variance is lower than this normalization upper limit, the sharpness rating is determined according to its ratio to the normalization upper limit. For example, if the normalization upper limit is set to 800, and the Laplacian variance of an image frame is 600, then the sharpness rating can be 0.75; if the Laplacian variance is 300, then the sharpness rating can be 0.375.

[0095] Image quality values ​​can be determined using a weighted method. As a preferred implementation, the weight of the brightness evaluation value can be set to 0.2, the weight of the distortion correction evaluation value can be set to 0.2, and the weight of the sharpness evaluation value can be set to 0.6. The weighted result of the three values ​​can then be used as the image quality value.

[0096] After obtaining the image quality value for each image frame, the image quality value is compared with a preset quality threshold. The preset quality threshold can be set according to the calculation method of the image quality value, the sensitivity of the subsequent recognition algorithm to image quality, and the actual installation environment. When a weighted method is used to determine the image quality value and the image quality value ranges from 0 to 1, the preset quality threshold can be set to 0.45, 0.50, 0.55, or 0.60.

[0097] If the image quality value of a certain image frame is less than a preset quality threshold, the image frame is considered to have problems such as insufficient brightness, overexposure, insufficient effective area after distortion correction, motion blur, or defocus, and is not suitable as input for subsequent image recognition, so it is removed from the door opening process image sequence. If the image quality value of a certain image frame is greater than or equal to the preset quality threshold, the image frame is retained. After removing low-quality image frames, the remaining image frames are arranged in the original acquisition time order to form the image frame sequence to be recognized.

[0098] To avoid excessively long time spans or insufficient time sampling points in the image frame sequence to be identified due to the removal of too many low-quality image frames, a preset minimum frame count can be set. The preset minimum frame count can be determined based on the preset acquisition duration and preset frame rate. Specifically, first determine the theoretical number of frames to be acquired based on the preset acquisition duration and preset frame rate. For example, if the preset acquisition duration is 6 seconds and the preset frame rate is 15 frames / second, the theoretical number of frames to be acquired is 90; if the preset acquisition duration is 5 seconds and the preset frame rate is 10 frames / second, the theoretical number of frames to be acquired is 50. The preset minimum frame count can be set to 40% to 70% of the theoretical number of frames to be acquired, preferably 50%, and no less than 10 frames.

[0099] After removing image frames with image quality values ​​lower than a preset quality threshold, if the number of remaining image frames is less than a preset minimum number of frames, several image frames are added from the removed image frames according to their image quality values ​​from high to low, until the number of image frames in the image frame sequence to be identified reaches the preset minimum number of frames. During the addition process, the original acquisition time of each image frame is retained, and the frames are reordered according to their acquisition time after the addition is completed.

[0100] After obtaining the image frame sequence to be identified, the image frame sequence is used as the image sequence for image recognition. Specifically, in subsequent steps, door motion reference features such as door frame edges, door seams, threshold lines, ground textures, or wall fixed points can be extracted based on the image frame sequence to be identified. Human target detection, target association, and motion trajectory generation can also be performed based on the image frame sequence to be identified.

[0101] Regarding S102 above:

[0102] As an optional implementation, image recognition is performed on the door opening process image sequence to extract door movement reference features, including:

[0103] Edge detection is performed on the image frames in the image frame sequence to be identified to obtain a set of edge pixels;

[0104] Line fitting is performed on the edge pixel set to obtain candidate line features;

[0105] Based on the direction, length, and image position of the candidate line features, determine the door frame edge, door seam line, or threshold line from the candidate line features;

[0106] Corner detection and texture detection are performed on the image frames in the image frame sequence to be identified to obtain candidate fixed feature points;

[0107] Based on the matching relationship between the candidate fixed feature points in adjacent image frames, the ground texture or wall fixed points are determined.

[0108] In practice, the first step is to perform grayscale processing and smoothing on any current image frame in the image frame sequence to be identified. For example, Gaussian filtering can be used, with the Gaussian filter kernel size set to 3×3 or 5×5 and the standard deviation set to 1.0 to 1.5.

[0109] Subsequently, edge detection is performed on the smoothed image frame to obtain a set of edge pixels. As an optional implementation, Canny edge detection can be used. The low and high thresholds for Canny edge detection can be adaptively set based on the median grayscale value of the image, or a fixed threshold can be used. For an 8-bit grayscale image, the low threshold can be set to 50, and the high threshold can be set to 150. After edge detection, a set of edge pixels consisting of multiple edge pixels is obtained. To remove isolated noise points, morphological opening operations or connected component filtering can be performed on the edge map to remove edge regions with a pixel count less than a preset connected component area threshold. The preset connected component area threshold can be set to 10 to 30 pixels, for example, 20 pixels.

[0110] Next, a straight line fit is performed on the edge pixel set to obtain candidate straight line features. As an optional implementation, a probabilistic Hough line transform can be used. Specifically, edge pixels supported by the same straight line are searched in the edge map. When the number of votes in the accumulator reaches a preset voting threshold, a candidate line segment is output. For example, for an image with a resolution of 1280×720, the Hough line voting threshold can be set to 40 to 80, preferably 60; the minimum line segment length can be set to 5% to 10% of the image height or width, for example, 60 pixels; the maximum allowed line segment break interval can be set to 10 to 30 pixels, for example, 20 pixels.

[0111] Each candidate line feature includes at least the pixel coordinates of the two endpoints of the line segment, the coordinates of the center point of the line segment, the line segment length, the orientation angle, the number of supporting edge pixels, and the fitting residual. The line segment length is used to determine whether the candidate line has a sufficiently stable structure; the orientation angle is used to determine whether the candidate line is closer to the vertical or horizontal direction; the center point coordinates and endpoint coordinates are used to determine the position of the candidate line in the image; and the number of supporting edge pixels and the fitting residual are used to determine whether the candidate line is reliable.

[0112] Then, based on the direction, length, and image position of the candidate line features, the door frame edge, door seam line, or threshold line are determined from the candidate line features. Specifically, the candidate line features can be first divided into vertical candidate lines and horizontal candidate lines. If the direction angle of the candidate line relative to the horizontal axis of the image is close to 90 degrees, for example, the direction angle is between 75 degrees and 105 degrees, then it is determined as a vertical candidate line; if the direction angle of the candidate line relative to the horizontal axis of the image is close to 0 degrees or 180 degrees, for example, the direction angle is between 0 degrees and 15 degrees or between 165 degrees and 180 degrees, then it is determined as a horizontal candidate line.

[0113] To determine the door frame edge, candidate lines can be selected from vertical candidate lines that are relatively long, located close to the left or right side of the image or the boundary of the doorway area, and can appear stably in adjacent frames. Specifically, in the image width direction, the left 5% to 40% width range and the right 60% to 95% width range can be used as the priority search area for the door frame edge. The length of the candidate line corresponding to the door frame edge is preferably not less than 30% of the image height, and more preferably not less than 40% of the image height. For example, in a 1280×720 image, the length of the candidate line segment for the door frame edge can preferably be greater than 250 pixels. If there are multiple candidate vertical lines in the same area, the candidate line with the longest length, the most supporting edge pixels, the smallest fitting residual, and the direction closer to the vertical direction can be preferentially selected as the door frame edge.

[0114] To determine the door seam line, candidate lines can be selected from vertical candidate lines. These lines should be located at the junction of the door and the door frame, be relatively narrow, have a significant difference in brightness on both sides, and exhibit continuous positional changes as the door opens. A door seam line typically appears in an image as a thin, elongated line segment with a clear contrast between light and dark areas. Its direction is close to vertical, and its length can be less than the edge of the door frame, but it usually still has a certain degree of continuity. Specifically, candidate lines can be searched near the edge of the determined door frame or in the middle of the image near the direction the door opens. The length of these candidate lines can be set to be no less than 20% of the image height, for example, no less than 140 pixels in a 720-pixel high image. To distinguish the door seam line from the edge of a human body, the average brightness difference between a preset width region on both sides of the candidate line can be calculated. For example, a strip region with a width of 3 to 8 pixels can be taken on each side of the line segment. If the average brightness difference on both sides is greater than a preset brightness difference threshold, the candidate line is considered more likely to be a door seam line. For 8-bit grayscale images, the preset brightness difference threshold can be set to 20 to 40, preferably 30.

[0115] To determine the threshold line, a candidate line located in the lower part of the image, with a relatively long length, a near-horizontal orientation, and aligned with the bottom of the doorway region can be selected from the horizontal candidate lines. Specifically, the lower 40% of the image height can be considered as the priority search area for the threshold line, for example, an area with a vertical orientation greater than 60% of the image height. The length of the candidate line segment corresponding to the threshold line is preferably no less than 25% of the image width, and more preferably no less than 30% of the image width. For example, in a 1280×720 image, the length of the candidate line segment for the threshold line can preferably be greater than 350 pixels. If multiple near-horizontal line segments exist in the same area, the candidate line with the longer length, more supporting edge pixels, smaller fitting residual, and smooth positional change in adjacent image frames can be preferentially selected as the threshold line.

[0116] Furthermore, corner detection and texture detection are performed on the image frames in the image frame sequence to be identified to obtain candidate fixed feature points. Specifically, for the current image frame, corner detection can be performed on the grayscale image or the luminance channel image. As an optional implementation, Harris corner detection can be used, and pixels with corner response values ​​greater than a preset corner response threshold can be used as initial corner points. To avoid the candidate fixed feature points being too dense, a minimum spacing between feature points can be set, for example, 5 to 15 pixels. The maximum number of candidate fixed feature points retained per frame can be set, for example, 200 to 1000, preferably around 500.

[0117] For texture detection, regions with obvious local textures can be found in the image frame, such as tile seams, ground particles, wall spots, door frame fixing screws, or wall markings. As an optional implementation, local binary mode features, gray-level co-occurrence matrix features, Gabor filter response, or local gradient change intensity can be used to evaluate texture saliency. Specifically, the image can be divided into multiple small regions, such as 16×16 pixels or 32×32 pixels each. The gradient change intensity or texture response value of each region is calculated, and regions with texture response values ​​greater than a preset texture threshold are selected as texture candidate regions. Corner points or local feature points are retained within these texture candidate regions. The preset texture threshold can be adaptively set according to the texture response distribution of the entire image frame; for example, the top 30% of all region texture response values ​​can be selected as texture candidate regions, or a fixed threshold can be set.

[0118] After obtaining candidate fixed feature points, the ground texture or wall fixed points are determined based on the matching relationship between the candidate fixed feature points in adjacent image frames. Specifically, feature descriptors are calculated for the candidate fixed feature points in the current image frame and adjacent image frames. The feature descriptors can be ORB descriptors, BRIEF descriptors, SIFT-like scale feature descriptors, or other descriptors that can characterize the local image structure. If a binary descriptor, such as the ORB descriptor, is used, the Hamming distance can be used to measure the similarity between two candidate fixed feature points; if a floating-point descriptor is used, the Euclidean distance can be used to measure the similarity. For each candidate fixed feature point, the nearest candidate fixed feature point in adjacent image frames is searched as the first candidate matching point, and the second nearest candidate fixed feature point is searched as the second candidate matching point. If the distance between the first and second candidate matching points is significantly smaller than the distance between the first and second candidate matching points, for example, the ratio of the first to second candidate matching points is less than 0.75 or 0.8, the matching relationship is considered reliable.

[0119] Optionally, geometric consistency verification is performed on the feature point pairs obtained from the initial matching. Specifically, a random sampling consensus algorithm can be used to estimate the planar transformation relationship or the basic affine transformation relationship between adjacent image frames, and matching point pairs that are inconsistent with the overall geometric relationship are eliminated. If the position error of a matching point pair after transformation prediction is less than a preset reprojection error threshold, it is retained as an inlier; if the error is greater than the preset reprojection error threshold, it is eliminated as an outlier. The preset reprojection error threshold can be set to 2 to 5 pixels. After geometric consistency verification, the retained candidate fixed feature points have a higher probability of belonging to ground texture or wall fixed points.

[0120] When determining ground texture or wall fixed points, image location and frame stability can be considered. For candidate fixed feature points located in the lower part of the image, near the threshold line, or below the waiting area outside a door, if they can stably match across multiple consecutive frames and exhibit significant local texture response, they can be identified as ground texture feature points. Specifically, the area located in the lower 40% to 50% of the image height can be considered the priority area for ground textures; if a threshold line has been detected, the area below or near the threshold line within a preset range can be considered the priority area for ground textures. For candidate fixed feature points located in the left, right, or upper background area of ​​the image, and which can stably match across multiple adjacent frames, they can be identified as wall fixed points. Specifically, the left and right edges of the image, the outer area of ​​the door frame, or the area above the door frame can be considered the priority areas for wall fixed points. If a candidate fixed feature point has a reliable matching relationship in at least three consecutive frames, and the feature descriptor similarity meets the threshold requirement and the geometric consistency verification shows it as an inlier, then it is identified as a ground texture or wall fixed point.

[0121] After completing the above processing, the motion reference features of the door in the current image frame and adjacent image frames are obtained. For linear reference features, their feature type, corresponding image frame, line segment endpoint coordinates, center point coordinates, orientation angle, and length can be recorded; for point reference features, their feature type, corresponding image frame, pixel coordinates, and feature descriptor can be recorded. Feature types can be door frame edges, door seam lines, threshold lines, ground textures, or fixed points on the wall.

[0122] Regarding the above S103:

[0123] As an optional implementation, based on the positional changes of the door motion reference features in different image frames, the viewpoint change parameters of each image frame relative to the reference frame are determined, including:

[0124] Image frames that are selected from the image frame sequence to be identified before the door is opened or when the door opening angle is less than a preset angle are used as reference frames.

[0125] Determine the reference features in the reference frame, and determine the current reference features in the current image frame;

[0126] The reference feature is matched with the current reference feature to obtain a reference feature matching pair;

[0127] Based on the coordinate differences between the reference feature matching pair in the reference frame and the current image frame, the viewpoint change parameters of the current image frame relative to the reference frame are determined.

[0128] In specific implementation, firstly, image frames before the door opens or with an opening angle less than a preset angle are selected from the image frame sequence to be identified obtained in step S101 as reference frames. As an optional implementation, the image frame with the earliest acquisition time in the image frame sequence to be identified can be selected as the reference frame. This method is suitable for scenarios where acquisition occurs immediately after unlocking and the first frame or the first few frames typically correspond to a situation where the door has not yet clearly opened. As another optional implementation, the image frame with the smallest door opening angle, the most reference features, and the highest image quality value can be selected from the first few frames of the image frame sequence to be identified as the reference frame. For example, selection can be made from the first 5 to 10 frames with the earliest acquisition time, prioritizing image frames that detect the door frame edge, door seam line, threshold line, and have a large number of fixed feature points. As yet another optional implementation, the door opening angle is estimated based on the door magnetic signal, door angle sensing results, or changes in door seam width, and image frames with door opening angles less than a preset angle are selected as reference frames. The preset angle can be set to 3 to 10 degrees, preferably 5 degrees.

[0129] When estimating the door opening angle, the change in distance between the door seam and the door frame edge can be used for judgment. Specifically, when the door is not open or the opening angle is small, the relative distance between the door seam line and the door frame edge is usually small, and the door seam area is narrow. As the door opens, the position of the door seam line relative to the door frame edge changes significantly, and the door seam area widens. The average pixel distance between the door seam line and the adjacent door frame edge can be used as a representation value of the door seam width. When this door seam width representation value is less than a preset door seam width threshold, the door opening angle is considered to be less than the preset angle. For example, for a 1280×720 resolution image, the preset door seam width threshold can be set to 1% to 3% of the image width, that is, approximately 13 to 38 pixels; in a typical entrance door scenario, it can be set to 25 pixels.

[0130] After determining the reference frame, the reference reference features in the reference frame are determined. The reference reference features can be derived from the door motion reference features extracted in step S102. For the current image frame whose viewpoint change parameters need to be determined, its current reference features are determined. The current reference features also originate from the door motion reference features extracted from the current image frame in step S102. The type of the current reference feature should correspond to the reference reference feature. For example, the door frame edge in the reference frame should preferentially match the door frame edge in the current image frame; the threshold line in the reference frame should preferentially match the threshold line in the current image frame; and the ground texture points or wall fixed points in the reference frame should preferentially match fixed feature points in the current image frame whose feature descriptors are similar and whose position changes conform to the overall viewpoint change trend.

[0131] Subsequently, the baseline reference feature and the current reference feature are matched to obtain reference feature matching pairs. For linear reference features, matching relationships can be established based on feature type, orientation angle difference, length difference, and position difference. Specifically, when a door frame edge in the baseline frame and a candidate door frame edge in the current image frame are of the same type, their orientation angle difference is less than a preset orientation difference threshold, their length ratio is within a preset range, and the change in the center point position does not exceed a preset position change range, they can be considered as a line feature matching pair. The preset orientation difference threshold can be set to 5 to 15 degrees, preferably 10 degrees; the length ratio can be set to 0.5 to 1.8, preferably 0.7 to 1.5; the center point position change range can be set according to the image size, for example, within 30% of the image width or height. For door seam lines, since the opening of the door will cause a large change in the position of the door seam line, the position change range can be appropriately relaxed, but its direction is still required to be close to vertical and the brightness difference on both sides of the line segment must meet the door seam line characteristics. For threshold lines, their direction can be required to be close to horizontal or conform to the perspective tilt trend, and located in the lower region of the image.

[0132] For point-based reference features, matching relationships can be established based on feature descriptor similarity, feature point position changes, and geometric consistency. If the reference feature is a ground texture point or a fixed wall point, the candidate point with the most similar descriptor is searched in the current reference feature of the current image frame. If binary descriptors such as ORB are used, Hamming distance can be used for similarity judgment; if floating-point descriptors are used, Euclidean distance can be used for similarity judgment. Subsequently, geometric consistency verification can be used to eliminate incorrect matches. For example, the preliminary planar transformation of the current image frame relative to the reference frame can be estimated from the preliminary matching point pairs using the random sampling consistency method. If the predicted position error of a matching point pair under this transformation is less than a preset error threshold, it is retained as a reference feature matching pair; if the error is greater than the preset error threshold, it is eliminated. The preset error threshold can be set to 2 to 5 pixels, preferably 3 pixels.

[0133] After obtaining the reference feature matching pairs, the viewpoint change parameters of the current image frame relative to the reference frame are determined based on the coordinate differences between the reference feature matching pairs in the reference frame and the current image frame. The viewpoint change parameters include translation parameters, rotation parameters, scale parameters, and perspective parameters. Specifically, different parameter calculation methods can be selected based on the number and type of available reference features in the current image frame. When there are few reliable matching features but at least one stable matching point or a stable line segment exists, the translation parameter can be determined; when there are linear matching features such as door frame edges, door seam lines, or threshold lines, the rotation parameter can be determined; when there are at least two reference feature matching pairs, the scale parameter can be determined; and when there are at least four spatially non-collinear reference feature matching pairs, the perspective parameter can be determined.

[0134] For translation parameters, they can be determined based on the pixel coordinate differences of the same door motion reference features in the reference frame and the current image frame. If the reference feature matching pair is a point feature matching pair, the horizontal and vertical differences of the pixel coordinates of the feature point in the current image frame relative to its pixel coordinates in the reference frame are calculated, and these horizontal and vertical differences are used as a translation candidate value. If multiple point feature matching pairs exist, the average or median value of the multiple translation candidate values ​​can be calculated as the translation parameter of the current image frame relative to the reference frame. It is preferable to use the median value or the average value after removing outliers to reduce the impact of mismatched points. If the reference feature matching pair is a line feature matching pair, the coordinate difference of the center point of the line segment in the current image frame and the reference frame can be calculated as the translation candidate value corresponding to the line segment; alternatively, the coordinate differences of the two endpoints of the line segment can be calculated separately, and the average result can be used as the translation candidate value corresponding to the line segment.

[0135] Rotation parameters can be determined based on the direction difference between the door frame edge, door seam line, or threshold line in the reference frame and the current image frame. Specifically, for a linear reference feature matching pair, the difference between its direction angle in the reference frame and its direction angle in the current image frame is calculated, and this difference is used as a rotation candidate value. If multiple linear reference feature matching pairs exist, such as the left door frame edge, right door frame edge, and threshold line simultaneously, the direction difference of each line segment is calculated separately, and weights are determined based on the line segment length, fitting residual, and continuous frame stability. A weighted average of the multiple rotation candidate values ​​is then applied. For vertical line segments such as door frame edges and door seam lines, their direction change relative to the vertical direction can be used to determine rotation candidate values; for horizontal line segments such as threshold lines, their direction change relative to the horizontal direction can be used. If both vertical and horizontal line segments are available, the rotation parameters can be determined by combining both. To avoid abrupt changes in rotation parameters caused by abnormal line segments, line segments whose direction difference from the majority of line segments exceeds a preset angle threshold can be discarded. The preset angle threshold can be set to 10 to 20 degrees.

[0136] The scale parameter can be determined based on the distance change between at least two reference feature matching pairs. Specifically, in the reference frame, reference coordinate points corresponding to two reference feature matching pairs are selected, and the pixel distance between them is calculated; in the current image frame, the pixel distance between two corresponding current coordinate points is calculated; the change ratio of the distance in the current image frame relative to the distance in the reference frame is used as the scale candidate value. If multiple reference feature matching pairs exist, multiple sets of point pairs or line segment endpoint pairs can be selected, and scale candidate values ​​can be calculated for each pair. Then, the median, average, or weighted average value is taken as the scale parameter.

[0137] Perspective parameters can be determined based on the pixel coordinates of at least four reference feature matching pairs. Specifically, when there are at least four non-collinear and spatially dispersed point feature matching pairs between the reference frame and the current image frame, the planar perspective transformation relationship of the current image frame relative to the reference frame can be estimated based on these matching point pairs. These at least four reference feature matching pairs can come from ground texture points, wall fixed points, or endpoints or intersections of linear features. For example, the intersection of the door frame edge and the threshold line, the endpoints of the door frame edge, the threshold line endpoints, stable wall fixed points, or ground texture points can all be used as coordinate points for estimating perspective parameters. When estimating perspective parameters, a direct linear transformation method can be used, combined with a random sampling consistency method to eliminate erroneous matching points. If more than four matching points are available, it is preferable to use all interior points for least squares optimization to obtain more stable perspective parameters. To ensure reliable perspective estimation, the selected matching points should not be concentrated in a small area, nor should they all be located on the same straight line; it is possible to require the matching points to cover at least two different areas in the image, such as the ground area at the bottom of the image and the side wall area of ​​the image, or the area near the edge of the door frame and the area near the threshold line.

[0138] If there are fewer than four matching points, or if the four matching points are approximately collinear, making it impossible to stably estimate the perspective transformation, then affine or similarity transformations can be used to determine the viewpoint change parameters. Specifically, when there are at least three non-collinear matching points, affine parameters including translation, rotation, scaling, and shearing can be determined; when there are only two or more matching points and at least one linear directional feature, similarity transformation parameters including translation, rotation, and scaling can be determined; when the number of reliable reference features is even smaller, only translation and rotation parameters need to be determined.

[0139] For each current image frame in the image frame sequence to be identified, its viewpoint change parameters relative to the reference frame are determined in the manner described above. If the current image frame itself is the reference frame, its translation parameter can be set to 0, its rotation parameter can be set to 0, its scale parameter can be set to 1, and its perspective parameter can be set to unit perspective. For each image frame following the reference frame, its translation, rotation, scale, and perspective parameters relative to the reference frame are determined respectively. Finally, the set of viewpoint change parameters corresponding to each image frame in the image frame sequence to be identified can be obtained.

[0140] Regarding S104 above:

[0141] As an optional implementation, based on the aforementioned viewpoint change parameters, the target position in each image frame is mapped to a unified doorway security area coordinate system to obtain registered target position data, including:

[0142] The door frame edge, door seam line, or threshold line in the reference frame are used as coordinate references to establish the coordinate system of the doorway safety area;

[0143] Mark the door gap area, threshold area, and waiting area outside the door in the coordinate system of the doorway safety area;

[0144] Based on the perspective change parameters corresponding to the current image frame, determine the coordinate transformation relationship from the current image frame to the coordinate system of the doorway safety area;

[0145] The target location is obtained as the center point of the target detection box, the target foot point, or the target contour point in the current image frame.

[0146] The target position is mapped according to the coordinate transformation relationship to obtain the registered target position data.

[0147] In practical implementation, the door frame edge, door seam line, or threshold line in the reference frame are first used as coordinate references to establish a coordinate system for the doorway safety area. Specifically, when establishing the doorway safety area coordinate system, the threshold line in the reference frame can be used as the horizontal coordinate reference, and the door frame edge or door seam line can be used as the vertical coordinate reference. If both the door frame edge and threshold line are detected in the reference frame, the intersection of the door frame edge and the threshold line can be used as the origin, the extension direction of the threshold line can be used as the horizontal axis, and the extension direction of the door frame edge or the direction approximately perpendicular to the threshold line can be used as the vertical axis.

[0148] To standardize coordinate scales, the pixel distance in the reference frame can be used as the coordinate unit of the doorway safety area coordinate system, or the pixel scale can be converted into a spatial scale based on the actual width of the doorway or the actual width of the threshold.

[0149] After establishing the coordinate system for the doorway safety zone, the door gap area, threshold area, and waiting area outside the door are marked within it. The door gap area can be determined based on the door gap line and the edge of the door frame in the reference frame. Specifically, when a door gap line is detected in the reference frame, a first preset width can be extended to both sides along a direction perpendicular to the door gap line, using the door gap line as the center line, to form a door gap strip area. The first preset width can be set according to the image resolution and the needs of door gap risk assessment. For example, for a 1280×720 image, it can be set to 20 to 60 pixels at the reference frame pixel scale, preferably 40 pixels; if using the actual spatial scale, it can be set to 5 to 15 centimeters, preferably 10 centimeters.

[0150] The threshold area can be determined based on the threshold line in the reference frame. Specifically, with the threshold line as the center, a second preset width is extended along the normal direction of the threshold line towards both the inside and outside of the door, forming a horizontal strip area. The second preset width can be set according to the actual width of the threshold or the image scale. For example, at the pixel scale, the threshold area width can be set to 30 to 80 pixels, preferably 50 pixels; at the actual spatial scale, the threshold area width can be set to 10 to 25 centimeters, preferably 15 centimeters.

[0151] The waiting area outside the door can be determined based on the threshold area and the direction outside the doorway. Specifically, in the doorway safety area coordinate system, the area within a preset distance outside the threshold line can be marked as the waiting area outside the door. The waiting area outside the door is used to represent the area where normal personnel wait for the door to open, and where visitors linger or pass through. If using a pixel scale, the waiting area outside the door can be set to an area extending 150 to 400 pixels from the threshold line outwards, preferably 300 pixels; if using an actual spatial scale, the waiting area outside the door can be set to an area extending 0.5 to 1.5 meters from the threshold line outwards, preferably 1 meter.

[0152] For each current image frame, based on the viewpoint change parameters corresponding to the current image frame, the coordinate transformation relationship from the current image frame to the doorway safety area coordinate system is determined. Based on the perspective parameters from step S103, these perspective parameters are used as the primary mapping relationship between the current image frame and the reference frame. Combined with the coordinate definition from the reference frame to the doorway safety area coordinate system, the coordinate transformation relationship from the current image frame to the doorway safety area coordinate system is obtained. Specifically, based on the planar perspective transformation relationship of the current image frame relative to the reference frame in step S103, the target position of the current image frame is mapped to the reference frame coordinates, and further expressed as doorway safety area coordinates.

[0153] The target location is the center point of the target detection box, the target's foot point, or the target's contour point in the current image frame, and can be obtained based on the human target detection results. If a human target detection box exists in the current image frame, the center point of the target detection box can be used as the target location. Specifically, the horizontal midpoint between the top-left and bottom-right corner coordinates of the target detection box is used as the x-coordinate of the center point, and the vertical midpoint between the top-left and bottom-right corner coordinates of the target detection box is used as the y-coordinate of the center point.

[0154] If the target foot position can be determined, it is preferable to obtain the target foot point as the target location. The target foot point can be obtained through human keypoint detection or approximated by the center point of the bottom edge of the human detection box. Specifically, if the left ankle point, right ankle point, left toe point, or right toe point is identified through human keypoint detection, the average position of the keypoints of the left and right feet can be used as the target foot point; if only the target detection box is available, the center point of the bottom edge of the target detection box can be used as the target foot point, that is, the horizontal position is taken as the midpoint of the left and right boundaries of the detection box, and the vertical position is taken as the bottom boundary of the detection box.

[0155] When it's necessary to determine whether a target occupies the door gap area or overlaps with the threshold area, the target's contour points can be obtained as its location. These contour points can be obtained from human segmentation results, foreground extraction results within the target detection bounding box, or human boundary estimation. Specifically, the outer contour point set of the human target can be extracted, or the bottom contour point, the side contour point near the door gap, and the center point of the contour can be selected from the target contour. To calculate the shortest distance between the target and the door gap area, the point closest to the door gap area in the target contour point set can be used; to determine whether the target has crossed the threshold area, the bottom contour point or the contour point near the feet can be used.

[0156] After obtaining the target position in the current image frame, the target position is mapped according to the coordinate transformation relationship to obtain the registered target position data. If the target position is a single point, such as the center point of the target detection box or the foot point of the target, the coordinates of this point are input into the coordinate transformation relationship to obtain its coordinates in the doorway safety area coordinate system. If the target position is multiple points, such as a set of target contour points, the coordinates of each contour point are transformed separately to obtain the registered contour point set; alternatively, after completing the contour point transformation, the center point, bottom point, point closest to the door seam area, or circumscribed frame of the registered contour can be calculated. If the target position is a detection box, the four corner points of the detection box can also be transformed separately, and the registered detection box area can be determined based on the transformed four corner points.

[0157] The registered target location data can include target identifier, image frame identifier, acquisition time, original target location, coordinate transformation relationship identifier, registered coordinates, and registered region type. Specifically, the target identifier indicates the detected object in the current image frame; the image frame identifier and acquisition time are used to generate the motion trajectory in chronological order; the original target location records the pixel position of the target in the current image frame; the registered coordinates indicate the target's position in the doorway security area coordinate system; and the registered region type indicates whether the target is currently located in the door gap area, threshold area, waiting area outside the door, or other areas.

[0158] Regarding the above S105:

[0159] As an optional implementation, human target detection and target association are performed on the door opening process image sequence, and combined with the registered target position data, motion trajectories of authorized user targets and unauthorized targets are generated in the coordinate system of the doorway security area, including:

[0160] Human detection is performed on each image frame in the door opening process image sequence to obtain human target detection results, which include target identifier, target detection box and target appearance features;

[0161] Based on the target appearance features of human target detection results in adjacent image frames and the registered target location data, human targets are associated to obtain target association results.

[0162] Based on the authentication information corresponding to the unlocking trigger event, the authorized user target is determined from the target association result, and human targets that are different from the authorized user target are determined as unauthorized targets;

[0163] Arrange the registered target position data corresponding to the authorized user target and the unauthorized target according to the acquisition time, and generate the motion trajectory of the authorized user target and the unauthorized target.

[0164] In specific implementation, it is preferable to perform the process on the sequence of image frames to be identified after quality screening in step S101. Human detection is performed on each image frame in the sequence to be identified, yielding human target detection results. Specifically, for each current image frame, the image frame can be input into a pre-trained human detection model, which outputs the target detection box and detection confidence score for each human target in the current image frame. The human detection model can be a target detection model based on a convolutional neural network, or any model capable of detecting human targets. This application does not limit the specific model structure, as long as it can input image frames and output human detection boxes.

[0165] As an optional implementation, a single-stage human detection model can be employed, which can consist of a backbone network, a feature fusion network, and a detection head. The backbone network extracts multi-scale visual features from the input image and can use lightweight convolutional networks, such as CSP, MobileNet, or ShuffleNet structures; alternatively, deeper residual networks or cross-stage partially connected networks can be used as the backbone. The feature fusion network fuses shallow spatial detail features and deep semantic features, employing feature pyramid structures and path aggregation structures to upsample, downsample, and concatenate feature maps at different scales, thereby improving the model's ability to detect small people in the distance, large people nearby, and partially occluded people. The detection head predicts human target bounding boxes, target class confidence, and bounding box regression offsets on feature maps at multiple scales. To adapt to the large scale variations of human targets in doorway scenes, three or four detection scales can be set, for example, corresponding to feature maps downsampled by 8, 16, and 32 times the input image.

[0166] This application constructs a human detection dataset for door opening scenes before model training. The human detection dataset for door opening scenes can be obtained through smart door lock cameras, door cameras, video peephole cameras, or access control cameras. The collected content includes images or video clips of the door in closed state, initial opening state, half-open state, large-angle opening state, as well as authorized users opening the door, unauthorized personnel approaching, multiple people walking together, delivery personnel stopping, neighbors passing by, partial occlusion of people, low light at night, backlight, and flickering corridor lights.

[0167] When constructing the dataset, image frames can be extracted from continuous videos at preset time intervals as training samples. For example, frames can be extracted from each video segment depicting a door opening process at a frequency of 3 to 10 frames per second, preferably 5 frames per second. Each image sample is labeled with a human target detection box, which should ideally encompass the visible human body area, including the head, torso, legs, and visible feet. Data annotation information can include a human category label, the coordinates of the top-left corner and bottom-right corner of the detection box, the target size level, the illumination level, and the scene type. Occlusion levels can be categorized as no occlusion, slight occlusion, moderate occlusion, and severe occlusion; target size levels can be classified as small targets, medium targets, and large targets based on the proportion of the detection box area to the image area; illumination levels can be categorized as normal illumination, low illumination, backlight, and strong reflection; scene types can include single person opening a door, multiple people walking together, passersby, and delivery interactions, etc.

[0168] As an example, a dataset of no fewer than 10,000 labeled images can be constructed, in which images of a normal single person opening a door account for 30% to 40%, images of multiple people appearing simultaneously account for 20% to 30%, images of people following or approaching a door gap account for 15% to 25%, low-light or backlit images account for 10% to 20%, and partially occluded images account for 10% to 20%. The dataset can be divided into a training set, a validation set, and a test set, with a ratio of 70%:15%:15% or 80%:10%:10%.

[0169] During model training, transfer learning can be used for initialization. Specifically, model parameters pre-trained on publicly available human detection or general object detection datasets can be used as initialization parameters, followed by fine-tuning using the door opening scene human detection dataset constructed in this application. The loss function during training can include bounding box regression loss, object confidence loss, and category classification loss. CIoU loss is preferably used for bounding box regression loss to simultaneously constrain bounding box overlap, center point distance, and aspect ratio; binary cross-entropy loss or focus loss can be used for object confidence loss and category classification loss.

[0170] The number of training epochs is set to 100 to 300, preferably 200; the batch size is set to 8 to 64, preferably 16 or 32 depending on the available GPU memory; the initial learning rate is set to 0.001 to 0.01, preferably 0.005; the optimizer can be the AdamW optimizer, and the weight decay can be set to 0.01. Learning rate scheduling can employ a cosine annealing strategy or a piecewise decay strategy.

[0171] For each image frame in the door-opening process image sequence, the image is input into the trained single-stage human detection model using the same preprocessing method as in the training phase. After the model outputs candidate human detection boxes, low-confidence detection boxes are first removed based on the detection confidence threshold, and then duplicate detection boxes are removed based on non-maximum suppression to obtain the human target detection result in the current image frame.

[0172] For each human target detection result, the temporary target identifier, target detection box, detection confidence score, and corresponding image frame acquisition time are recorded. The target detection box region is then further cropped for target appearance feature extraction. Specifically, for each retained human target detection box, a single-frame temporary target identifier is assigned, and target appearance features are extracted. Target appearance features can be extracted from the image region corresponding to the human detection box. As an optional implementation, the human detection box region can be cropped, scaled to a fixed size, such as 128×256 pixels or 64×128 pixels, and then input into a human body weight recognition feature extraction network to obtain a feature vector representing the appearance of the human target.

[0173] Simultaneously, based on the registered target position data obtained in step S104, the position of each human target detection result in the coordinate system of the doorway safety area is determined. In step S104, the foot point, center point, or contour point of the detection frame is mapped, and then the corresponding registered target position data is read. Preferably, the registered position corresponding to the target foot point is used in the target association stage because the target foot point is closer to the actual standing position of the human body on the ground, and is more suitable for judging the movement trajectory and mutual distance of the person than the center point of the detection frame.

[0174] After obtaining the human target detection results for each image frame, the human targets are associated based on the target appearance features and registered target position data of the human target detection results in adjacent image frames to obtain target association results. Specifically, the human detection results can be processed frame by frame according to the acquisition time sequence, maintaining one or more existing target trajectories. For each human detection result in the current image frame, the association cost between it and the end detection results of existing target trajectories in the previous image frame or several previous image frames is calculated. The association criteria include appearance similarity, registered position distance, motion direction consistency, detection time interval, and target detection confidence.

[0175] Appearance similarity can be determined based on the distance between target appearance features. If the target appearance features are normalized feature vectors, cosine similarity can be used; the higher the similarity, the more likely the two detection results belong to the same person. As an optional implementation, when using cosine similarity, the appearance similarity threshold can be set to 0.45 to 0.70, preferably 0.55; if the appearance similarity between two targets is lower than this threshold, they are generally not associated as the same target.

[0176] The distance between the registered positions can be determined based on the positional differences between the two detection results in the coordinate system of the safe area at the entrance. Specifically, the distance between the registered foot point of the current detection result and the registered foot point at the end of the existing target trajectory can be calculated. If this distance is less than a preset association distance threshold, the positions are considered to have a high probability of correlation. The preset association distance threshold can be set according to the acquisition frame rate and the normal human movement speed. If using a pixel scale, it can be set to 50 to 150 pixels, preferably 80 pixels; if using a real spatial scale, it can be set to 0.3 meters to 0.8 meters, preferably 0.5 meters. The higher the frame rate, the smaller the change in human position between adjacent frames, and the smaller the association distance threshold can be; when the frame rate is low or the entrance is crowded, the association distance threshold can be appropriately increased.

[0177] Consistency of motion direction is used to help determine whether target association is reasonable. For a target trajectory that has formed at least two historical location points, the target's motion direction can be estimated based on its two or more most recent registered location points, and the possible location of the target at the current acquisition time can be predicted. If the registered location of the current detection result is close to the predicted location, the association confidence is increased; if it deviates significantly from the predicted location, the association confidence is decreased. The prediction method can be uniform prediction, that is, estimating the current location based on the position difference between the previous time and the time before that.

[0178] When associating based on both appearance similarity and post-registration location distance, an association cost can be calculated. As an optional implementation, the association cost is obtained by weighting the appearance cost and the location cost. The appearance cost can be obtained by subtracting the appearance similarity from 1, and the location cost can be obtained by dividing the post-registration location distance by a preset association distance threshold, with costs exceeding the threshold being recorded as higher costs. The weights can be set according to the scenario; for example, the appearance cost weight can be 0.5, and the location cost weight can be 0.5.

[0179] After obtaining the association costs between each detection result in the current image frame and existing target trajectories, the association relationship can be determined using the minimum cost matching method. If the number of targets is small, the matching relationship with the lowest cost and less than the preset cost threshold can be selected one by one; if the number of targets is large, the Hungarian matching algorithm can be used to solve for the minimum total cost matching globally. The preset cost threshold can be set to 0.5 to 0.8, preferably 0.6. If the association cost between a current detection result and an existing trajectory is less than the preset cost threshold, the detection result is associated with the existing trajectory; if all association costs are greater than the preset cost threshold, a new target trajectory is created for the detection result.

[0180] After the target association result is formed, based on the authentication information corresponding to the unlocking trigger event, the authorized user target is determined from the target association result, and human targets different from the authorized user target are determined as unauthorized targets. The authentication information can be obtained from the unlocking trigger event in step S101. For example, when the unlocking trigger event is fingerprint verification successful, the authentication information may include the fingerprint user number, verification time, and lock body position; when the unlocking trigger event is face verification successful, the authentication information may include the face user number, verification time, and the position of the verified face frame; when the unlocking trigger event is password verification successful or access card verification successful, the authentication information may include the password user number or access card user number and verification time; when the unlocking trigger event is remote unlocking via mobile terminal, the authentication information may include the remote unlocking account, unlocking confirmation time, and optional user confirmation image information.

[0181] When determining the authorized user target, different matching rules can be used based on the authentication method. If the unlocking trigger event is a successful face verification, and the verified face region corresponds to a human target detection box in the image frame, then the trajectory of the human target containing that face region or being spatially closest to that face region is preferentially determined as the authorized user target. Specifically, the location of the face verification box can be searched in several image frames near the trigger time. If a human detection box contains the face verification box, or the center point of the face verification box is located in the upper region of the human detection box, then the target trajectory to which the human detection result belongs is determined as the authorized user target. The "near the trigger time" can be set to 0.5 to 2 seconds before and after the trigger time, preferably 1 second before and after.

[0182] If the unlocking event is triggered by fingerprint, password, access card, or mobile terminal near-field unlocking, since the verification location is usually near the door lock or door handle, the target trajectory closest to the door lock operation area or the door opening side near the trigger time can be identified as the authorized user target. Specifically, the door lock operation area can be pre-marked in the doorway security area coordinate system, which can be located near the corresponding lock on the outside of the door frame or door. For human targets appearing near the trigger time, the distance between their registered target position and the door lock operation area is calculated, and the judgment is made in conjunction with whether their hand area or upper body is close to the lock position. If a target trajectory is closest to the door lock operation area near the trigger time, and its position is outside the waiting area or threshold area, and it moves towards the door after unlocking, then the target trajectory is identified as the authorized user target. If the hand area cannot be detected, the judgment can also be made based on the relative position of the target's feet and the door lock operation area in the doorway security area coordinate system.

[0183] After identifying the authorized user target, human targets in the target association results that differ from the authorized user target are designated as unauthorized targets. It's important to note that unauthorized targets are relative to the current unlocking trigger event; that is, they are user targets not confirmed for this unlocking verification. Unauthorized targets can include tailgating personnel, companions, neighbors, delivery personnel, or passersby. If the target association results only contain authorized user targets, the unauthorized target list can be left empty. If multiple trajectories differ from the authorized user target, each is marked as an unauthorized target, and their respective registered target location data and trajectory identifiers are retained.

[0184] Finally, the registered target position data for authorized user targets and unauthorized targets are arranged according to the acquisition time, generating the motion trajectories of both targets. Specifically, for each target trajectory in the target association results, the registered target position data corresponding to each acquisition time is extracted and arranged from earliest to latest acquisition time. If a target is detected at a certain acquisition time, the trajectory points at that time include the acquisition time, the coordinates of the registered foot point, the registered detection box or contour region, the region status, and the detection confidence.

[0185] The data structure of a motion trajectory can include trajectory identifier, target identity type, acquisition time series, registered position sequence, area state sequence, and detection confidence sequence. The target identity type can be an authorized user target or an unauthorized target. The acquisition time series represents the temporal order of each trajectory point. The registered position sequence represents the continuous position of the target in the doorway security zone coordinate system. The area state sequence represents the target's location at each time point in the waiting area outside the door, the threshold area, the door gap area, or other areas.

[0186] Regarding S106 above:

[0187] like Figure 2 As shown, the flowchart for generating risk identification results provided in this application embodiment includes steps S201 to S204, wherein:

[0188] S201, based on the motion trajectory of the unauthorized target, calculate the first distance sequence between the unauthorized target and the door gap area at multiple acquisition times;

[0189] S202, based on the motion trajectory of the unauthorized target and the motion trajectory of the authorized user target, calculate the second distance sequence between the unauthorized target and the authorized user target at multiple acquisition times;

[0190] S203, Based on the registered target position data of the unauthorized target at adjacent acquisition times, determine the movement direction of the unauthorized target;

[0191] S204, match the first distance sequence, the second distance sequence, and the direction of movement with preset risk determination conditions to obtain risk identification results.

[0192] In practice, based on the motion trajectory generated in step S105, the distance change and movement direction of each unauthorized target relative to the door gap area and the authorized user target are determined, and a risk identification result is generated. If there are no unauthorized targets in the target association results, a risk identification result without unauthorized target risk can be directly generated; if there are one or more unauthorized targets, distance sequence calculation, movement direction determination, and risk condition matching are performed for each unauthorized target, and the highest risk level can be selected from all the risk identification results of unauthorized targets as the overall risk identification result of this door opening process.

[0193] First, based on the motion trajectory of the unauthorized target, a first distance sequence between the unauthorized target and the door gap region is calculated at multiple acquisition times. For a given unauthorized target, its motion trajectory includes multiple registered target position data arranged according to acquisition time. Each registered target position data may include a registered foot point, a registered detection box region, or a registered contour point set. Preferably, if a registered contour point set exists, the shortest distance between the unauthorized target contour point set and the door gap region is used as the first distance at that acquisition time; if no contour point set exists but a foot point exists, the shortest distance from the registered foot point to the boundary of the door gap region or the center line of the door gap region is used as the first distance; if only the target detection box center point exists, the shortest distance from the target detection box center point to the door gap region is used as the first distance.

[0194] When calculating the distance from a point to the door gap area, if the door gap area is represented as a strip-shaped polygon in the doorway safety area coordinate system, it can be determined whether the target location point is located within this strip-shaped polygon area. If the target location point is located within the door gap area, the first distance at that acquisition moment can be recorded as 0; if the target location point is located outside the door gap area, the shortest Euclidean distance from that point to the boundary line segment of the door gap area is calculated, and this distance is taken as the first distance. If the target location is a set of contour points, the distance from each contour point to the door gap area is calculated separately, and the minimum value is taken as the first distance.

[0195] For the first distance of an unauthorized target at multiple acquisition times, arrange them in ascending order of acquisition time to obtain the first distance sequence. The first distance sequence includes the distance value at each time, the acquisition time, and the target area status.

[0196] Then, based on the motion trajectories of the unauthorized target and the authorized user target, a second distance sequence between the unauthorized target and the authorized user target at multiple acquisition moments is calculated. Specifically, for each acquisition moment, the trajectory points of the authorized user target that are within a preset synchronization threshold at that moment or time difference are found. The preset synchronization threshold can be set according to the acquisition frame rate. For example, under the condition of acquisition of 15 frames / second, the time interval between adjacent frames is approximately 0.067 seconds, and the synchronization threshold can be set to 0.05 seconds to 0.15 seconds.

[0197] After obtaining the registered positions of the unauthorized target and the authorized user target at the same acquisition time, the distance between them is calculated. Preferably, the Euclidean distance between their registered foot points is used as the second distance. If a registered contour region exists, the shortest distance between the contour regions of the unauthorized target and the authorized user target can also be calculated to determine whether they are close or overlapping; when the two contour regions overlap, the second distance can be recorded as 0. The distance between the center points of the detection boxes can also be used as an auxiliary distance. Arranging the second distances corresponding to multiple acquisition times according to the acquisition time yields the second distance sequence.

[0198] When calculating the second distance sequence, the relative positional relationship between the unauthorized target and the authorized user target can also be calculated simultaneously. For example, it can be determined whether the unauthorized target is behind, to the side or rear of the authorized user target, or outside the door. Specifically, the normal direction of the threshold line or the entrance direction can be determined in the doorway safety zone coordinate system, and the direction in which the authorized user target points towards the threshold area or the interior direction can be used as the entry direction reference. This relative positional relationship can be used as part of the subsequent risk assessment criteria.

[0199] Next, based on the registered target position data of the unauthorized target at adjacent acquisition times, the movement direction of the unauthorized target is determined. Specifically, for the unauthorized target at time... and time After registration, the target position can be determined by calculating the coordinate difference between the two, yielding a displacement vector. The direction of this displacement vector represents the instantaneous motion direction within that time period. If the foot point is used as the trajectory point, the motion direction is determined by the coordinate difference of the foot point. If the foot point is missing but the center point of the detection box exists, the motion direction can be approximately determined by the coordinate difference of the center point. To reduce the impact of single-frame jitter, multiple consecutive trajectory points can be used to determine the average motion direction. For example, the position points from the most recent 3 to 5 frames can be taken, and the vector pointing from the earliest position point to the latest position point can be used as the motion direction within that window.

[0200] Subsequently, the first distance sequence, the second distance sequence, and the direction of movement are matched with preset risk judgment conditions to obtain the risk identification result. The preset risk judgment conditions may include door gap approach condition, following authorized user condition, orientation entry condition, duration condition, and exclusion condition. The door gap approach condition is used to determine whether an unauthorized target is approaching the door gap area. For example, the first distance sequence shows a decreasing trend over multiple consecutive acquisition moments, or the first distance decreases beyond a preset distance change threshold within a preset time window, and ultimately the first distance is less than the door gap approach threshold. The preset time window can be set to 0.5 seconds to 2 seconds, preferably 1 second; the door gap approach threshold can be set to 30 pixels to 80 pixels at the pixel scale, preferably 50 pixels; and can be set to 10 centimeters to 30 centimeters at the actual spatial scale, preferably 20 centimeters. The preset distance change threshold can be set to 30% to 80% of the door gap approach threshold, or to 0.2 meters to 0.5 meters.

[0201] The "follow authorized user" condition is used to determine whether an unauthorized target closely follows an authorized user target. For example, this can be determined if the second distance sequence shows a decreasing trend over multiple consecutive acquisition times, or if the second distance is less than the follow distance threshold and remains below it for a preset duration. The follow distance threshold can be set based on the doorway width and the distance people can pass through. If using an actual spatial scale, it can be set to 0.4 meters to 1.0 meters, preferably 0.6 meters; if using a pixel scale, it can be set based on the doorway width in the reference frame, for example, to 20% to 50% of the doorway width, preferably 30%. The preset duration can be set to 0.3 seconds to 1 second, preferably 0.5 seconds.

[0202] The orientation entry condition is used to determine whether an unauthorized target has a tendency to move towards the door gap, threshold, or interior. Specifically, if the angle between the unauthorized target's movement direction and the direction pointing towards the door gap area is less than 45 degrees, or the angle between the unauthorized target's movement direction and the direction pointing towards the threshold area is less than 45 degrees, or the direction is consistent with the authorized user's target entry direction, then the orientation entry condition is considered met.

[0203] Duration conditions are used to avoid false alarms caused by single-frame false detections or accidental proximity. Specifically, false alarms only occur when the door gap proximity condition, the authorized user contact condition, or the orientation towards entry condition are continuously maintained for at least [duration missing]. The risk level is only increased when the frame or duration exceeds a preset time. This can be set according to the capture frame rate, for example, at 15 frames per second. The frame rate can be set from 3 to 8 frames, with 5 frames being preferred; the duration can be set from 0.3 to 0.8 seconds, with 0.5 seconds being preferred. If the distance is small only in the first frame, but the preceding and following frames do not show a closeness trend, it can be marked as low risk or ignored instead of being considered high risk.

[0204] Exclusion criteria are used to reduce false alarms related to normal behavior. For example, if an unauthorized target appears in the frame, but its first distance sequence consistently exceeds the door gap approach threshold, its second distance sequence consistently exceeds the follow-the-door distance threshold, and its movement direction is opposite to the door gap area, it is considered low-risk or no-risk. If an unauthorized target briefly pauses in the waiting area outside the door but does not move towards the door gap area or approach the authorized user target, it is considered low-risk. If an unauthorized target maintains a large distance from the authorized user target and passes through the doorway from the side, and its trajectory does not enter the door gap area or threshold area, it is considered a passing target, corresponding to low risk. If an unauthorized target has already left the doorway area before the authorized user opens the door, and does not approach the door gap area after the door is opened, it is considered no-risk.

[0205] As an optional implementation, a risk scoring method can be used to generate risk identification results. Specifically, for each unauthorized target, a door gap proximity score, a following score, a direction score, and a duration score are calculated. The door gap proximity score increases as the first distance decreases and the faster the first distance decreases; the following score increases as the second distance decreases and the second distance decreases; the direction score increases as the angle between the movement direction and the door gap direction or entry direction decreases; and the duration score increases as the duration of the aforementioned dangerous state increases. The total risk score is obtained by weighted summation of multiple scores. As an example, the weight of the door gap proximity score can be set to 0.35, the following score weight to 0.30, the direction score weight to 0.25, and the duration score weight to 0.10. If the total risk score is less than 0.3, it is judged as low risk or no risk; if the total risk score is between 0.3 and 0.6, it is judged as medium risk; and if the total risk score is greater than 0.6 or 0.7, it is judged as high risk. The above thresholds can be adjusted according to the application scenario, doorway width, camera angle, and false alarm tolerance.

[0206] Risk identification results can include risk level, risk type, risk occurrence time, and risk basis. Risk level can be no risk, low risk, medium risk, or high risk; risk type can be approaching a door gap, following an authorized user, suspected tailgating, crossing a threshold, or passing a target; risk occurrence time can be the moment when the risk judgment conditions are first met or the moment when the risk is highest; risk basis can record the minimum distance in the first distance sequence, the minimum distance in the second distance sequence, the distance change trend, the angle of the movement direction, and the number of consecutive frames.

[0207] After generating the risk identification result in step S106, proceed to step S107 to generate and send remote security control instructions based on the risk identification result. Specifically, if the risk identification result is no risk or low risk, a normal recording instruction or a low-priority event upload instruction can be generated. For example, key image frames, target trajectory summaries, and low-risk markers during the door opening process can be saved to local storage or cloud logs without triggering a strong alarm. If the risk identification result is medium risk, a reminder-type remote control instruction can be generated, such as sending door image pushes, video clips, and risk warning information to the user terminal, alerting the user that an unauthorized target is approaching the door gap or close to an authorized user. If the risk identification result is high risk, a strong control instruction can be generated, such as sending alarm notifications to the user terminal and the security platform, controlling the smart door lock to remain locked or delaying automatic locking, triggering a doorbell or voice alarm, starting local recording and saving, or requiring the user to confirm again.

[0208] Remote security control commands can include command type, command target, risk target information, risk level, image or video evidence, execution time limit, and feedback requirements. Command types can include image push, video upload, alarm notification, door lock hold, secondary verification, voice alert, or platform reporting. Command targets can be user mobile terminals, indoor control screens, smart door lock control modules, property security platforms, or cloud servers. Risk target information can include unauthorized target identifiers, their trajectory points in the doorway security area coordinate system, closest distance to the door gap area, closest distance to the authorized user, and the risk trigger time. Image or video evidence can include image frames or video clips several seconds before and after the risk occurs, such as a video clip from 2 seconds before to 3 seconds after the risk trigger time. The execution time limit is used to restrict the execution of control commands within a short period of time; for example, a high-risk alarm should be pushed to the user terminal or security platform within 1 to 3 seconds.

[0209] When sending remote security control commands, the sending method can be selected based on network connection status and risk level. For high-risk outcomes, alarms can be pushed to user terminals or security platforms via real-time communication channels. For medium-risk outcomes, images and prompts can be pushed first, awaiting user viewing or confirmation. For low-risk outcomes, only event logs can be uploaded or stored locally.

[0210] Based on the same inventive concept, this application also provides an image recognition-based remote security management system corresponding to an image recognition-based remote security management method. Since the principle of the system in this application is similar to the image recognition-based remote security management method described above, the implementation of the system can refer to the implementation of the method, and the repeated parts will not be described again.

[0211] like Figure 3As shown in the diagram, an embodiment of this application provides a schematic diagram of a remote security control system based on image recognition, including:

[0212] The image acquisition module is used to acquire continuous image frames during the door opening process in response to the unlocking trigger event, and obtain the door opening process image sequence;

[0213] The reference feature extraction module is used to perform image recognition on the door opening process image sequence and extract door motion reference features, which include at least one of the following: door frame edge, door seam line, threshold line, ground texture or wall fixed point;

[0214] The perspective change determination module is used to determine the perspective change parameters of each image frame relative to the reference frame based on the position change of the door motion reference feature in different image frames.

[0215] The image registration module is used to map the target position in each image frame to a unified doorway security area coordinate system based on the viewpoint change parameters, so as to obtain the registered target position data.

[0216] The target trajectory generation module is used to perform human target detection and target association on the door opening process image sequence, and combine the registered target position data to generate the motion trajectory of authorized user targets and unauthorized targets in the coordinate system of the doorway security area;

[0217] The risk identification module is used to determine the distance change and movement direction of the unauthorized target relative to the door gap area and the authorized user target based on the movement trajectory, and generate risk identification results;

[0218] The remote control module is used to generate and send remote security control commands based on the risk identification results.

[0219] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A remote security control method based on image recognition, characterized in that, include: In response to the unlocking trigger event, continuous image frames are acquired during the door opening process to obtain an image sequence of the door opening process; Image recognition is performed on the image sequence of the door opening process to extract door motion reference features, which include at least one of the following: door frame edge, door seam line, threshold line, ground texture or wall fixed point; Based on the positional changes of the door motion reference features in different image frames, the perspective change parameters of each image frame relative to the reference frame are determined. Based on the aforementioned viewpoint change parameters, the target positions in each image frame are mapped to a unified doorway safety area coordinate system to obtain registered target position data. Human target detection and target association are performed on the image sequence of the door opening process, and the motion trajectories of authorized user targets and unauthorized targets in the coordinate system of the doorway security area are generated by combining the registered target position data; Based on the motion trajectory, the distance change and motion direction of the unauthorized target relative to the door gap area and the authorized user target are determined, and a risk identification result is generated; Based on the risk identification results, remote security control instructions are generated and sent.

2. The remote security control method based on image recognition according to claim 1, characterized in that, In response to the unlocking trigger event, continuous image frames are acquired during the door opening process to obtain an image sequence of the door opening process, including: In response to the detection of an unlock trigger event, the trigger time is obtained; The image acquisition start time is determined based on the trigger time, and continuous image frames during the door opening process are acquired at a preset frame rate within a preset acquisition duration. The consecutive image frames are sorted according to the acquisition time to obtain the door opening process image sequence.

3. The remote security control method based on image recognition according to claim 2, characterized in that, After obtaining the image sequence of the door opening process, the method further includes: Brightness equalization, distortion correction, and sharpness evaluation are performed on each image frame in the image sequence of the door opening process to obtain the image quality value of each image frame; The image quality value is compared with a preset quality threshold, and image frames with image quality values ​​less than the preset quality threshold are removed to obtain a sequence of image frames to be identified. The sequence of image frames to be identified is used as the image sequence for image recognition.

4. The remote security control method based on image recognition according to claim 3, characterized in that, The step of performing image recognition on the image sequence of the door opening process and extracting reference features of the door movement includes: Edge detection is performed on the image frames in the image frame sequence to be identified to obtain a set of edge pixels; Line fitting is performed on the edge pixel set to obtain candidate line features; Based on the direction, length, and image position of the candidate line features, determine the door frame edge, door seam line, or threshold line from the candidate line features; Corner detection and texture detection are performed on the image frames in the image frame sequence to be identified to obtain candidate fixed feature points; Based on the matching relationship between the candidate fixed feature points in adjacent image frames, the ground texture or wall fixed points are determined.

5. The remote security control method based on image recognition according to claim 4, characterized in that, The determination of the viewpoint change parameters of each image frame relative to the reference frame based on the positional changes of the door motion reference features in different image frames includes: Image frames that are selected from the image frame sequence to be identified before the door is opened or when the door opening angle is less than a preset angle are used as reference frames. Determine the reference features in the reference frame, and determine the current reference features in the current image frame; The reference feature is matched with the current reference feature to obtain a reference feature matching pair; Based on the coordinate differences between the reference feature matching pair in the reference frame and the current image frame, the viewpoint change parameters of the current image frame relative to the reference frame are determined.

6. The remote security control method based on image recognition according to claim 5, characterized in that, The viewpoint change parameters include translation parameters, rotation parameters, scale parameters, and perspective parameters; determining the viewpoint change parameters of the current image frame relative to the reference frame based on the coordinate difference between the reference feature matching pair in the reference frame and the current image frame includes: The translation parameters are determined based on the pixel coordinate difference between the reference frame and the current image frame using the same gate motion reference feature. The rotation parameters are determined based on the direction difference between the door frame edge, door seam line, or threshold line in the reference frame and the current image frame. The scale parameter is determined based on the distance change between at least two reference feature matching pairs; The perspective parameters are determined based on the pixel coordinates of at least four reference feature matching pairs.

7. The remote security control method based on image recognition according to claim 6, characterized in that, Based on the aforementioned viewpoint change parameters, the target position in each image frame is mapped to a unified doorway safety area coordinate system to obtain registered target position data, including: The door frame edge, door seam line, or threshold line in the reference frame are used as coordinate references to establish the coordinate system of the doorway safety area; Mark the door gap area, threshold area, and waiting area outside the door in the coordinate system of the doorway safety area; Based on the perspective change parameters corresponding to the current image frame, determine the coordinate transformation relationship from the current image frame to the coordinate system of the doorway safety area; The target location is obtained as the center point of the target detection box, the target foot point, or the target contour point in the current image frame. The target position is mapped according to the coordinate transformation relationship to obtain the registered target position data.

8. The remote security control method based on image recognition according to claim 7, characterized in that, The step of performing human target detection and target association on the image sequence of the door opening process, and combining the registered target position data to generate the motion trajectories of authorized user targets and unauthorized targets in the coordinate system of the doorway security area, includes: Human detection is performed on each image frame in the door opening process image sequence to obtain human target detection results, which include target identifier, target detection box and target appearance features; Based on the target appearance features of human target detection results in adjacent image frames and the registered target location data, human targets are associated to obtain target association results. Based on the authentication information corresponding to the unlocking trigger event, the authorized user target is determined from the target association result, and human targets that are different from the authorized user target are determined as unauthorized targets; Arrange the registered target position data corresponding to the authorized user target and the unauthorized target according to the acquisition time, and generate the motion trajectory of the authorized user target and the unauthorized target.

9. A remote security control method based on image recognition according to claim 8, characterized in that, Based on the motion trajectory, the distance change and direction of movement of the unauthorized target relative to the door gap area and the authorized user target are determined, and a risk identification result is generated, including: Based on the motion trajectory of the unauthorized target, calculate the first distance sequence between the unauthorized target and the door gap area at multiple acquisition times; Based on the motion trajectory of the unauthorized target and the motion trajectory of the authorized user target, calculate a second distance sequence between the unauthorized target and the authorized user target at multiple acquisition times; Based on the registered target position data of the unauthorized target at adjacent acquisition times, the movement direction of the unauthorized target is determined; The first distance sequence, the second distance sequence, and the direction of movement are matched with preset risk determination conditions to obtain risk identification results.

10. A remote security control system based on image recognition, used to implement the remote security control method based on image recognition as described in any one of claims 1-9, characterized in that, include: The image acquisition module is used to acquire continuous image frames during the door opening process in response to the unlocking trigger event, and obtain the door opening process image sequence; The reference feature extraction module is used to perform image recognition on the door opening process image sequence and extract door motion reference features, which include at least one of the following: door frame edge, door seam line, threshold line, ground texture or wall fixed point; The perspective change determination module is used to determine the perspective change parameters of each image frame relative to the reference frame based on the position change of the door motion reference feature in different image frames. The image registration module is used to map the target position in each image frame to a unified doorway security area coordinate system based on the viewpoint change parameters, so as to obtain the registered target position data. The target trajectory generation module is used to perform human target detection and target association on the door opening process image sequence, and combine the registered target position data to generate the motion trajectory of authorized user targets and unauthorized targets in the coordinate system of the doorway security area; The risk identification module is used to determine the distance change and movement direction of the unauthorized target relative to the door gap area and the authorized user target based on the movement trajectory, and generate risk identification results; The remote control module is used to generate and send remote security control commands based on the risk identification results.

Images