A fisheye image fall recognition method and related device

Abstract

This invention discloses a fall recognition method and related equipment for fisheye images. By pre-marking reference lines on the fisheye image and establishing a reference angle mapping table for each pixel, this invention can effectively correct the impact of distortion on the calculation of human posture angles, thus avoiding false judgments caused by lens distortion in fall detection. Furthermore, this invention employs a three-tiered cascade architecture: a human detection model for initial screening, a posture estimation model for verification, and a large visual model for final judgment. Each model is independent yet complementary. Specifically, the human detection model quickly eliminates scenes without people, the posture estimation model uses posture key points for fine angle comparison, and the large visual model serves as a supplementary judgment result for extremely difficult cases. Finally, the results of the three are fused and output, ensuring detection efficiency while significantly improving the accuracy and robustness of fall recognition in complex environments, effectively reducing false positives and false negatives, and can be widely applied in the field of image processing technology.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and related equipment for recognizing falls in fisheye images. Background Technology

[0002] With the aging population and increasing public safety demands, fall detection under video surveillance has become an important research direction in the field of computer vision. Traditional fall detection relies on real-time viewing of surveillance footage by humans, which is not only inefficient but also prone to visual fatigue due to prolonged monitoring, leading to missed or false alarms. In recent years, deep learning-based object detection and pose estimation methods have been widely applied to fall recognition tasks, such as using YOLO series networks to detect human bounding boxes or using pose estimation models such as OpenPose to analyze changes in key human points. However, existing technologies still have significant shortcomings in practical deployment: on the one hand, ordinary cameras have limited fields of view and cannot cover large areas, while fisheye cameras, although providing wide-angle images, suffer from inherent radial distortion that causes nonlinear deformation of human pose, making it easy to generate a large number of false alarms when directly applying conventional fall detection logic (such as based on human aspect ratio, descent speed of the center point, etc.); on the other hand, single detection models (using only object detection or only pose estimation) have poor robustness in the face of occlusion, changes in lighting, and complex backgrounds, and the accuracy of fall detection is difficult to meet the actual security needs. Summary of the Invention

[0003] The main objective of this invention is to provide a fisheye image fall recognition method, device, electronic device, storage medium, and program product, aiming to solve at least one problem in the prior art.

[0004] To achieve the above objectives, one aspect of the present invention proposes a fisheye image fall recognition method, the method comprising: Acquire the fisheye image of the target area captured by the fisheye camera; wherein the fisheye image is marked with reference coordinate axes and multiple reference lines perpendicular to the ground; The distortion center point is determined based on the intersection of all reference lines. A calibration angle mapping table is established based on the angle between the line connecting each pixel in the fisheye image to be detected and the distortion center point and the reference coordinate axis. The calibration angle mapping table includes the reference angle of each pixel in the fisheye image to be detected. The fisheye image to be detected is input into the human detection model and processed to obtain the human detection results in the target area. The human detection results include the set of human key points of each person in the target area. The human detection model is trained using a first training image with pre-annotated human key points. Based on the angle between the lines connecting each human key point in the human key point set and the reference coordinate axis, and combined with the reference angle corresponding to each human key point in the human key point set in the calibration angle mapping table, the first fall judgment is made to obtain the first result. If the first result is a fall, the fisheye image to be detected is input into the pose estimation model and processed to obtain the pose detection result of the person in the target area; wherein, the pose detection result includes the set of pose key points of each person in the target area, which is obtained by training the human body detection model using a second training image with pre-annotated pose key points; Based on the angle between the line connecting the preset posture key point pairs in the posture key point set and the reference coordinate axis, and combined with the reference angle corresponding to each posture key point in the posture key point set in the calibration angle mapping table, a second fall judgment is made to obtain a second result. The fisheye image to be detected and the preset prompt words are input into the visual big model for the third fall judgment, and the third result is obtained; The target fall result is obtained by fusing the first, second, and third results.

[0005] In some embodiments, determining the distortion center point based on the intersection of all reference lines includes the following steps: Collect the intersection points of all reference lines from the fisheye image to be detected; If the intersection of the reference lines is unique, the unique intersection point will be taken as the distortion center point; If the intersection points of the reference lines are not unique, the distortion center point can be obtained based on the arithmetic mean or weighted average of all intersection points.

[0006] In some embodiments, the method further includes the following steps: Obtain the YOLOv10n network; For the YOLOv10n network, the first standard convolutional layer is replaced with a depthwise separable convolution, the second standard convolutional layer is replaced with a Ghost convolution, and the middle standard convolutional layer is replaced with a dilated convolution to obtain the initial visual model. The initial visual model is trained for human detection using the first training image. Based on the results of the human detection training, the initial visual model is optimized and its parameters are tuned to obtain the human detection model.

[0007] In some embodiments, the human body key point set includes the center point of the foot position, the center point of the head position, and the center point of the human body frame. Based on the angle between the line connecting each human body key point in the human body key point set and the reference coordinate axis, and combined with the reference angle corresponding to each human body key point in the human body key point set in the calibration angle mapping table, a first fall determination is made to obtain a first result, including the following steps: Extract the first reference angle corresponding to the center point of the foot position, the second reference angle of the center point of the head position, and the third reference angle of the center point of the human body frame from the calibration angle mapping table; The first angle is the angle between the line connecting the center point of the head position and the center point of the human body frame and the reference coordinate axis. The first angle difference is determined based on the absolute value of the difference between the second reference angle and the first included angle, and the second angle difference is determined based on the absolute value of the difference between the third reference angle and the first included angle. The second angle is the angle between the line connecting the center point of the foot position and the center point of the human body frame and the reference coordinate axis. The third angle difference is determined based on the absolute value of the difference between the first reference angle and the second angle, and the fourth angle difference is determined based on the absolute value of the difference between the third reference angle and the second angle. If at least two of the first angle difference, second angle difference, third angle difference, and fourth angle difference are greater than the first preset threshold, the first result is determined to be a fall.

[0008] In some embodiments, the method further includes the following steps: Obtain the YOLOv8pose network as the original visual model; The original visual model is trained using the second training image for pose detection. Based on the results of the pose detection training, the original visual model is optimized and its parameters are tuned to obtain the pose estimation model.

[0009] In some embodiments, the set of posture key points includes a left hip key point, a left ankle key point, a right hip key point, and a right ankle key point. Based on the angle between the line connecting preset pairs of posture key points in the set of posture key points and the reference coordinate axis, and combined with the reference angle corresponding to each posture key point in the set of posture key points in the calibration angle mapping table, a second fall determination is performed to obtain a second result, including the following steps: Extract the fourth reference angle corresponding to the left hip key point, the fifth reference angle corresponding to the left ankle key point, the sixth reference angle corresponding to the right hip key point, and the seventh reference angle corresponding to the right ankle key point from the calibration angle mapping table. The left hip key point and the left ankle key point are designated as the first pose key point pair, and the right hip key point and the right ankle key point are designated as the second pose key point pair. The angle between the line connecting the first pose keypoint pairs and the reference coordinate axis is taken as the third angle; The fifth angle difference is determined based on the absolute value of the difference between the fourth reference angle and the third angle, and the sixth angle difference is determined based on the absolute value of the difference between the fifth reference angle and the third angle. The angle between the line connecting the key points of the second pose and the reference coordinate axis is taken as the fourth angle. The seventh angle difference is determined based on the absolute value of the difference between the sixth reference angle and the fourth angle, and the eighth angle difference is determined based on the absolute value of the difference between the seventh reference angle and the fourth angle. If the difference between the fifth angle and the sixth angle are both greater than the second preset threshold, or if the difference between the seventh angle and the eighth angle are both greater than the second preset threshold, the second result is determined to be a fall.

[0010] In some embodiments, the target fall result is obtained by fusing the first result, the second result, and the third result, including the following steps: If the target human body keypoint is missing from the set of human body keypoints, or if all posture keypoints in the set of posture keypoints cannot form any preset posture keypoint pair, the third result is taken as the target fall result; otherwise... The target fall result is determined to be a fall if at least two of the first, second, and third results are fallable.

[0011] To achieve the above objectives, another aspect of the present invention provides a fisheye image fall recognition device, the device comprising: The first module is used to acquire the fisheye image to be detected captured by the fisheye camera in the target area; wherein, the fisheye image to be detected is marked with reference coordinate axes and multiple reference lines perpendicular to the ground; The second module is used to determine the distortion center point based on the intersection of all reference lines. It establishes a calibration angle mapping table based on the angle between the line connecting each pixel in the fisheye image to be detected and the distortion center point and the reference coordinate axis. The calibration angle mapping table includes the reference angle of each pixel in the fisheye image to be detected. The third module is used to input the fisheye image to be detected into the human detection model and process it to obtain the human detection results in the target area. The human detection results include the set of human key points of each person in the target area. The human detection model is trained using the first training image with pre-annotated human key points. The fourth module is used to make a first fall judgment based on the angle between the line connecting each human key point in the human key point set and the reference coordinate axis, combined with the reference angle corresponding to each human key point in the human key point set in the calibration angle mapping table, and obtain the first result. The fifth module is used to input the fisheye image to be detected into the pose estimation model if the first result is a fall, and process it to obtain the pose detection result of the person in the target area; wherein, the pose detection result includes the set of pose key points of each person in the target area, and the human body detection model is trained using a second training image with pre-annotated pose key points; The sixth module is used to make a second fall determination based on the angle between the line connecting the preset posture key point pairs in the posture key point set and the reference coordinate axis, combined with the reference angle corresponding to each posture key point in the posture key point set in the calibration angle mapping table, and obtain a second result. The seventh module is used to input the fisheye image to be detected and the preset prompt words into the visual big model for the third fall determination, and obtain the third result; The eighth module is used to obtain the target fall result by fusing the first, second, and third results.

[0012] In some embodiments, the apparatus further includes a ninth module for performing the following operations: Obtain the YOLOv10n network; For the YOLOv10n network, the first standard convolutional layer is replaced with a depthwise separable convolution, the second standard convolutional layer is replaced with a Ghost convolution, and the middle standard convolutional layer is replaced with a dilated convolution to obtain the initial visual model. The initial visual model is trained for human detection using the first training image. Based on the results of the human detection training, the initial visual model is optimized and its parameters are tuned to obtain the human detection model.

[0013] In some embodiments, the apparatus further includes a tenth module for performing the following operations: Obtain the YOLOv8pose network as the original visual model; The original visual model is trained using the second training image for pose detection. Based on the results of the pose detection training, the original visual model is optimized and its parameters are tuned to obtain the pose estimation model.

[0014] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned method.

[0015] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0016] To achieve the above objectives, another aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0017] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a fisheye image fall recognition method, device, electronic device, storage medium, and program product. This solution acquires a fisheye image to be detected captured by a fisheye camera in a target area; wherein, the fisheye image to be detected is marked with a reference coordinate axis and multiple reference lines perpendicular to the ground; the distortion center point is determined based on the intersection of all reference lines; a calibration angle mapping table is established based on the angle between the line connecting each pixel in the fisheye image to the distortion center point and the reference coordinate axis; wherein, the calibration angle mapping table includes the reference angle of each pixel in the fisheye image to be detected; the fisheye image to be detected is input into a human detection model, and the person detection result in the target area is obtained; wherein, the person detection result includes a set of human key points for each person in the target area, and the human detection model is trained using a first training image pre-labeled with human key points; based on the human key point set, each human key point... The angle between the lines connecting the points and the reference coordinate axis is used to perform a first fall determination, combined with the reference angles corresponding to each human keypoint in the set of human keypoints in the calibrated angle mapping table, to obtain a first result. If the first result indicates a fall, the fisheye image to be detected is input into the pose estimation model and processed to obtain the pose detection result of the person in the target area. The pose detection result includes the set of pose keypoints of each person in the target area, which is obtained by training the human detection model using a second training image with pre-annotated pose keypoints. The angle between the lines connecting the preset pose keypoint pairs in the pose keypoint set and the reference coordinate axis is used to perform a second fall determination, combined with the reference angles corresponding to each pose keypoint in the set of pose keypoints in the calibrated angle mapping table, to obtain a second result. The fisheye image to be detected and the preset prompt words are input into the visual big model for a third fall determination, to obtain a third result. The target fall result is obtained by fusing the first, second, and third results. This invention, by pre-marking reference lines on fisheye images and establishing a reference angle mapping table for each pixel, effectively corrects the impact of distortion on human posture angle calculation, thus avoiding misjudgments caused by lens distortion in fall detection. Based on this, this invention employs a three-tiered cascade architecture: a human detection model for initial screening, a posture estimation model for verification, and a large visual model for final decision-making. Each model is independent yet complementary. Specifically, the human detection model quickly eliminates scenes without people, the posture estimation model performs fine angle comparisons using posture key points, and the large visual model serves as a supplementary judgment result for extremely difficult cases. Finally, the results of the three models are fused and output, ensuring detection efficiency while significantly improving the accuracy and robustness of fall recognition in complex environments, effectively reducing false positives and false negatives. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of an implementation environment for the fisheye image fall recognition method provided in this embodiment of the invention; Figure 2 This is a flowchart illustrating the fisheye image fall recognition method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the original YOLOv10n network structure provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the improved YOLOv10n network design provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the overall process architecture of the fisheye image fall recognition method provided in the embodiments of the present invention; Figure 6 This is a schematic diagram of the fisheye image fall recognition device provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0020] It is understood that the terms "first," "second," etc., used in this invention may be used to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of embodiments of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to determination," or "in the event of a determination."

[0021] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0022] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this invention is for descriptive purposes only and is not intended to limit the invention.

[0023] In related technologies, existing methods typically use fixed thresholds or rules for judgment, lacking means to correct pixel-level angular shifts in distorted images, and failing to fully utilize the complementary advantages of multiple models. Therefore, how to accurately and robustly identify fall events in fisheye distortion images remains a pressing technical problem to be solved.

[0024] In view of this, this invention provides a fisheye image fall recognition method and related equipment. This method acquires a fisheye image of a target area captured by a fisheye camera. The fisheye image is marked with a reference coordinate axis and multiple reference lines perpendicular to the ground. A distortion center point is determined based on the intersection of all reference lines. A calibration angle mapping table is established based on the angle between the line connecting each pixel in the fisheye image to the distortion center point and the reference coordinate axis. The calibration angle mapping table includes the reference angle of each pixel in the fisheye image. The fisheye image is input into a human detection model, and the resulting data is processed to obtain the personnel detection results for the target area. The personnel detection results include a set of human key points for each person in the target area. The human detection model is trained using a first training image pre-annotated with human key points. The falling detection results are based on the angle between the line connecting each human key point in the set and the reference coordinate axis. The invention uses the angle between the coordinate axes and the reference angles corresponding to each human keypoint in the set of human keypoints in the calibrated angle mapping table to perform a first fall judgment, obtaining a first result. If the first result indicates a fall, the fisheye image to be detected is input into the pose estimation model, and the pose detection result of the person in the target area is obtained. The pose detection result includes the set of pose keypoints of each person in the target area, which is obtained by training the human detection model using a second training image with pre-labeled pose keypoints. Based on the angle between the line connecting the preset pose keypoint pairs in the pose keypoint set and the reference coordinate axis, and combined with the reference angles corresponding to each pose keypoint in the set of pose keypoints in the calibrated angle mapping table, a second fall judgment is performed, obtaining a second result. The fisheye image to be detected and the preset prompt words are input into the visual large model for a third fall judgment, obtaining a third result. The target fall result is obtained by fusing the first, second, and third results. This embodiment of the invention can effectively correct the influence of distortion on the calculation of human pose angles by pre-calibrating reference lines on the fisheye image and establishing a reference angle mapping table for each pixel, thereby avoiding misjudgment caused by lens distortion in fall judgment. Based on this, the embodiments of the present invention adopt a three-level cascaded architecture of initial screening by a human body detection model, verification by a pose estimation model, and final decision by a large visual model. Each model is independent yet complementary. Specifically, the human body detection model quickly eliminates scenes without people, the pose estimation model uses pose key points for fine angle comparison, and the large visual model can serve as a supplementary judgment result for extremely difficult cases. Finally, the results of the three are fused and output, which not only ensures detection efficiency but also significantly improves the accuracy and robustness of fall recognition in complex environments, effectively reducing false alarms and false negatives.

[0025] It is understood that the fisheye image fall recognition method provided by this invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet, laptop, or desktop computer, but it is not limited to these.

[0026] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0027] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0028] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0029] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0030] For example, based on Figure 1 The implementation environment shown in this embodiment of the invention provides a fisheye image fall recognition method. The following description uses the application of this fisheye image fall recognition method in server 101 as an example. It can be understood that this fisheye image fall recognition method can also be applied in terminal 102.

[0031] Reference Figure 2 , Figure 2 This is an optional flowchart of a fisheye image fall recognition method provided in an embodiment of the present invention. The execution subject of the fisheye image fall recognition method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 The method may include, but is not limited to, steps S100 to S800.

[0032] Step S100: Obtain the fisheye image to be detected captured by the fisheye camera in the target area; Among them, the fisheye image to be detected is marked with reference coordinate axes and multiple reference lines perpendicular to the ground; For example, in some specific implementations, due to the high resolution of fisheye cameras, which are composed of 360-degree images stitched together, distortion is easily caused. Firstly, reference lines need to be marked on the recognition image (i.e., the fisheye image to be detected). For example, objects in the target scene, such as walls, pillars, and desks that are perpendicular to the ground, can be selected to determine multiple vertical lines as reference lines.

[0033] Step S200: Determine the distortion center point based on the intersection of all reference lines, and establish a calibration angle mapping table based on the angle between the line connecting each pixel in the fisheye image to be detected and the distortion center point and the reference coordinate axis. The calibration angle mapping table includes the reference angle of each pixel in the fisheye image to be detected; It should be noted that, in some embodiments, determining the distortion center point based on the intersection of all reference lines may include the following steps: acquiring the intersection of all reference lines from the fisheye image to be detected; if the intersection of the reference lines is unique, taking the unique intersection as the distortion center point; if the intersection of the reference lines is not unique, obtaining the distortion center point based on the arithmetic mean or weighted average of all intersections.

[0034] For example, in some specific implementations, firstly, based on multiple lines drawn perpendicular to the ground, intersection points will be generated due to distortion. These intersection points are calculated. If there is only one intersection point, the distortion center point is directly determined. If multiple lines may have multiple intersection points, the average of these intersection points (or a weighted average based on the degree of distortion in different locations) is calculated to determine the distortion center point. Then, the angle between the line connecting each pixel in the image to this distortion center point and a reference coordinate axis (usually the X-axis, subject to testing) is calculated using the tangent trigonometric function principle as the reference angle for the corresponding pixel. Finally, the reference angles of all pixels are summarized to establish a calibration angle mapping table.

[0035] Step S300: Input the fisheye image to be detected into the human detection model and process it to obtain the human detection results in the target area; The personnel detection results include a set of human key points for each person in the target area. The human detection model is trained using a first training image with pre-annotated human key points. It should be noted that in some embodiments, the method may further include the following steps: obtaining a YOLOv10n network; for the YOLOv10n network, replacing the first standard convolutional layer with a depthwise separable convolution, replacing the second standard convolutional layer with a Ghost convolution, and replacing the middle standard convolutional layer with a dilated convolution to configure an initial visual model; using the first training image to train the initial visual model for human detection, and optimizing and tuning the parameters of the initial visual model based on the results of the human detection training to obtain a human detection model.

[0036] For example, in some specific embodiments, the embodiments of the present invention mainly improve the YOLOv10n algorithm to monitor whether there are people in the scene, and if so, extract the key points of the people's bodies. Specifically, such as Figure 3 (Original YOLOv10n network structure) and Figure 4 As shown in the (Improved YOLOv10n Network Design), the construction of the human detection model can be achieved as follows: Replace the first layer: In the model configuration file, find the first standard convolutional layer (Conv2d, k=3, s=2, c=64) and replace it with a depthwise separable convolution module (DSConv). This module first performs a 3×3 depthwise convolution with 64 channels, and then performs a 1×1 pointwise convolution.

[0037] Replace the second layer: Replace the standard convolutions in the second layer with the GhostConv module. This module uses a small number of inherent convolutions to generate some feature maps, and then generates more feature maps through inexpensive linear transformations, reducing computational cost.

[0038] Replace the middle layer: In the middle layer of the model (such as the 8th layer), change the dilation coefficient of the standard convolution from 1 to 2 to make it a dilated convolution, which expands the receptive field without increasing the number of parameters.

[0039] After improving the YOLOv10n network, it can be trained using, for example, a fisheye image dataset containing 5000 labeled head, feet, and center point of the human body. The initial learning rate is set to 0.001, and the network parameters are iteratively optimized for 200 epochs to finally obtain the human detection model.

[0040] in, Figure 3 and Figure 4 The meanings of each module in the network structure are shown in Table 1 below: Table 1

[0041] Step S400: Based on the angle between the line connecting each human key point in the human key point set and the reference coordinate axis, and combined with the reference angle corresponding to each human key point in the human key point set in the calibration angle mapping table, a first fall judgment is made to obtain the first result. It should be noted that the set of key human body points includes the center point of the foot position, the center point of the head position, and the center point of the human body frame. In some embodiments, step S400 may include the following steps: extracting the first reference angle corresponding to the center point of the foot position, the second reference angle of the center point of the head position, and the third reference angle of the center point of the human body frame from the calibration angle mapping table; taking the angle between the line connecting the center point of the head position and the center point of the human body frame and the reference coordinate axis as the first included angle; determining the first angle difference based on the absolute value of the difference between the second reference angle and the first included angle, and determining the second angle difference based on the absolute value of the difference between the third reference angle and the first included angle; taking the angle between the line connecting the center point of the foot position and the center point of the human body frame and the reference coordinate axis as the second included angle; determining the third angle difference based on the absolute value of the difference between the first reference angle and the second included angle, and determining the fourth angle difference based on the absolute value of the difference between the third reference angle and the second included angle; if at least two of the first angle difference, second angle difference, third angle difference, and fourth angle difference are greater than a first preset threshold, the first result is determined to be a fall.

[0042] For example, in some specific implementations, if a person is detected, the center point A of the foot position, the center point B of the head, and the center point C of the human body frame are extracted; then, a first-level angle comparison and judgment step is performed, calculating the first angle D between the line connecting B and C and the reference coordinate axis, and looking up the calibration angles corresponding to point B and C in the calibration angle mapping table, calculating the absolute value of the difference between the calibration angles of D and B, and the absolute value of the difference between the calibration angles of D and C; simultaneously calculating the second angle E between the line connecting C and A and the reference coordinate axis, and looking up the calibration angles corresponding to point A and C, calculating the absolute value of the difference between the calibration angles of E and A, and the absolute value of the difference between the calibration angles of E and C; if at least two of the above four absolute values ​​of difference are greater than a preset threshold, it is determined as a first-level suspected fall; In addition, in some optional implementations, if only one part of the head or feet is obscured, making it impossible to extract B or A, then only the absolute value of the difference corresponding to the unobscured part is calculated. If the absolute value of the difference is greater than the first preset threshold, it is determined to be a first-level suspected fall. If both the head and feet are obscured, then the process jumps directly to step S700 (i.e., fall determination is performed directly through the visual large model).

[0043] Step S500: If the first result is a fall, input the fisheye image to be detected into the pose estimation model and process it to obtain the pose detection result of the person in the target area. The pose detection results include a set of pose key points for each person in the target area, which is obtained by training the human body detection model using a second training image with pre-annotated pose key points. It should be noted that in some embodiments, the method may further include the following steps: obtaining a YOLOv8pose network as the original visual model; using a second training image to train the original visual model for pose detection; and optimizing and tuning the original visual model based on the results of the pose detection training to obtain a pose estimation model.

[0044] For example, in some implementations, a pre-trained YOLOv8pose model, trained on the COCO dataset and capable of detecting 17 keypoints, is first downloaded. Next, 8000 fisheye images annotated with hip and ankle keypoints are collected as second training images. Fine-tuning can be performed using the PyTorch framework, freezing the parameters of the first 10 layers and training only the last few layers. Then, a batch size of 16 and an initial learning rate of 0.0001 are set, and the model is trained for 50 epochs using the SGD optimizer until the keypoint similarity (OKS) on the validation set reaches 0.85. Finally, the trained model is converted to the TensorRT engine and deployed on Jetson Orin to form the final usable pose estimation model.

[0045] Step S600: Based on the angle between the line connecting the preset posture key point pairs in the posture key point set and the reference coordinate axis, and combined with the reference angle corresponding to each posture key point in the posture key point set in the calibration angle mapping table, a second fall judgment is made to obtain a second result. It should be noted that the set of posture key points includes the left hip key point, left ankle key point, right hip key point, and right ankle key point. In some embodiments, step S600 may include the following steps: extracting the fourth reference angle corresponding to the left hip key point, the fifth reference angle corresponding to the left ankle key point, the sixth reference angle corresponding to the right hip key point, and the seventh reference angle corresponding to the right ankle key point from the calibration angle mapping table; taking the left hip key point and the left ankle key point as the first posture key point pair, and the right hip key point and the right ankle key point as the second posture key point pair; and making an angle between the line connecting the first posture key point pairs and the reference coordinate axis. The third included angle is determined; the fifth angle difference is determined based on the absolute value of the difference between the fourth reference angle and the third included angle, and the sixth angle difference is determined based on the absolute value of the difference between the fifth reference angle and the third included angle; the angle between the line connecting the key points of the second posture and the reference coordinate axis is taken as the fourth included angle; the seventh angle difference is determined based on the absolute value of the difference between the sixth reference angle and the fourth included angle, and the eighth angle difference is determined based on the absolute value of the difference between the seventh reference angle and the fourth included angle; if both the fifth angle difference and the sixth angle difference are greater than the second preset threshold, or if both the seventh angle difference and the eighth angle difference are greater than the second preset threshold, the second result is determined to be a fall.

[0046] For example, in some specific implementations, when step S400 determines that it is a first-level suspected fall (i.e., the first result is a fall), the fisheye image to be detected is input into the YOLOv8-pose posture estimation model to extract two sets of key points: the left hip and left ankle, and the right hip and right ankle. If there is at least one set of valid key points, the fall judgment is performed on the set of key points according to the same angle difference calculation method as in step S400. If the absolute value of the difference is greater than the second preset threshold, the second-level confirmed fall is output (i.e., the second result is determined to be a fall).

[0047] Step S700: Input the fisheye image to be detected and the preset prompt words into the visual large model for the third fall judgment to obtain the third result; For example, in some specific implementations, the image is fed into a large model (a general visual large model, not specifically limited) for analysis to determine whether someone has fallen in the image. Specifically, input is given by prompt words: "Determine if someone has fallen in the image; if yes, output 'yes'; if no, output 'no'." Or, more detailed descriptions can be given: "Determine if someone is lying on the ground in the image, or if a fall has occurred (sitting on a chair doesn't count), etc.; output 'yes' if present, output 'no' if absent." No further responses are needed. The prompt words and the image to be detected enable the visual large model to output a third result.

[0048] Step S800: Based on the fusion of the first result, the second result, and the third result, the target fall result is obtained; It should be noted that in some embodiments, step S800 may include the following steps: if the target human body key point is missing in the human body key point set or all posture key points in the posture key point set cannot form any preset posture key point pair, the third result is taken as the target fall result; otherwise, based on the fact that at least two of the first, second and third results are fall results, the target fall result is determined to be a fall.

[0049] For example, in some specific embodiments, the present invention provides adjustable alarm logic for different fisheye cameras based on their height, angle, etc. For complex environments, all three models can be set to detect a fall as a fall, while for moderately challenging environments, at least two models can be set to detect a fall and output a fall message. For other requirements, the final result output by the largest model can also be used.

[0050] The purpose of YOLOv10n is to detect people and determine if someone has fallen. If no one is detected, the process will not continue. If someone is detected but no fall is detected, the data can be sent to YOLOv8-pose and the larger model.

[0051] The specific fusion method can be set according to the application scenario requirements. If some scenarios require all three models to detect a fall simultaneously for it to be effective, then it is a sequential process. If some scenarios only require two models to detect a fall, then it can be processed simultaneously. To avoid the server predicting too many images, YOLOv10n's main task is to detect people before continuing to send them to the subsequent networks.

[0052] To explain in detail the principle of the technical solution of the present invention, the overall process of the present invention will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and should not be regarded as a limitation of the present invention.

[0053] First, it should be noted that traditional manual diagnostic methods are obviously inefficient for the design of existing video surveillance fall detection systems and cannot meet the needs of current fall detection methods.

[0054] To improve the accuracy and efficiency of automated fall detection, this invention provides a fall recognition method combining YOLOv10 fisheye images. For example... Figure 5 As shown, the embodiments of the present invention can achieve the following: S1: Dataset preparation phase.

[0055] We collected and labeled a dataset of people from different scenes. LabelImg was used for labeling. The main labels were people, heads, and feet.

[0056] S2: Model training phase.

[0057] The dataset is divided into training, testing, and validation sets for training. Training parameters can be adjusted according to the actual training equipment. This results in a model that performs as required.

[0058] S3: Data calibration stage.

[0059] Because fisheye cameras have high resolution and are composed of stitched 360-degree images, they are prone to distortion. Therefore, manual marking of reference lines on the recognition image is necessary. Multiple vertical lines can be drawn manually, such as on walls, pillars, or desks, as references. The angle between this reference line and the selected reference coordinate axis is calculated. This angle is then mapped to each pixel to obtain the reference angle for each pixel.

[0060] First, based on the multiple straight lines drawn perpendicular to the ground, which will intersect due to distortion, the intersection points are calculated. Multiple lines will have multiple intersection points. The average of the intersection points is calculated (or a weighted average is calculated based on the degree of distortion in different places). Then, the angle between the line connecting the pixel in the image to the intersection point and the reference axis (usually the X-axis, subject to testing) is calculated using the principle of tangent trigonometric functions.

[0061] S4: Model Application.

[0062] The YOLOv10n model, which is relatively small, was improved through training and testing. When detecting the foot, the center point of the foot was used as reference point A, the center point of the head as reference point B, and the center point of the human bounding box as reference point C. The angle D between points B and C and the fixed coordinate axis of the image (based on trigonometric function calculations, generally the X-axis) was calculated, and the corresponding labeled angles for B and C were found. D was subtracted from the corresponding scaled angles of B and C, and the comparison value was taken (this is a comparison of the calculated angles with the previously labeled angles of the corresponding pixels; it's simply a subtraction of two angles and taking the absolute value). The labeled angle was calculated to form the angle E between the vector formed by points C and A and the fixed coordinate axis of the image. The corresponding labeled angles of A and C were then subtracted from E, and the absolute value was taken. If the foot or head was occluded, only the remaining angles were calculated for comparison. Since both parts are obscured, the image is directly fed into the larger model for analysis and confirmation (because the feet and head are obscured, YOLOv10n cannot correctly identify them, and YOLOv8pose also cannot locate the key points, so it can only be directly fed into the larger model). If there are two points where the absolute value of the included angle is greater than the set threshold, it is considered a fall.

[0063] S5: Application of the Yolov8pose model.

[0064] In the first-stage improved YOLOv10n model, when a fall is detected, the image is sent to YOLOv8pose for further verification. This process retrieves four key points of the person in that area: left hip and left ankle, right hip and right ankle, following the same judgment procedure as the first detection. This step requires at least one pair of key points to be detected ((left hip, left ankle) counts as one pair, (right hip, right ankle) counts as one pair). If no pair exists, the detection is ineffective.

[0065] S6: Application of large models.

[0066] The image is fed into a large model (a general vision model) for analysis to determine if someone has fallen in the image. Specifically, the input is a prompt: determine if someone has fallen in the image; if yes, output "yes"; otherwise, output "no".

[0067] Or, to provide more detail: Determine if the image shows the person lying on the ground or in a fallen state (sitting on a chair doesn't count). Output "yes" if the image exists, and "no" if it doesn't. No further explanation is needed.

[0068] The embodiments of this invention provide adjustable alarm logic for different fisheye cameras based on their height, angle, etc. For complex environments, all three models can be configured to detect a fall as a fall, while for moderately challenging environments, at least two models can be configured to detect a fall and output a fall message. For other requirements, the final result output by the largest model can be used as the standard.

[0069] The purpose of YOLOv10n is to detect people and determine if someone has fallen. If no one is detected, the process will not continue. If someone is detected but no fall is detected, the data can be sent to YOLOv8-pose and the larger model.

[0070] The specific fusion method can be set according to the application scenario requirements. If some scenarios require all three models to detect a fall simultaneously for it to be effective, then it is a sequential process. If some scenarios only require two models to detect a fall, then it can be processed simultaneously. To avoid the server predicting too many images, YOLOv10n's main task is to detect people before continuing to send them to the subsequent networks.

[0071] In summary, this invention, by pre-marking reference lines on fisheye images and establishing a reference angle mapping table for each pixel, can effectively correct the impact of distortion on human posture angle calculation, thus avoiding misjudgments caused by lens distortion in fall detection. Based on this, the embodiments of this invention employ a three-tiered cascade architecture: a human detection model for initial screening, a posture estimation model for verification, and a large visual model for final decision-making. Each model is independent yet complementary. Specifically, the human detection model quickly eliminates scenes without people, the posture estimation model uses posture key points for fine angle comparison, and the large visual model serves as a supplementary judgment result for extremely difficult cases. Finally, the results of the three are fused and output, ensuring detection efficiency while significantly improving the accuracy and robustness of fall recognition in complex environments, effectively reducing false positives and false negatives.

[0072] like Figure 6 As shown, this embodiment of the invention also provides a fisheye image fall recognition device 900, which can implement the above-described method. This device may include: The first module 901 is used to acquire the fisheye image to be detected captured by the fisheye camera in the target area; wherein, the fisheye image to be detected is marked with reference coordinate axes and multiple reference lines perpendicular to the ground; The second module 902 is used to determine the distortion center point based on the intersection of all reference lines, and to establish a calibration angle mapping table based on the angle between the line connecting each pixel in the fisheye image to be detected and the distortion center point and the reference coordinate axis; wherein, the calibration angle mapping table includes the reference angle of each pixel in the fisheye image to be detected. The third module 903 is used to input the fisheye image to be detected into the human detection model and process it to obtain the human detection result of the target area; wherein, the human detection result includes the set of human key points of each person in the target area, and the human detection model is trained using the first training image with pre-annotated human key points; The fourth module 904 is used to make a first fall judgment based on the angle between the line connecting each human key point in the human key point set and the reference coordinate axis, combined with the reference angle corresponding to each human key point in the human key point set in the calibration angle mapping table, and obtain the first result. The fifth module 905 is used to input the fisheye image to be detected into the pose estimation model if the first result is a fall, and process it to obtain the pose detection result of the person in the target area; wherein, the pose detection result includes the set of pose key points of each person in the target area, and the human body detection model is trained using a second training image with pre-annotated pose key points; The sixth module 906 is used to make a second fall judgment based on the angle between the line connecting the preset posture key point pairs in the posture key point set and the reference coordinate axis, combined with the reference angle corresponding to each posture key point in the posture key point set in the calibration angle mapping table, and obtain a second result. The seventh module 907 is used to input the fisheye image to be detected and the preset prompt words into the visual large model for the third fall judgment and obtain the third result; Module 8, 908, is used to obtain the target fall result by fusing the first, second, and third results.

[0073] In some embodiments, the apparatus further includes a ninth module for performing the following operations: Obtain the YOLOv10n network; For the YOLOv10n network, the first standard convolutional layer is replaced with a depthwise separable convolution, the second standard convolutional layer is replaced with a Ghost convolution, and the middle standard convolutional layer is replaced with a dilated convolution to obtain the initial visual model. The initial visual model is trained for human detection using the first training image. Based on the results of the human detection training, the initial visual model is optimized and its parameters are tuned to obtain the human detection model.

[0074] In some embodiments, the apparatus further includes a tenth module for performing the following operations: Obtain the YOLOv8pose network as the original visual model; The original visual model is trained using the second training image for pose detection. Based on the results of the pose detection training, the original visual model is optimized and its parameters are tuned to obtain the pose estimation model.

[0075] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0076] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0077] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0078] like Figure 7 As shown, Figure 7The hardware structure of an electronic device 1000 according to another embodiment is illustrated. The electronic device 1000 includes: The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (aSIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention. The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RaM). The memory 1002 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001. Input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004); The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0079] The electronic device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0080] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0081] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0082] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0083] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0084] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0085] This invention provides a fisheye image fall recognition method, device, electronic device, storage medium, and program product. It acquires a fisheye image of a target area captured by a fisheye camera. The fisheye image is marked with a reference coordinate axis and multiple reference lines perpendicular to the ground. A distortion center point is determined based on the intersection of all reference lines. A calibration angle mapping table is established based on the angle between the line connecting each pixel in the fisheye image to the distortion center point and the reference coordinate axis. The calibration angle mapping table includes the reference angle of each pixel in the fisheye image. The fisheye image is input into a human detection model, and the resulting data is processed to obtain the detection results of people in the target area. The detection results include a set of human key points for each person in the target area. The human detection model is trained using a first training image pre-annotated with human key points. The detection results are based on the line connecting the human key points in the set of human key points. Compared to the angle of the reference coordinate axis, a first fall judgment is made by combining the reference angles corresponding to each human key point in the human key point set in the calibrated angle mapping table, resulting in a first result. If the first result indicates a fall, the fisheye image to be detected is input into the pose estimation model, and the pose detection result of the person in the target area is obtained. The pose detection result includes the set of pose key points of each person in the target area, which is obtained by training the human detection model using a second training image with pre-labeled pose key points. Based on the angle between the line connecting the preset pose key point pairs in the pose key point set and the reference coordinate axis, a second fall judgment is made by combining the reference angles corresponding to each pose key point in the pose key point set in the calibrated angle mapping table, resulting in a second result. The fisheye image to be detected and the preset prompt words are input into the visual large model for a third fall judgment, resulting in a third result. The target fall result is obtained by fusing the first, second, and third results. This embodiment of the invention can effectively correct the influence of distortion on the calculation of human pose angles by pre-calibrating reference lines on the fisheye image and establishing a reference angle mapping table for each pixel, thereby avoiding misjudgment caused by lens distortion in fall judgment. Based on this, the embodiments of the present invention adopt a three-level cascaded architecture of initial screening by a human body detection model, verification by a pose estimation model, and final decision by a large visual model. Each model is independent yet complementary. Specifically, the human body detection model quickly eliminates scenes without people, the pose estimation model uses pose key points for fine angle comparison, and the large visual model can serve as a supplementary judgment result for extremely difficult cases. Finally, the results of the three are fused and output, which not only ensures detection efficiency but also significantly improves the accuracy and robustness of fall recognition in complex environments, effectively reducing false alarms and false negatives.

[0086] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0087] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0088] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0090] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A method for fall recognition using fisheye images, characterized in that, The method includes the following steps: Acquire a fisheye image of the target area captured by a fisheye camera; wherein the fisheye image is marked with reference coordinate axes and multiple reference lines perpendicular to the ground; The distortion center point is determined based on the intersection of all the reference lines, and a calibration angle mapping table is established based on the angle between the line connecting each pixel in the fisheye image to be detected and the distortion center point and the reference coordinate axis; wherein, the calibration angle mapping table includes the reference angle of each pixel in the fisheye image to be detected. The fisheye image to be detected is input into the human detection model and processed to obtain the personnel detection result of the target area; wherein, the personnel detection result includes the set of human key points of each person in the target area, and the human detection model is trained using a first training image with pre-annotated human key points; Based on the angle between the line connecting each human key point in the set of human key points and the reference coordinate axis, and combined with the reference angle corresponding to each human key point in the set of human key points in the calibration angle mapping table, a first fall judgment is made to obtain a first result. If the first result is a fall, the fisheye image to be detected is input into the pose estimation model and processed to obtain the pose detection result of the person in the target area; wherein, the pose detection result includes the set of pose key points of each person in the target area, and the human body detection model is trained using a second training image with pre-annotated pose key points; Based on the angle between the line connecting the preset posture key point pairs in the posture key point set and the reference coordinate axis, and combined with the reference angle corresponding to each posture key point in the posture key point set in the calibration angle mapping table, a second fall determination is made to obtain a second result. The fisheye image to be detected and the preset prompt words are input into the visual big model for the third fall determination to obtain the third result; The target fall result is obtained by fusing the first result, the second result, and the third result.

2. The method according to claim 1, characterized in that, Determining the distortion center point based on the intersection of all the reference lines includes the following steps: Collect the intersection points of all the reference lines from the fisheye image to be detected; If the intersection point of the reference lines is unique, the unique intersection point shall be taken as the distortion center point; If the intersection of the reference lines is not unique, the distortion center point is obtained based on the arithmetic mean or weighted average of all intersection points.

3. The method according to claim 1, characterized in that, The method further includes the following steps: Obtain the YOLOv10n network; For the YOLOv10n network, the first standard convolutional layer is replaced with a depthwise separable convolution, the second standard convolutional layer is replaced with a Ghost convolution, and the middle standard convolutional layer is replaced with a dilated convolution to obtain the initial visual model. The initial visual model is trained for human detection using the first training image, and the parameters of the initial visual model are optimized and tuned based on the results of the human detection training to obtain the human detection model.

4. The method according to claim 1, characterized in that, The set of human body key points includes the center point of the foot position, the center point of the head position, and the center point of the human body frame. The first fall determination is made based on the angle between the line connecting each human body key point in the set and the reference coordinate axis, combined with the reference angle corresponding to each human body key point in the set in the calibration angle mapping table, to obtain a first result, including the following steps: Extract the first reference angle corresponding to the center point of the foot position, the second reference angle of the center point of the head position, and the third reference angle of the center point of the human body frame from the calibration angle mapping table; The angle between the line connecting the center point of the head position and the center point of the human body frame and the reference coordinate axis is taken as the first angle; The first angle difference is determined based on the absolute value of the difference between the second reference angle and the first included angle, and the second angle difference is determined based on the absolute value of the difference between the third reference angle and the first included angle. The angle between the line connecting the center point of the foot position and the center point of the human body frame and the reference coordinate axis is taken as the second angle; A third angle difference is determined based on the absolute value of the difference between the first reference angle and the second included angle, and a fourth angle difference is determined based on the absolute value of the difference between the third reference angle and the second included angle. If at least two of the first angle difference, the second angle difference, the third angle difference, and the fourth angle difference are greater than a first preset threshold, the first result is determined to be a fall.

5. The method according to claim 1, characterized in that, The method further includes the following steps: Obtain the YOLOv8pose network as the original visual model; The original visual model is trained using the second training image for pose detection, and the original visual model is optimized and its parameters are tuned based on the results of the pose detection training to obtain the pose estimation model.

6. The method according to claim 1, characterized in that, The set of posture key points includes the left hip key point, left ankle key point, right hip key point, and right ankle key point. The second fall determination is performed based on the angle between the line connecting preset pairs of posture key points in the set and the reference coordinate axis, combined with the reference angle corresponding to each posture key point in the set in the calibrated angle mapping table, to obtain a second result. This includes the following steps: Extract the fourth reference angle corresponding to the left hip key point, the fifth reference angle corresponding to the left ankle key point, the sixth reference angle corresponding to the right hip key point, and the seventh reference angle corresponding to the right ankle key point from the calibration angle mapping table. The left hip key point and the left ankle key point are designated as the first posture key point pair, and the right hip key point and the right ankle key point are designated as the second posture key point pair. The angle between the line connecting the first pair of attitude key points and the reference coordinate axis is taken as the third angle. The fifth angle difference is determined based on the absolute value of the difference between the fourth reference angle and the third angle, and the sixth angle difference is determined based on the absolute value of the difference between the fifth reference angle and the third angle. The angle between the line connecting the second attitude key point pair and the reference coordinate axis is taken as the fourth angle; The seventh angle difference is determined based on the absolute value of the difference between the sixth reference angle and the fourth angle, and the eighth angle difference is determined based on the absolute value of the difference between the seventh reference angle and the fourth angle. If the fifth angle difference and the sixth angle difference are both greater than the second preset threshold, or if the seventh angle difference and the eighth angle difference are both greater than the second preset threshold, the second result is determined to be a fall.

7. The method according to claim 1, characterized in that, The process of obtaining the target fall result by fusing the first result, the second result, and the third result includes the following steps: If the target human body key point is missing from the set of human body key points, or if all posture key points in the set of posture key points cannot form any of the preset posture key point pairs, the third result is taken as the target fall result; otherwise... Based on the existence of at least two falls among the first result, the second result, and the third result, the target fall result is determined to be a fall.

8. A fisheye image fall recognition device, characterized in that, The device includes: The first module is used to acquire a fisheye image of the target area captured by a fisheye camera; wherein the fisheye image of the target area is marked with a reference coordinate axis and multiple reference lines perpendicular to the ground; The second module is used to determine the distortion center point based on the intersection of all the reference lines, and to establish a calibration angle mapping table based on the angle between the line connecting each pixel in the fisheye image to be detected and the distortion center point and the reference coordinate axis; wherein, the calibration angle mapping table includes the reference angle of each pixel in the fisheye image to be detected. The third module is used to input the fisheye image to be detected into the human detection model and process it to obtain the personnel detection result of the target area; wherein, the personnel detection result includes the set of human key points of each person in the target area, and the human detection model is trained using a first training image with pre-annotated human key points; The fourth module is used to make a first fall determination based on the angle between the line connecting each human key point in the set of human key points and the reference coordinate axis, and in combination with the reference angle corresponding to each human key point in the set of human key points in the calibration angle mapping table, so as to obtain a first result. The fifth module is used to input the fisheye image to be detected into the pose estimation model if the first result is a fall, and process it to obtain the pose detection result of the person in the target area; wherein, the pose detection result includes the set of pose key points of each person in the target area, and the human body detection model is trained using a second training image with pre-annotated pose key points; The sixth module is used to perform a second fall determination based on the angle between the line connecting the preset pairs of posture key points in the posture key point set and the reference coordinate axis, combined with the reference angle corresponding to each posture key point in the posture key point set in the calibration angle mapping table, to obtain a second result. The seventh module is used to input the fisheye image to be detected and the preset prompt words into the visual big model for the third fall determination, and to obtain the third result; The eighth module is used to obtain the target fall result by fusing the first result, the second result, and the third result.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images