A fall detection system and detection method based on computer vision and mobile electronic device

Abstract

The present invention relates to a fall detection system based on computer vision, comprising a millimeter wave radar module, a vision module, a processor and a storage module; by performing relevant clustering algorithm operations on millimeter wave radar data fed back by a scene, valid data objects whose millimeter wave feedback signal strength exceeds a threshold in the scene are obtained, and then based on geometric relationship algorithm calculations, the position and height information of objects within the effective range of module detection are obtained; when a fall event is detected by the millimeter wave radar module, based on a target detection algorithm and a key point detection algorithm, the system drives a camera to move and shoot a scene picture and uploads the picture together with corresponding detection information to a storage module for algorithm application; the processor receives the detection information and image data, processes the image data based on a multi-modal fine-tuning large model recognition algorithm, obtains key valid content from the output result according to a prompt word, and when the output content meets the defined relevant conditions, it is determined that the multi-modal large model recognition algorithm detects a fall event.

Description

Technical Field

[0001] The present invention relates to the technical field of fall detection, and in particular to a fall detection system and detection method based on computer vision and a mobile electronic device. Background Art

[0002] At present, there are three models of elderly care in my country: home-based elderly care, community-based elderly care and institutional elderly care. Among them, more than 99% of the elderly live in homes and communities, and less than 1% live in institutions. As the elderly age, their aging symptoms and degenerative lesions will gradually accelerate and deepen, their physiological functions will gradually decline, and their cognitive abilities will also decline. In home and community environments, elderly people living alone are prone to fall safety hazards.

[0003] How to automatically monitor falls for the elderly, patients, and people in special places is an urgent problem to be solved. At present, in the field of fall monitoring, mainstream monitoring methods based on wearable devices, environmental sensors, detection radars, etc. face problems such as complex equipment and high costs. Among them, the fall detection method based on computer vision technology can generally be divided into two detection methods: dynamic and static. Based on artificial intelligence algorithms, the dynamic method detects a certain video clip, and the static method detects a certain picture.

[0004] The dynamic method iteratively runs the neural network algorithm in the form of video frames in chronological order. This method requires high-speed camera hardware to record video frames, and the amount of data processing is large. The algorithm used requires the device to be adapted, which brings higher edge device deployment costs. The network-based data transmission to the cloud server for processing will also face greater transmission bandwidth pressure.

[0005] The static method runs algorithm processing on two-dimensional image data. This method is limited by the scale of the neural network model and the training data. It is difficult to adapt to complex and diverse home scenarios and is prone to missed recognition and misrecognition. Summary of the invention

[0006] In order to overcome the shortcomings and deficiencies in the prior art, the purpose of the present invention is to provide a fall detection system based on computer vision. The fall detection system is equipped with a millimeter-wave radar module and a vision module, and collaborates with the edge device and the server to run a millimeter-wave radar position detection algorithm, a basic image recognition algorithm, and a multimodal fine-tuning large model recognition algorithm to realize fall event detection based on static images.

[0007] The object of the present invention is achieved by the following technical solution: A fall detection system based on computer vision, characterized in that it includes a millimeter wave radar module, a vision module, a processor and a storage module;

[0008] The millimeter wave radar module performs a related clustering algorithm operation on the millimeter wave radar data fed back by the scene through the system to obtain valid data objects whose millimeter wave feedback signal strength exceeds a threshold in the scene, and obtains the position and height information of the objects within the effective range of the module detection based on further geometric relationship algorithm calculation;

[0009] The visual module, when the millimeter wave radar module detects a fall event, based on the target detection algorithm and the key point detection algorithm, the system drives the camera to move and take pictures of the scene and uploads the pictures together with the corresponding detection information to the storage module of the algorithm application;

[0010] The processor receives detection information and image data, processes the image data based on a multimodal fine-tuning large model recognition algorithm, obtains key valid content from the output result according to the prompt word, and when the output content meets the defined relevant conditions, it is determined that the multimodal large model recognition algorithm detects a fall event.

[0011] Preferably, the execution of the multimodal fine-tuning large model recognition algorithm includes the following steps:

[0012] S1, input image data img, text word prompt, mean matrix mean, standard deviation matrix std;

[0013] S2. Execute img i =img i -mean; where i = (1, N);

[0014] S3. Execute img i =img i ÷std;

[0015] S4, adapting multi-modal fine-tuning large model reasoning;

[0016] S5、event=VL func(img i , prompt)

[0017] S6, ifevent==True then

[0018] S7, if the event is correct, then output;

[0019] S8. Otherwise, output others and continue executing until the end.

[0020] Preferably, when the multimodal fine-tuning large model recognition algorithm detects a fall event, the image data is processed by a semantic segmentation algorithm to obtain a segmentation area data matrix of specific categories of objects in the image scene, which is recorded as ground area, human body, bed, sofa, chair, and TV area, and the image data is processed by an object detection algorithm to record the human body, bed, sofa, chair, and TV detection frame information, and the above information is used for filtering processing.

[0021] Preferably, the filtering process includes detection frame overlap judgment, detection frame and segmentation matrix area overlap judgment, segmentation matrix area overlap judgment, and fall event judgment based on detection and segmentation.

[0022] Preferably, when making a detection frame overlap judgment, the human body detection frame and the TV detection frame are traversed, the intersection of the human body detection frame and the TV detection frame is calculated, and the maximum proportion of the intersection to the area of ​​the TV detection frame or the area of ​​the human body detection frame is calculated. When the proportion meets the set threshold, the fall event of the current human body detection frame is ignored.

[0023] Preferably, the millimeter wave radar module and the vision module are communicatively connected to a local server or a cloud server, and the processor and the storage module are part of the local server or the cloud server.

[0024] Preferably, the camera is an RGB camera, an RGBD camera, or an IR camera.

[0025] The present invention also provides a fall detection method based on computer vision, and the fall detection method uses the above-mentioned fall detection system based on computer vision to detect falls.

[0026] The present invention also provides a mobile electronic device, which includes a processor, a storage module, and a computer program stored in the storage module and executable on the processor.

[0027] The beneficial effects of the present invention are:

[0028] (1) Dynamic fall detection generally requires algorithmic processing of video segments and usually requires video segments of a certain length. Based on the calculation of three-channel images with a resolution of 720x480 and 15 frames per second, the transmission of each video segment based on the H264 video encoding method consumes at least 1502KB of traffic, which consumes a lot of traffic. The present invention performs fall detection based on a static method and cloud-edge cooperative computing method. Based on the calculation of three-channel images with a resolution of 720x480, a single detection consumes a maximum of 150KB of traffic, which can greatly reduce the pressure on network transmission bandwidth and reduce network communication charges.

[0029] (2) To implement fall detection in a dynamic way, it is necessary to shoot high-frame rate video, and then extract frames and encode the video segments. This requires the use of high-performance cameras and corresponding video encoding and decoding hardware, which increases the production cost of the equipment. At the same time, this solution requires the deployment of video detection algorithms on edge devices, which places high demands on the algorithm running equipment and the equipment system resource occupation, which will greatly increase the equipment production cost. The present invention implements detection based on static images, which can reduce the performance requirements for the camera and avoid video encoding and decoding operations, further reducing the equipment hardware cost and system resource occupation. The device side only needs to run conventional image detection algorithms, which can be adapted to a variety of computing hardware, avoiding the equipment cost of specific computing units and the R&D investment of related algorithm adaptation.

[0030] (3) Balance between accuracy and performance. To implement fall detection in a dynamic manner, it is necessary to intercept video clips of the interval where the event may occur according to the trigger conditions, and then integrate them into matrix data for algorithm processing. Based on the three-channel image with a resolution of 720x480 and a 10-second video segment with 3 frames per second, each detection needs to process about 30MB of data. Based on this solution, affected by the diverse trigger conditions, multiple video captures and algorithm processing may occur, which brings complicated calculations to the application functions, occupies system operating resources, and reduces the real-time performance of the system. Based on the solution of the present invention, each time the trigger conditions are met, the three-channel image with a resolution of 720x480 is calculated, and each detection only needs to process a maximum of 1MB of data, which reduces the occupation of system operating resources and improves the real-time performance and stability of the overall system functions. At the same time, based on this solution, real-time image data of all trigger conditions can be processed. Under the combination of the basic image recognition algorithm on the edge device side and the high-precision large model recognition algorithm on the server side, the trigger situation can be fully detected to achieve a balance between accuracy and performance. DETAILED DESCRIPTION

[0031] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with embodiments, and the contents mentioned in the implementation modes are not intended to limit the present invention.

[0032] A fall detection system based on computer vision, characterized in that it includes a millimeter wave radar module, a vision module, a processor and a storage module;

[0033] The millimeter wave radar module performs a related clustering algorithm operation on the millimeter wave radar data fed back by the scene through the system to obtain valid data objects whose millimeter wave feedback signal strength exceeds a threshold in the scene, and obtains the position and height information of the objects within the effective range of the module detection based on further geometric relationship algorithm calculation;

[0034] The visual module, when the millimeter wave radar module detects a fall event, based on the target detection algorithm and the key point detection algorithm, the system drives the camera to move and take pictures of the scene and uploads the pictures together with the corresponding detection information to the storage module of the algorithm application;

[0035] The processor receives detection information and image data, processes the image data based on a multimodal fine-tuning large model recognition algorithm, obtains key valid content from the output result according to the prompt word, and when the output content meets the defined relevant conditions, it is determined that the multimodal large model recognition algorithm detects a fall event.

[0036] The execution of the multi-modal fine-tuning large model recognition algorithm includes the following steps:

[0037] S1, input image data img, text word prompt, mean matrix mean, standard deviation matrix std;

[0038] S2. Execute img i =img i -mean; where i = (1, N);

[0039] S3. Execute img i =img i÷ std;

[0040] S4, adapting multi-modal fine-tuning large model reasoning;

[0041] S5, event = VLfunc (img i , prompt)

[0042] S6, ifevent==True then

[0043] S7, if the event is correct, then output;

[0044] S8. Otherwise, output others and continue executing until the end.

[0045] When the multimodal fine-tuning large model recognition algorithm detects a fall event, the image data is processed by a semantic segmentation algorithm to obtain a segmentation area data matrix of specific categories of objects in the image scene, which is recorded as ground area, human body, bed, sofa, chair, and TV area. The image data is also processed by an object detection algorithm to record the human body, bed, sofa, chair, and TV detection frame information, and the above information is used for filtering processing.

[0046] The filtering process includes detection frame overlap judgment, detection frame and segmentation matrix area overlap judgment, segmentation matrix area overlap judgment, and fall event judgment based on detection and segmentation.

[0047] Detection box overlap judgment algorithm

[0048] Input: detection boxes box1 and box2

[0049] Output: True for overlap, False for non-overlap

[0050] 1: intersection=area(box1∩box2)

[0051] 2: min_area=min(area(box1), area(box2))

[0052] 3: ratio=intersection / min_area

[0053] 4:ifratio>=intersection threshold then

[0054] 5: Output True

[0055] 6: else

[0056] 7: Output False

[0057] 8: end if.

[0058] Detection box and segmentation matrix area overlap judgment algorithm

[0059] Input: detection box and segmentation matrix area mask

[0060] Output: True for overlap, False for non-overlap

[0061] 1: intersection=area(box∩mask)

[0062] 2: min_area=min(area(box), area(mask))

[0063] 3: ratio=intersection / min_area

[0064] 4:ifratio>=intersection threshold then

[0065] 5: Output True

[0066] 6: else

[0067] 7: Output False

[0068] 8: end if.

[0069] Segmentation matrix region overlap judgment algorithm

[0070] Input: segmentation matrix region mask and mask2

[0071] Output: True for overlap, False for non-overlap

[0072] 1: intersection=area(mask1∩mask2)

[0073] 2: min_area=min(area(mask1), area(mask2))

[0074] 3: ratio=intersection / min_area

[0075] 4:ifratio>=intersection thresholdthen

[0076] 5: Output True

[0077] 6: else

[0078] 7. Output False

[0079] 8: end if.

[0080] Fall event judgment algorithm based on detection and segmentation

[0081] Input: image img, fall event fall event

[0082] Output: fall event

[0083] 1: for i = 1 to N do

[0084] 2: #Segmentation algorithm reasoning

[0085] 3: masks = Segmentation (img i )

[0086] 4: #Remember the floor and carpet area as the ground area floor_mask

[0087] 5: ifsum(floor_mask)>floor_thresholdthen

[0088] 6:#Detection algorithm reasoning

[0089] 7: boxes = Detection (img i )

[0090] 8: #Remember that the human body frame is person_box and the TV frame is tv_box

[0091] 9: for i=1to size(person_box)do

[0092] 10:forj=1to size(tv_box)do

[0093] 11: if algorithm 2(person_box i , tv_box j )==True then

[0094] 12: Ignore the current fall_event

[0095] 13: else

[0096] 14: # Posture algorithm reasoning

[0097] 15:pose=Classification(person_box i )

[0098] 16: ifpose == True then

[0099] 17: Ignore the current fall_event

[0100] 18: else

[0101] 19: for k=1to size(bed_box)do

[0102] 20: if algorithm 2 (personbox i , bedbox k )==True then

[0103] 21: Ignore the current fall_event

[0104] 22: end if

[0105] 23: end for #bed_box

[0106] 24:for m=1to size(couch_box)do

[0107] 25: if algorithm 2(person_box i , couch_box m )==True then

[0108] 26: Ignore the current fall_event

[0109] 27: end if

[0110] 28: end for#couchbox

[0111] 29: for n = 1 to size(chairbox) do

[0112] 30: if algorithm2(personbox i , chair_box n ) == True then

[0113] 31: Ignore the current fall_event

[0114] 32: end if

[0115] 33: end for#chair_box

[0116] 34: for p = 1 to size(person_mask) do

[0117] 35: if algorithm3(personbox i , person_mask p ) == True then

[0118] 36: for h = 1 to size(tv_mask) do

[0119] 37: if algorithm4(person mask p , tv_mask h ) == False then

[0120] 38: for f = 1 to size(bed_mask) do

[0121] 39: if algorithm4(person mask p , bed_mask f ) == True then

[0122] 40: Ignore the current fall_event

[0123] 41: end if

[0124] 42: end for

[0125] 43: for g = 1 to size(couch_mask) do

[0126] 44: if algorithm 4(person_mask p , couch_mask g )==True then

[0127] 45: Ignore the current fall_event

[0128] 46: end if

[0129] 47: end for

[0130] 48: Output fall_event

[0131] 49: end if #person tv_mask intersection

[0132] 50: end for #tv_mask

[0133] 51: end if #person_box_mask intersection

[0134] 52: end for #person_mask

[0135] 53: end if #pose == True

[0136] 54: end if #person_tv_box intersection

[0137] 55: end for #tv_box

[0138] 56: end for #person_box

[0139] 57: else

[0140] 58: Ignore the current fall_event

[0141] 59: end if

[0142] 60: end for #picture

[0143] (1) Traverse the human body detection frame and the TV detection frame, calculate the intersection of the human body detection frame and the TV detection frame, and calculate the maximum proportion of the intersection to the area of ​​the TV detection frame or the area of ​​the human body detection frame. When the proportion meets the set value, ignore the fall event of the current human body detection frame.

[0144] (2) The posture recognition algorithm is used to process the human body detection frame. When the posture corresponding to the human body detection frame is normal, the fall event of the current human body detection frame is ignored. Otherwise, the horizontal coordinate center of the current human body frame people_midx, the vertical coordinate half bottom point of the human body frame people_half_bottom_y, the vertical coordinate half bottom point of the human body frame people_halfbottom_y, and the vertical coordinate bottom point of the human body frame people_bottom_y are recorded.

[0145] (3) Record the upper left horizontal coordinate bed_x1, the lower right horizontal coordinate bed_x2, and the bottom vertical coordinate bedbottom_area_y of the bed detection frame. Calculate the intersection of the human detection frame and the bed detection frame, and calculate the maximum proportion of the intersection to the area of ​​the bed detection frame or the area of ​​the human detection frame. When the proportion meets the set threshold, and bed_bottom_area_y is greater than people_bottom_y and people_mid_x is within the range of [bed_x1, bed_x2], ignore the fall event of the current human detection frame.

[0146] (4) Record the upper left horizontal coordinate couch_x1, the lower right horizontal coordinate couch_x2, and the bottom vertical coordinate couchbottom_area_y of the sofa detection frame. Calculate the intersection of the human detection frame and the sofa detection frame, and calculate the maximum proportion of the intersection to the area of ​​the sofa detection frame or the area of ​​the human detection frame. When the proportion meets the set value, and couch bottom area y is greater than peoplehalf bottom_ypeople_mid_x and is within the range of [couch_xl, couch_x2], ignore the fall event of the current human detection frame.

[0147] (5) Record the upper left horizontal coordinate chair_x1, the lower right horizontal coordinate chair_x2, and the bottom vertical coordinate chairbottom area_y of the chair detection frame. Calculate the intersection of the human detection frame and the chair detection frame, and calculate the maximum proportion of the intersection to the area of ​​the chair detection frame or the area of ​​the human detection frame. When the proportion meets the set value and chairbottom_area_y is greater than the top vertical coordinate people_y of the human detection frame, ignore the fall event of the current human detection frame.

[0148] (6) Take the TV segmentation matrix area and the human body segmentation matrix area, calculate the intersection of the relevant areas, and ignore the human body area that intersects with the TV area. Traverse all human body segmentation matrix areas, calculate the intersection of the human body area and the human body detection frame, and calculate the maximum proportion of the intersection to the human body area area or the human body detection frame area. When the proportion does not meet the set threshold, ignore the current fall event.

[0149] (7) Record the partition matrix areas of the bed and sofa, calculate the intersection of the relevant area and the human body area, and calculate the maximum proportion of the intersection to the area of ​​the relevant area or the area of ​​the human body. When the proportion meets the set threshold, ignore the current fall event.

[0150] After being processed by the above algorithm, the cloud server computing node sends the fa11_event signal to the device system, realizing the fall event detection for static images based on the collaborative operation of the millimeter-wave radar position detection algorithm, basic image recognition algorithm and multimodal large model algorithm based on the edge device and server.

[0151] The millimeter wave radar module and the vision module are communicatively connected to a local server or a cloud server, and the processor and the storage module are part of the local server or the cloud server.

[0152] The camera is an RGB camera, an RGBD camera, or an IR camera.

[0153] A fall detection method based on computer vision, wherein the fall detection method uses the above-mentioned fall detection system based on computer vision to detect falls.

[0154] A mobile electronic device comprises a processor, a storage module and a computer program stored in the storage module and executable on the processor.

[0155] The above embodiments are preferred implementation schemes of the present invention. In addition, the present invention may also be implemented in other ways. Any obvious replacement without departing from the concept of the present invention is within the protection scope of the present invention.

Claims

1. A fall detection system based on computer vision, characterized in that: Includes millimeter-wave radar module, vision module, processor and storage module; The millimeter wave radar module performs a related clustering algorithm operation on the millimeter wave radar data fed back by the scene through the system to obtain valid data objects whose millimeter wave feedback signal strength exceeds a threshold in the scene, and obtains the position and height information of the objects within the effective range of the module detection based on further geometric relationship algorithm calculation; The visual module, when the millimeter wave radar module detects a fall event, based on the target detection algorithm and the key point detection algorithm, the system drives the camera to move and take pictures of the scene and uploads the pictures together with the corresponding detection information to the storage module of the algorithm application; The processor receives detection information and image data, processes the image data based on a multimodal fine-tuning large model recognition algorithm, obtains key valid content from the output result according to the prompt word, and when the output content meets the defined relevant conditions, it is determined that the multimodal large model recognition algorithm detects a fall event.

2. A fall detection system based on computer vision according to claim 1, characterized in that: The execution of the multi-modal fine-tuning large model recognition algorithm includes the following steps: S1, input image data img, text word prompt, mean matrix mean, standard deviation matrix std; S2. Execute img i =img i -mean; where i = (1, N); S3. Execute img i =img i ÷std; S4, adapting multi-modal fine-tuning large model reasoning; S5、event=VL func(img i ,prompt) S6, ifevent==True then S7, if the event is correct, then output; S8. Otherwise, output others and continue executing until the end.

3. A fall detection system based on computer vision according to claim 2, characterized in that: When the multimodal fine-tuning large model recognition algorithm detects a fall event, the image data is processed by a semantic segmentation algorithm to obtain a segmentation area data matrix of specific categories of objects in the image scene, which is recorded as ground area, human body, bed, sofa, chair, and TV area. The image data is also processed by an object detection algorithm to record the human body, bed, sofa, chair, and TV detection frame information, and the above information is used for filtering processing.

4. A fall detection system based on computer vision according to claim 3, characterized in that: The filtering process includes detection frame overlap judgment, detection frame and segmentation matrix area overlap judgment, segmentation matrix area overlap judgment, and fall event judgment based on detection and segmentation.

5. A fall detection system based on computer vision according to claim 5, characterized in that: When making a detection frame overlap judgment, traverse the human body detection frame and the TV detection frame, calculate the intersection of the human body detection frame and the TV detection frame, and calculate the maximum proportion of the intersection to the area of ​​the TV detection frame or the area of ​​the human body detection frame. When the proportion meets the set threshold, ignore the fall event of the current human body detection frame.

6. A fall detection system based on computer vision according to claim 1, characterized in that: The millimeter wave radar module and the vision module are communicatively connected to a local server or a cloud server, and the processor and the storage module are part of the local server or the cloud server.

7. The computer vision-based fall detection system according to claim 1, characterized in that: The camera is an RGB camera, an RGBD camera, or an IR camera.

8. A fall detection method based on computer vision, characterized in that: The fall detection method uses the fall detection system described in any one of claims 1-7 to detect falls.

9. A mobile electronic device, characterized in that: The mobile electronic device includes a processor, a storage module, and a computer program stored in the storage module and executable on the processor.