Method and apparatus for evaluating degree of user behavior abnormality, electronic device, and medium

Abstract

The present disclosure relates to the technical field of image processing, and provides a method and apparatus for evaluating a degree of user behavior abnormality, an electronic device, and a medium. The method comprises: collecting synchronous behavior videos of a user to be evaluated under at least two different viewing angles; on the basis of the synchronous behavior videos, performing behavioral analysis of said user to obtain behavior quantization indexes of said user; and inputting the behavior quantization indexes into a degree of behavioral abnormality evaluation model to obtain degree of behavioral abnormality information of said user outputted by the degree of behavioral abnormality evaluation model, wherein the degree of behavioral abnormality evaluation model is obtained by performing model training on the basis of sample behavior quantization indexes and abnormality degree labels thereof. In the present disclosure, behavioral analysis is performed on multiple synchronous behavior videos, so that the analysis result is comprehensive, behavior quantization indexes of the user within a long time period can be obtained, and thus, degree of behavioral abnormality information of said user can be accurately obtained, thereby improving the evaluation accuracy of the degree of user behavior abnormality.

Description

Method and device for evaluating abnormality degree of user behavior, electronic device and medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The present disclosure claims priority to the Chinese patent application No. 2024107880383, filed on June 18, 2024, and entitled "Method and device for evaluating abnormality degree of user behavior, electronic device and medium", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates to the technical field of image processing, and particularly relates to a method and device for evaluating abnormality degree of user behavior, and an electronic device and medium. BACKGROUND

[0004] At present, the field of automatic medical diagnosis assisted by artificial intelligence technology is developing rapidly. There are mainly two types of related behavior abnormality degree evaluation algorithms. One type is to construct a classification model through a commonly used abnormal behavior (such as autism spectrum disorder) diagnosis observation scale to make a predictive diagnosis. The other type is to use a camera to detect hand gestures and facial orientations and other behavior characteristics in specific interactive scenes.

[0005] However, the classification model based on the scale mainly relies on the comprehensive judgment of medical history, behavior observation, and scale evaluation. The professional background, clinical experience, and length of time for seeing patients of the doctor will all affect the accuracy of the diagnosis, and the diagnosis result has a strong subjectivity. Moreover, the current behavior recognition algorithm based on the camera usually only uses a single camera, and can only detect a single local behavior characteristic in different interactive scenes, and can only identify the belonging action category, but cannot distinguish the difference between the same action, lacks a certain comprehensiveness, and is difficult to capture the behavior data of the infant for different test units in a long time range, and accurately evaluate the severity and risk coefficient of the abnormal behavior.

[0006] Therefore, the accuracy of the current evaluation of the abnormality degree of user behavior is low.

[0007] SUMMARY

[0008] The present disclosure provides a method and device for evaluating abnormality degree of user behavior, and an electronic device and medium, to solve the problem of low accuracy in the current evaluation of the abnormality degree of user behavior, and to improve the accuracy of the evaluation of the abnormality degree of user behavior.

[0009] The disclosure provides an abnormality degree evaluation method of user behavior, comprising: collecting synchronous behavior videos of a user to be evaluated under at least two different perspectives; performing behavior analysis of the user to be evaluated according to each synchronous behavior video to obtain a behavior quantitative indicator of the user to be evaluated; inputting the behavior quantitative indicator into a behavior abnormality degree evaluation model to obtain abnormality degree information of the behavior of the user to be evaluated output by the behavior abnormality degree evaluation model; and the behavior abnormality degree evaluation model is obtained by model training of sample behavior quantitative indicators and their abnormality degree labels.

[0010] Optionally, the behavior analysis of the user to be evaluated according to each synchronous behavior video to obtain the behavior quantitative indicator of the user to be evaluated comprises: determining a target behavior video from each synchronous behavior video; performing skeleton key point detection on the target behavior video to obtain first coordinate information of each skeleton key point of the user to be evaluated in a coordinate system corresponding to the target behavior video; and performing behavior analysis based on the first coordinate information to obtain the behavior quantitative indicator of the user to be evaluated.

[0011] Optionally, the behavior analysis based on the first coordinate information to obtain the behavior quantitative indicator of the user to be evaluated comprises: performing behavior recognition based on the first coordinate information to obtain behavior classification information of the user to be evaluated; performing coordinate conversion based on the first coordinate information to obtain second coordinate information of each skeleton key point of the user to be evaluated in a world coordinate system; and determining the behavior quantitative indicator of the user to be evaluated according to the behavior classification information and the second coordinate information.

[0012] Optionally, the determination of the behavior quantitative indicator of the user to be evaluated according to the behavior classification information and the second coordinate information comprises: determining a quantitative indicator of the user to be evaluated based on each second coordinate in the second coordinate information and time information corresponding to each second coordinate in the target behavior video; and determining the behavior quantitative indicator of the user to be evaluated based on the behavior classification information and the quantitative indicator.

[0013] Optionally, the coordinate conversion based on the first coordinate information to obtain the second coordinate information of each skeleton key point of the user to be evaluated in the world coordinate system comprises: obtaining intrinsic parameters of a camera corresponding to each perspective synchronous behavior video and extrinsic parameters relative to the world coordinate system; and performing coordinate conversion on the first coordinate in the first coordinate information based on the intrinsic parameters of each camera and the extrinsic parameters to obtain the second coordinate information of each skeleton key point of the user to be evaluated in the world coordinate system.

[0014] Optionally, the determining the target behavior video from each synchronized behavior video comprises: performing occlusion detection on each synchronized behavior video to obtain an occlusion detection result; and determining the target behavior video from each synchronized behavior video according to the occlusion detection result.

[0015] According to the method for evaluating the abnormal degree of user behavior provided in the present disclosure, the occlusion detection result comprises an occlusion degree and an occlusion duration; and the determining the target behavior video from each synchronized behavior video according to the occlusion detection result comprises: determining a first comparison result between each occlusion degree in the occlusion detection result; determining a second comparison result between each occlusion duration in the occlusion detection result; and determining the target behavior video from each synchronized behavior video according to the first comparison result and the second comparison result.

[0016] The present disclosure further provides a device for evaluating the abnormal degree of user behavior, comprising: a collection module configured to collect synchronized behavior videos of a user to be evaluated from at least two different perspectives; an analysis module configured to perform behavior analysis on the user to be evaluated according to each synchronized behavior video to obtain a behavior quantization index of the user to be evaluated; and an evaluation module configured to input the behavior quantization index into a behavior abnormal degree evaluation model to obtain abnormal degree information of the behavior of the user to be evaluated output by the behavior abnormal degree evaluation model; wherein the behavior abnormal degree evaluation model is obtained by model training based on sample behavior quantization indexes and abnormal degree labels thereof.

[0017] The present disclosure further provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method for evaluating the abnormal degree of user behavior according to any one of the above when executing the program.

[0018] The present disclosure further provides a medium, which is a non-transitory computer-readable storage medium, and a computer program is stored on the medium, wherein the computer program is executable on a processor to implement the method for evaluating the abnormal degree of user behavior according to any one of the above.

[0019] The method, device, electronic equipment and medium for evaluating the abnormality degree of user behavior provided by the present disclosure, by collecting synchronous behavior videos of a user to be evaluated under at least two different perspectives; and performing behavior analysis on the user to be evaluated according to each synchronous behavior video to obtain a behavior quantitative index of the user to be evaluated; thus, the behavior quantitative index can be input into a behavior abnormality degree evaluation model, and since the behavior abnormality degree evaluation model is obtained by model training based on sample behavior quantitative indexes and their abnormality degree labels, the abnormality degree information of the behavior of the user to be evaluated output by the behavior abnormality degree evaluation model can be obtained. Since synchronous behavior videos under multiple perspectives are collected, and the multiple synchronous behavior videos are analyzed for behavior, the differences between the same actions can be distinguished, the analysis result is comprehensive, and thus the behavior quantitative index of the user in a long time range is obtained, and then the abnormality degree evaluation model obtained by model training based on sample behavior quantitative indexes and their abnormality degree labels can be used to accurately obtain the abnormality degree information of the behavior of the user to be evaluated, and the accuracy of the evaluation of the abnormality degree of user behavior can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present disclosure or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] FIG. 1 is a flowchart of a method for evaluating the abnormality degree of user behavior provided by the present disclosure.

[0022] FIG. 2 is an architectural diagram of a device for evaluating the abnormality degree of user behavior provided by the present disclosure.

[0023] FIG. 3 is a structural diagram of an electronic device provided by the present disclosure. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present disclosure clearer, the technical solutions in the present disclosure will be described clearly and completely below with reference to the drawings in the present disclosure. Obviously, the described embodiments are some embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present disclosure.

[0025] It should be noted that in the description of the present disclosure, the terms "comprising", "containing" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device comprising the element. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present disclosure and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present disclosure. Unless otherwise specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication between two elements. For those skilled in the art, the specific meaning of the above terms in the present disclosure can be understood according to the specific circumstances.

[0026] The terms "first", "second", and the like in the present disclosure are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a class, and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" means at least one of the connected objects, and the character " / ", generally means that the front and rear associated objects are in a "or" relationship.

[0027] The user behavior abnormality degree evaluation method, device, electronic equipment and medium of the present disclosure are described below in conjunction with FIG. 1-3.

[0028] FIG. 1 is a flow diagram of a user behavior abnormality degree evaluation method provided by the present disclosure, as shown in FIG. 1, the method comprises the following steps 100 to 300.

[0029] Step 100, collecting synchronous behavior videos of the user to be evaluated under at least two different perspectives.

[0030] Step 200, performing behavior analysis of the user to be evaluated according to each synchronous behavior video to obtain behavior quantitative indicators of the user to be evaluated.

[0031] In step 300, the behavior quantification index is input into the behavior abnormality degree evaluation model to obtain abnormality degree information of the behavior of the user to be evaluated output by the behavior abnormality degree evaluation model. The behavior abnormality degree evaluation model is obtained by model training based on sample behavior quantification indexes and abnormality degree labels thereof.

[0032] It should be noted that the execution subject of the user behavior abnormality degree evaluation method provided in the embodiments of the present disclosure can be a server, a computer device, etc., such as a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a wearable device, an Ultra-mobile Personal Computer (UMPC), a netbook, or a Personal Digital Assistant (PDA), etc. The server or the computer device of the present disclosure can be provided with or connected to a user behavior abnormality degree evaluation apparatus. The user behavior abnormality degree evaluation apparatus can be controlled in the present disclosure to execute the user behavior abnormality degree evaluation method of the present disclosure by controlling the user behavior abnormality degree evaluation apparatus.

[0033] In one embodiment, the user behavior abnormality degree evaluation method of the present disclosure can be deployed on a local server to receive a server request in the same local area network, and to realize multi-path synchronous camera video analysis work in a period of time. If a certain delay is tolerated, the algorithm can also be deployed on a cloud server, and the server can call the algorithm interface of the present disclosure in the form of remote access.

[0034] The present disclosure can realize multi-path camera data alignment according to the absolute time stamp of each frame of the network camera, for example, aligning four different angle synchronous behavior videos.

[0035] The present disclosure can also be executed offline before the algorithm analysis work after the task scene arrangement is completed, and the multi-frame images are collected using the checkerboard and the ground control points, the pose solution is executed to complete the calibration work and output the internal and external parameter files of the calibrated camera, and the internal and external parameter files can include the camera internal and external parameters.

[0036] It should be noted that the abnormal behavior of the user in the present disclosure can be autism spectrum disorder (also known as autism, hereinafter referred to as autism).

[0037] The present disclosure can collect videos of the same user to be evaluated by using multiple synchronous cameras or cameras at the same time. The multiple synchronous cameras or cameras are located at different angles, respectively. The user to be evaluated can be any user who needs to be evaluated. In general, the user to be evaluated is a child.

[0038] Therefore, the synchronization behavior videos of the user to be evaluated at at least two different perspectives can be obtained. In an embodiment, the disclosure can capture the synchronization behavior videos of the user to be evaluated at four different perspectives, i.e., four synchronization behavior videos are obtained.

[0039] Further, the disclosure can determine an optimal synchronization behavior video from the plurality of synchronization behavior videos, and perform behavior analysis on the optimal synchronization behavior video, to obtain a behavior quantitative index of the user to be evaluated in a period of time after the analysis is completed.

[0040] The behavior quantitative index includes action classification information of the performed action and a quantitative index of the action performance. For example, for a running action, the behavior quantitative index can be running speed, running distance, etc. For a hand lifting action, the behavior quantitative index can be hand lifting amplitude.

[0041] It should be noted that the disclosure can pre-acquire the behavior quantitative index of the sample user as a sample behavior quantitative index, and acquire the abnormality degree label of the sample user, i.e., the abnormality degree label corresponding to the sample behavior quantitative index. The sample user can include a user with abnormal behavior (e.g., autism) and a user without abnormal behavior. The abnormality degree label is the abnormality degree of the sample user. For the sample user without abnormal behavior, the abnormality degree label can be normal. For the sample user with abnormal behavior, the abnormality degree label can be corresponding mild, moderate or severe.

[0042] Further, a random forest classifier can be pre-determined as a diagnosis classification model, and the sample behavior quantitative index and the abnormality degree index thereof are used as training data to train the diagnosis classification model, to obtain a behavior abnormality degree evaluation model which can predict and output whether the behavior is abnormal and the abnormality degree according to the input behavior quantitative index after the training is completed.

[0043] Therefore, after obtaining the behavior quantitative index of the user to be evaluated, the behavior quantitative index of the user to be evaluated can be input into the behavior abnormality degree evaluation model to obtain the abnormality degree information of the behavior of the user to be evaluated output by the behavior abnormality degree evaluation model.

[0044] The abnormality degree information can be normal, mild, moderate or severe.

[0045] The behavior features screened by the disclosure input the potential reactions and interaction information of autistic children to various tasks, which can fully express a series of social barrier behaviors, stereotyped behaviors and anxiety behaviors commonly existing in autistic children, and diagnose the severity of autism from the fine quantitative indexes of various actions, effectively assisting doctors to make targeted intervention treatment.

[0046] The method for evaluating the abnormal degree of user behavior provided by the embodiments of the present disclosure comprises: collecting synchronous behavior videos of a user to be evaluated from at least two different perspectives; and performing behavior analysis on the user to be evaluated according to each synchronous behavior video to obtain a behavior quantitative index of the user to be evaluated. Thus, the behavior quantitative index can be input into a behavior abnormal degree evaluation model. Since the behavior abnormal degree evaluation model is obtained by model training based on sample behavior quantitative indexes and their abnormal degree labels, the abnormal degree information of the behavior of the user to be evaluated output by the behavior abnormal degree evaluation model can be obtained. Since synchronous behavior videos from multiple perspectives are collected and the multiple synchronous behavior videos are analyzed, the differences between the same actions can be distinguished, and the analysis results are comprehensive. Thus, the behavior quantitative index of the user in a long time range can be obtained, and the abnormal degree evaluation model obtained by model training based on sample behavior quantitative indexes and their abnormal degree labels can be used to accurately obtain the abnormal degree information of the behavior of the user to be evaluated, thereby improving the accuracy of the evaluation of the abnormal degree of user behavior.

[0047] In one embodiment, the behavior analysis on the user to be evaluated according to each synchronous behavior video to obtain a behavior quantitative index of the user to be evaluated comprises:

[0048] determining a target behavior video from each synchronous behavior video;

[0049] performing skeleton key point detection on the target behavior video to obtain first coordinate information of each skeleton key point of the user to be evaluated in a coordinate system corresponding to the target behavior video;

[0050] performing behavior analysis based on the first coordinate information to obtain the behavior quantitative index of the user to be evaluated.

[0051] The present disclosure can first determine an optimal video from each synchronous behavior video as a target behavior video. Specifically, the optimal video from each synchronous behavior video can be determined as a target behavior video through occlusion detection and / or quality evaluation.

[0052] For the quality evaluation method, the present disclosure can analyze each synchronous behavior video in terms of clarity, stability, lighting, etc., and obtain a quality evaluation value of each synchronous behavior video by comprehensively analyzing the results in terms of clarity, stability, lighting, etc. Then, the target behavior video can be determined from each synchronous behavior video according to the quality evaluation value.

[0053] If the occlusion detection and quality evaluation method is used, after obtaining the occlusion degree and quality evaluation value of each synchronous behavior video, the target behavior video can be determined from each synchronous behavior video according to the comprehensive value of the occlusion degree and quality evaluation value of each synchronous behavior video.

[0054] Further, skeleton key points detection can be performed on the target behavior video, so as to detect all skeleton key points of the user to be evaluated, and a first coordinate of each skeleton key point in an image corresponding to different time nodes, wherein the first coordinate is a coordinate in a coordinate system corresponding to the target behavior video. That is, each skeleton key point includes a plurality of first coordinates.

[0055] Specifically, in the present disclosure, high-dimensional feature extraction can be performed on the original video, i.e., the target behavior video, and then the extracted high-dimensional features are inferred to detect and locate the time nodes at which the action occurs. According to the detected action interval, the corresponding video segment is intercepted for two-dimensional skeleton and three-dimensional skeleton acquisition, which reduces the interference and unnecessary inference time caused by other non-key frames. At the same time, due to the time synchronization of the four-way video, only one-way video needs to be detected for timing positioning, and the timing positioning result can be applied to other videos for synchronous cutting.

[0056] Further, the present disclosure uses a top-down pose estimation algorithm (RTMPose) to perform human bounding box detection on the cropped video frames, and then cut out the local pictures of the human body for skeleton key point detection. It includes two deep learning models for human detection and key point detection, and uses TensorRT for model optimization and acceleration. Compared with general skeleton key point detection algorithms, it has higher accuracy and faster inference. TensorRT is a high-performance inference engine configured for deep learning inference acceleration. It can optimize deep learning models to improve inference speed and reduce inference latency, so as to more efficiently deploy and run deep learning models in production environments.

[0057] In the present disclosure, the skeleton key points can include nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0058] Further, the first coordinate information is formed by all the first coordinates.

[0059] Further, behavior recognition and quantitative index analysis can be performed according to each first coordinate in the first coordinate information, to obtain action classification information and corresponding quantitative indexes.

[0060] Further, the behavior quantitative index can be formed according to the action classification information and the corresponding quantitative indexes.

[0061] The disclosure obtains the behavior quantitative indicators of the user to be evaluated by performing behavior analysis on the user to be evaluated according to each synchronous behavior video after collecting synchronous behavior videos of the user to be evaluated at at least two different angles; and the behavior quantitative indicators are input into the behavior abnormality degree evaluation model. Since the behavior abnormality degree evaluation model is obtained by model training based on sample behavior quantitative indicators and abnormality degree labels, the abnormality degree information of the behavior of the user to be evaluated output by the behavior abnormality degree evaluation model can be obtained. Since synchronous behavior videos at multiple angles are collected and the multiple synchronous behavior videos are analyzed, the differences between the same actions can be distinguished, the analysis results are comprehensive, the behavior quantitative indicators of the user in a long time range are obtained, and then the abnormality degree evaluation model is obtained by model training based on sample behavior quantitative indicators and abnormality degree labels, so that the abnormality degree information of the behavior of the user to be evaluated can be accurately obtained, and the accuracy of the abnormality degree evaluation of the user behavior can be improved.

[0062] In one embodiment, the target behavior video is determined from each synchronous behavior video, including:

[0063] The occlusion detection is performed on each synchronous behavior video to obtain an occlusion detection result.

[0064] The target behavior video is determined from each synchronous behavior video according to the occlusion detection result.

[0065] After obtaining each synchronous behavior video, the occlusion degree and the continuous occlusion time of each synchronous behavior video can be detected respectively, so that the occlusion detection result is formed by the occlusion degree and the continuous occlusion time corresponding to each synchronous behavior video.

[0066] Further, the occlusion degree and the continuous occlusion time of each synchronous behavior video in the occlusion detection result are compared respectively, and the target behavior video is determined from each synchronous behavior video according to the comparison results.

[0067] Further, the target behavior video is determined from each synchronous behavior video according to the occlusion detection result, including:

[0068] A first comparison result between the occlusion degrees in the occlusion detection result is determined.

[0069] A second comparison result between the continuous occlusion times in the occlusion detection result is determined.

[0070] The target behavior video is determined from each synchronous behavior video according to the first comparison result and the second comparison result.

[0071] Specifically, after obtaining the occlusion detection result, the disclosure can compare the sizes of the occlusion degrees in the occlusion detection result, thereby determining the size relationship between the occlusion degrees as a first comparison result. In addition, the sizes of the continuous occlusion times in the occlusion detection result are compared, thereby determining the size relationship between the continuous occlusion times as a second comparison result.

[0072] Further, after obtaining the first comparison result and the second comparison result, the synchronous behavior video with the smallest occlusion degree and the smallest continuous occlusion time in each synchronous behavior video can be selected as the target behavior video.

[0073] The disclosure can realize action recognition under multiple perspectives. Video data obtained by multiple perspective cameras can provide more abundant information, which helps to improve the accuracy and reliability of action recognition.

[0074] Compared with general single-perspective recognition mode, the resistance to non-standard child behavior is improved, and human intervention in child posture and body orientation is not required, thereby avoiding causing child stress reaction, and behavior data under multiple perspectives is more comprehensive.

[0075] Further, based on the first coordinate information, behavior analysis is performed to obtain behavior quantitative indicators of the user to be evaluated, including:

[0076] Based on the first coordinate information, behavior recognition is performed to obtain behavior classification information of the user to be evaluated;

[0077] Based on the first coordinate information, coordinate conversion is performed to obtain second coordinate information of each skeleton key point of the user to be evaluated in the world coordinate system;

[0078] According to the behavior classification information and the second coordinate information, the behavior quantitative indicators of the user to be evaluated are determined.

[0079] After obtaining the first coordinate information containing a plurality of first coordinates corresponding to each skeleton key point, the disclosure can perform behavior recognition, i.e., action recognition, according to the first coordinate information by using the PoseC3D algorithm, to obtain behavior classification information of the user to be evaluated, i.e., what action is performed by the user to be evaluated.

[0080] Specifically, the PoseC3D algorithm is an action recognition algorithm combining pose estimation and three-dimensional convolutional neural network (3D CNN). When performing action recognition based on each first coordinate in the two-dimensional skeleton information, i.e., the first coordinate information, the PoseC3D algorithm can convert the coordinate data of each two-dimensional skeleton key point extracted from the video into pose information in a three-dimensional space, and combine the 3D CNN to realize recognition of the action.

[0081] And, the disclosure can respectively perform coordinate conversion on each two-dimensional first coordinate in the first coordinate information to obtain second coordinate information of each skeleton key point of the user to be evaluated in a world coordinate system, wherein each second coordinate in the second coordinate information is a three-dimensional coordinate.

[0082] Further, the disclosure can determine a quantitative indicator according to the second coordinate information, and determine a behavior quantitative indicator of the user to be evaluated according to the quantitative indicator and corresponding behavior classification information.

[0083] Further, the coordinate conversion based on the first coordinate information to obtain the second coordinate information of each skeleton key point of the user to be evaluated in the world coordinate system comprises:

[0084] Obtaining the intrinsic parameters of the camera corresponding to the synchronous behavior video of each view angle and the extrinsic parameters relative to the world coordinate system;

[0085] Performing coordinate conversion on the first coordinate in the first coordinate information based on the intrinsic parameters of each camera and the extrinsic parameters to obtain the second coordinate information of each skeleton key point of the user to be evaluated in the world coordinate system.

[0086] Specifically, according to the first coordinate of the two-dimensional skeleton key point detected by the multi-camera synchronous detection and the camera intrinsic parameters calibrated by the calibration module in advance, and the extrinsic parameters of each camera relative to the world coordinate system, accurate three-dimensional coordinate automatic conversion from the image coordinate system to the world coordinate system is realized.

[0087] The specific implementation adopts a triangulation algorithm. The intrinsic parameter matrices of four cameras numbered 1, 2, 3, and 4 in turn are K1, K2, K3, and K4, and the extrinsic parameters relative to the world coordinate system are T1 = [R1|t1], T2 = [R2|t2], T3 = [R3|t3], and T4 = [R4|t4], wherein R and t are the rotation matrix and the translation vector from the world coordinate system to the camera coordinate system. The pixel coordinates of the two-dimensional skeleton key points detected in the current synchronous frame are [u1, v1], [u2, v2], [u3, v3], and [u4, v4]. The three-dimensional key point coordinate calculation method is:

[0088] First, the projection matrix of each camera is obtained: P i = K i [R i |t i ], i ∈ (1, 2, 3, 4);

[0089] P i is a 3x4 matrix, which is split into 3 row vectors to facilitate subsequent calculation:

[0090] Let the homogeneous coordinate form of the real world coordinate of the three-dimensional key point X be,

[0091] X = [x, y, z, 1] T ;

[0092] Then the coordinate conversion relationship of the matching point pixel coordinates under each view camera is as follows:

[0093] λ is the coefficient after projection to the normalized image plane, and the two sides are cross multiplied Since the cross product of the same vector is 0, we have:

[0094] Expand the above formula by cross product calculation method,

[0095] The third equation in this formula is linearly related to the first two, and can be simplified as:

[0096] The projection of a three-dimensional point X to a camera view can obtain a set of such constraints, and multiple views can obtain multiple sets of constraints, and the essential matrix A is obtained:

[0097] SVD decomposition is performed on A, A = UDV T The last column vector of matrix V is the solution of the equation, and the optimal solution of the 3D point world coordinate X is obtained. According to the joint number, the first coordinates of each key point in the world coordinate system are calculated by matching the 17 skeleton key points of the current four frames and performing the above operation, and the complete three-dimensional skeleton accurate coordinates in the world coordinate system at the current time are obtained, that is, the second coordinate information of each skeleton key point of the user to be evaluated in the world coordinate system is obtained.

[0098] Further, according to the behavior classification information and the second coordinate information, the behavior quantitative indicators of the user to be evaluated are determined, including:

[0099] Based on each second coordinate in the second coordinate information and the time information corresponding to each second coordinate in the target behavior video, the quantitative indicators of the user to be evaluated are determined.

[0100] Based on the behavior classification information and the quantitative indicators, the behavior quantitative indicators of the user to be evaluated are determined.

[0101] Specifically, the disclosure can identify the motion amount according to the second coordinates of each skeleton key point in the second coordinate information and the time information (or time information) corresponding to each second coordinate in the target behavior video, thereby determining the quantitative indicators of the user to be evaluated. The quantitative indicators may, for example, be distance, amplitude, etc.

[0102] Further, the disclosure can determine the behavior quantification indicators of the user to be evaluated according to each behavior type of the behavior classification information and the corresponding quantification indicators.

[0103] For example, after recognizing the running action, the speed and distance of the movement are automatically calculated, and after recognizing the hand-raising action, the amplitude of the hand-raising is automatically calculated. At the same time, the response time response_time and the action duration of the infant are automatically calculated according to the task start time and the timing action given by the virtual reality (VR) device to locate the detected action start time during the execution of each task. Compared with other video behavior analysis algorithms, the algorithm has more detailed and accurate behavior quantification indicators, improves the fine granularity of behavior features, and contains more powerful and rich deep information.

[0104] The disclosure fully captures the behavior data of the infant at each angle by arranging multiple different view cameras, uses multi-view camera calibration to obtain the internal and external parameters of the multiple synchronous cameras, realizes three-dimensional skeleton reconstruction in the world coordinate system, obtains accurate three-dimensional key point coordinates, and provides a data basis for subsequent calculation of skeleton timing changes and action completion standards.

[0105] The disclosure can automatically detect the occurrence time and end time of the reaction and action of the autism child in the video, and the category of the action based on the timing action positioning model and the skeleton-based action recognition model. By frame extraction, only the video in the time interval of interest is processed and analyzed, greatly reducing the workload of video analysis, and reflecting the differences in the execution of task actions by children.

[0106] The disclosure also includes the following advantages:

[0107] Efficient behavior feature extraction: the algorithm has the ability to efficiently extract key behavior features related to autism from video data. These features may include social interaction disorders, communication difficulties, stereotyped behaviors, etc. The algorithm can accurately capture and quantify these features.

[0108] Accurate behavior pattern recognition: after extracting the features, the algorithm needs to use machine learning techniques to recognize the patterns of these features to distinguish the differences between autistic and non-autistic children. This requires the algorithm to have high classification accuracy and robustness.

[0109] Precise action quantification analysis: the algorithm has the ability to accurately calculate three-dimensional skeleton coordinates, can obtain the external parameters of each camera to the world coordinate system through multi-view camera joint calibration, and realize fine three-dimensional skeleton reconstruction. The algorithm can adapt to any number of cameras (two or more), and still has stability in the case of unavailability of any video.

[0110] Multi-view action recognition: Compared with the general single-view recognition mode, the resistance to children's behavior irregularities is improved, without human intervention in children's sitting posture, body orientation, etc., to avoid causing children's stress reaction, and the behavior data under multiple views are more comprehensive.

[0111] The user behavior abnormality degree evaluation device provided by the present disclosure is described below, and the user behavior abnormality degree evaluation device described below can be mutually corresponding with the user behavior abnormality degree evaluation method described above.

[0112] Further, the present disclosure also provides a user behavior abnormality degree evaluation device.

[0113] FIG. 2 is a schematic diagram of the architecture of the user behavior abnormality degree evaluation device provided by the present disclosure, as shown in FIG. 2, in one embodiment, the user behavior abnormality degree evaluation device includes:

[0114] The camera synchronous acquisition module, the two-dimensional skeleton key point detection module, the time sequence action positioning module, the multi-view camera calibration module, the skeleton three-dimensional coordinate conversion module, the action recognition module and the action quantitative analysis module, the algorithm automatically analyzes and processes the synchronous video data input by multiple cameras, and outputs the infant behavior feature analysis result in this period of time, including the detected action number, action category, action occurrence start and end time and duration, three-dimensional coordinate change relationship of skeleton joint points in the action occurrence process, etc., all features are used as input of the autism child diagnosis model.

[0115] Among them, the camera synchronous acquisition module includes camera 1, camera 2, camera 3 and camera 4, the time sequence action positioning module is configured to perform video feature extraction and time interval extraction, the offline multi-view camera calibration module is configured to perform camera intrinsic parameter calibration and camera extrinsic parameter calibration, the two-dimensional skeleton key point detection module is configured to perform human body detection and 17 key point detection, the skeleton three-dimensional coordinate conversion module is configured to perform three-dimensional skeleton determination, the action recognition module is configured to perform action classification, the action quantitative analysis module is configured to determine quantitative indicators, and the diagnosis module is configured to determine autism diagnosis results.

[0116] In one embodiment, the user behavior abnormality degree evaluation device includes: an acquisition module configured to acquire synchronous behavior videos of a user to be evaluated at at least two different views; an analysis module configured to perform behavior analysis of the user to be evaluated according to each synchronous behavior video, to obtain behavior quantitative indicators of the user to be evaluated; and an evaluation module configured to input the behavior quantitative indicators into a behavior abnormality degree evaluation model, to obtain abnormality degree information of the behavior of the user to be evaluated output by the behavior abnormality degree evaluation model; the behavior abnormality degree evaluation model is obtained by model training of sample behavior quantitative indicators and their abnormality degree labels.

[0117] In one embodiment, the analysis module is specifically configured to:

[0118] determine a target behavior video from the behavior videos;

[0119] perform skeleton key point detection on the target behavior video to obtain first coordinate information of each skeleton key point of the user to be evaluated in a coordinate system corresponding to the target behavior video;

[0120] perform behavior analysis based on the first coordinate information to obtain a behavior quantitative indicator of the user to be evaluated.

[0121] In one embodiment, the analysis module includes an analysis unit, which is configured to:

[0122] perform behavior recognition based on the first coordinate information to obtain behavior classification information of the user to be evaluated;

[0123] perform coordinate conversion based on the first coordinate information to obtain second coordinate information of each skeleton key point of the user to be evaluated in a world coordinate system;

[0124] determine a behavior quantitative indicator of the user to be evaluated according to the behavior classification information and the second coordinate information.

[0125] In one embodiment, the analysis module includes a first determination unit, which is configured to:

[0126] determine a quantitative indicator of the user to be evaluated based on each second coordinate in the second coordinate information and time information corresponding to each second coordinate in the target behavior video;

[0127] determine a behavior quantitative indicator of the user to be evaluated based on the behavior classification information and the quantitative indicator.

[0128] In one embodiment, the analysis module includes a conversion unit, which is configured to:

[0129] obtain intrinsic parameters of a camera corresponding to each perspective behavior video and extrinsic parameters relative to a world coordinate system;

[0130] perform coordinate conversion on a first coordinate in the first coordinate information based on the intrinsic parameters of each camera and the extrinsic parameters to obtain second coordinate information of each skeleton key point of the user to be evaluated in a world coordinate system.

[0131] In one embodiment, the analysis module includes a second determination unit, which is configured to:

[0132] perform occlusion detection on each behavior video to obtain an occlusion detection result;

[0133] According to the occlusion detection result, a target behavior video is determined from each synchronized behavior video.

[0134] In an embodiment, the analysis module comprises a third determination unit configured to:

[0135] determine a first comparison result between each occlusion degree in the occlusion detection result;

[0136] determine a second comparison result between each occlusion duration in the occlusion detection result;

[0137] According to the first comparison result and the second comparison result, a target behavior video is determined from each synchronized behavior video.

[0138] The user behavior abnormality evaluation device provided by the embodiments of the present disclosure can collect synchronized behavior videos of a user to be evaluated at at least two different angles, and perform behavior analysis on the user to be evaluated according to each synchronized behavior video to obtain behavior quantitative indicators of the user to be evaluated. Thus, the behavior quantitative indicators can be input into a behavior abnormality evaluation model. Since the behavior abnormality evaluation model is obtained by model training based on sample behavior quantitative indicators and abnormality degree labels, the behavior abnormality evaluation model can output abnormality degree information of the behavior of the user to be evaluated. Since synchronized behavior videos at multiple angles are collected and the multiple synchronized behavior videos are analyzed, the differences between the same actions can be distinguished, and the analysis results are comprehensive. Thus, behavior quantitative indicators of the user in a long time range can be obtained, and the abnormality degree evaluation model obtained by model training based on sample behavior quantitative indicators and abnormality degree labels can accurately obtain the abnormality degree information of the behavior of the user to be evaluated, thereby improving the accuracy of the abnormality degree evaluation of the user behavior.

[0139] FIG. 3 illustrates an entity structure diagram of an electronic device. As shown in FIG. 3, the electronic device can include a processor 310, a communications interface 320, a memory 330, and a communications bus 340. The processor 310, the communications interface 320, and the memory 330 can communicate with each other through the communications bus 340. The processor 310 can invoke logical instructions in the memory 330 to execute a user behavior abnormality evaluation method, which includes collecting synchronized behavior videos of a user to be evaluated at at least two different angles;

[0140] performing behavior analysis on the user to be evaluated according to each synchronized behavior video to obtain behavior quantitative indicators of the user to be evaluated;

[0141] The behavior quantitative index is input into a behavior abnormality degree evaluation model to obtain abnormality degree information of the behavior of the user to be evaluated output by the behavior abnormality degree evaluation model.

[0142] In addition, the logical instructions in the memory 330 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present disclosure essentially or the part that contributes to the related art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present disclosure. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0143] In yet another aspect, the present disclosure also provides a non-transitory computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement a user behavior abnormality degree evaluation method provided by the above-mentioned method, the method comprising: collecting synchronous behavior videos of a user to be evaluated at at least two different angles;

[0144] performing behavior analysis on the user to be evaluated according to each synchronous behavior video to obtain behavior quantitative indexes of the user to be evaluated;

[0145] inputting the behavior quantitative indexes into a behavior abnormality degree evaluation model to obtain abnormality degree information of the behavior of the user to be evaluated output by the behavior abnormality degree evaluation model; the behavior abnormality degree evaluation model is obtained by model training of sample behavior quantitative indexes and their abnormality degree labels.

[0146] The device embodiments described above are only schematic, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0147] Those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary universal hardware platform, and of course can also be implemented by hardware, based on the description of the foregoing embodiments. Based on such understanding, the foregoing technical solutions can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions to cause a computer device (which can be a personal computer, a server, or a network device, and the like) to execute the methods described in each embodiment or some parts of the embodiments.

[0148] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present disclosure, and not to limit them; although the foregoing embodiments of the present disclosure have been described in detail, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features thereof; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure. Industrial applicability

[0149] By using the above scheme, the synchronous behavior videos of the user to be evaluated at least two different angles are collected; and the behavior analysis of the user to be evaluated is performed according to each synchronous behavior video to obtain the behavior quantitative index of the user to be evaluated; thus, the behavior quantitative index can be input into the behavior abnormality degree evaluation model, and since the behavior abnormality degree evaluation model is obtained by model training of sample behavior quantitative indexes and their abnormality degree labels, the abnormality degree information of the behavior of the user to be evaluated output by the behavior abnormality degree evaluation model can be obtained. Since the synchronous behavior videos at multiple angles are collected, the multiple synchronous behavior videos are analyzed for behavior, the differences between the same actions can be distinguished, the analysis result is comprehensive, the behavior quantitative index of the user in a long time range is obtained, and then the abnormality degree evaluation model obtained by model training of sample behavior quantitative indexes and their abnormality degree labels can be used to accurately obtain the abnormality degree information of the behavior of the user to be evaluated, and the accuracy of the abnormality degree evaluation of the user behavior can be improved.

Claims

1. A method of evaluating an abnormality degree of user behavior, characterized by, The method comprises: collecting synchronous behavior videos of a to-be-evaluated user at at least two different perspectives; performing behavior analysis on the to-be-evaluated user according to each synchronous behavior video to obtain a behavior quantitative indicator of the to-be-evaluated user; inputting the behavior quantitative indicator into a behavior abnormality degree evaluation model to obtain abnormality degree information of the behavior of the to-be-evaluated user output by the behavior abnormality degree evaluation model; the behavior abnormality degree evaluation model is obtained by model training based on sample behavior quantitative indicators and abnormality degree labels thereof.

2. The user behavior abnormality degree evaluation method according to claim 1, characterized by, The method comprises: determining a target behavior video from each synchronous behavior video; performing skeleton key point detection on the target behavior video to obtain first coordinate information of each skeleton key point of the to-be-evaluated user in a coordinate system corresponding to the target behavior video; performing behavior analysis based on the first coordinate information to obtain the behavior quantitative indicator of the to-be-evaluated user.

3. The user behavior abnormality degree evaluation method according to claim 2, characterized by, The method comprises: performing behavior recognition based on the first coordinate information to obtain behavior classification information of the to-be-evaluated user; performing coordinate conversion based on the first coordinate information to obtain second coordinate information of each skeleton key point of the to-be-evaluated user in a world coordinate system; determining the behavior quantitative indicator of the to-be-evaluated user according to the behavior classification information and the second coordinate information.

4. The user behavior abnormality degree evaluation method according to claim 3, characterized by, The method comprises: determining a quantitative indicator of the to-be-evaluated user based on each second coordinate in the second coordinate information and time information corresponding to each second coordinate in the target behavior video; determining the behavior quantitative indicator of the to-be-evaluated user based on the behavior classification information and the quantitative indicator.

5. The user behavior abnormality degree evaluation method according to claim 3, characterized by, The method comprises: obtaining intrinsic parameters of cameras corresponding to synchronous behavior videos of each perspective and extrinsic parameters relative to a world coordinate system; performing coordinate conversion on a first coordinate in the first coordinate information based on the intrinsic parameters of each camera and the extrinsic parameters to obtain second coordinate information of each skeleton key point of the to-be-evaluated user in the world coordinate system.

6. The user behavior abnormality degree evaluation method according to claim 2, characterized by, The method comprises: performing occlusion detection on each synchronous behavior video to obtain an occlusion detection result; determining a target behavior video from each synchronous behavior video according to the occlusion detection result.

7. The user behavior abnormality degree evaluation method according to claim 6, characterized by, The occlusion detection result comprises an occlusion degree and an occlusion duration. The method comprises: determining a first comparison result between each occlusion degree in the occlusion detection result; determining a second comparison result between each occlusion duration in the occlusion detection result; determining a target behavior video from each synchronous behavior video according to the first comparison result and the second comparison result.

8. An abnormality degree evaluation device of user behavior, characterized by comprising: The method comprises: A collection module configured to collect a video of synchronized behavior of a user to be evaluated at at least two different angles of view; An analysis module configured to perform behavior analysis of the user to be evaluated according to each video of synchronized behavior, to obtain a behavior quantification index of the user to be evaluated; An evaluation module configured to input the behavior quantification index into a behavior abnormality degree evaluation model, to obtain abnormality degree information of behavior of the user to be evaluated output by the behavior abnormality degree evaluation model; The behavior abnormality degree evaluation model is obtained by model training using sample behavior quantification indexes and abnormality degree labels thereof.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the user behavior abnormality degree evaluation method according to any one of claims 1 to 7 when executing the program.

10. A medium, which is a non-transitory computer-readable storage medium, having stored thereon a computer program, characterized by, The computer program implements the user behavior abnormality degree evaluation method according to any one of claims 1 to 7 when executed by the processor.

Images