Edge end multi-mode abnormal behavior detection method and system

Abstract

The invention relates to an edge end multi-mode abnormal behavior detection method and system, and relates to the technical field of abnormal behavior detection, and the method comprises the steps: collecting image detection information and sound detection information; generating image behavior features based on the image detection information; generating behavior reference sound information and image detection result information based on the image behavior characteristics; generating sound detection result information based on the sound detection information and the behavior reference sound information; and determining abnormal behavior detection result information based on the image detection result information and the sound detection result information, and outputting the abnormal behavior detection result information. The method has the effect of improving the accuracy of the abnormal behavior detection result.

Description

Technical Field

[0001] The present invention relates to the technical field of abnormal behavior detection, and in particular to an edge - side multimodal abnormal behavior detection method and system. Background Art

[0002] Abnormal behavior detection refers to the process of identifying abnormal activities that deviate significantly from normal behavior patterns through technologies such as data analysis, machine learning, or pattern recognition.

[0003] Currently, in the process of detecting abnormal behavior, generally, after collecting raw data such as images and videos through a camera, it is transmitted to the cloud or a central server in real - time or in batches through a network. Then, after pre - processing the raw data such as denoising, normalization, and object detection, general features are extracted from a large amount of non - abnormal image data as normal samples, and then the collected images or videos are matched with the normal samples to detect abnormal behavior.

[0004] Since currently, when detecting abnormal behavior, after collecting images and videos through a camera, generally, the behavior features in the images are analyzed to obtain the detection results of abnormal behavior, and the same behavior feature may have different meanings in different scenarios, which easily leads to errors in the final detected abnormal behavior detection results. Summary of the Invention

[0005] In order to improve the accuracy of abnormal behavior detection results, the present invention provides an edge - side multimodal abnormal behavior detection method and system.

[0006] In a first aspect, the present invention provides an edge - side multimodal abnormal behavior detection method, adopting the following technical solutions: An edge - side multimodal abnormal behavior detection method includes: S1: Collect image detection information and sound detection information; S2: Generate image behavior features based on the image detection information; S3: Generate behavior - reference sound information and image detection result information based on the image behavior features; S4: Generate sound detection result information based on the sound detection information and the behavior - reference sound information; S5: Determine abnormal behavior detection result information based on the image detection result information and the sound detection result information, and output the abnormal behavior detection result information.

[0007] Optionally, the method for generating the image behavior features includes: S21: Retrieve image detection position points and detection time points based on the image detection information; S22: Determine the behavior reference image information based on the image detection position points; S23: Determine the image similarity information based on the image detection information and the behavior reference image information; S24: Retrieve the similar position points and the similarity degree values based on the image similarity information; S25: Generate a similarity reference value based on the detection time point, the similar position points, and the similarity degree values; S26: Generate the selected reference image information based on the similarity reference value; S27: Determine the similar behavior features based on the selected reference image information, and use the similar behavior features as the image behavior features.

[0008] Optionally, the method for generating the similarity reference value includes: S251: Determine the number of position values based on the similar position points; S252: Determine the similarity average value based on the similarity degree value and the number of position values; S253: Determine the time reference number value based on the detection time point; S254: Determine whether the number of position values is greater than the time reference number value; S255: If yes, determine the average reference value based on the similarity average value, and use the average reference value as the similarity reference value; S256: If no, calculate the difference between the number of position values and the time reference number value and use it as the number deviation value; S257: Determine the number deviation reference value based on the number deviation value and the similarity average value, and use the number deviation reference value as the similarity reference value.

[0009] Optionally, the method for generating the selected reference image information includes: S261: Determine whether there is a similarity reference value greater than a preset similarity reference value; S262: If yes, use the similarity reference value corresponding to the value greater than the preset similarity reference value as the selection reference value, and use the behavior reference image information corresponding to the selection reference value as the selected initial image information; S263: Determine the relevant time point of the initial image based on the selected initial image information; S264: Calculate the difference between the relevant time point of the initial image and the detection time point and use it as the interval time value; S265: Sort the interval time values from smallest to largest, and use the selected initial image information corresponding to the interval time value ranked first as the selected reference image information; S266: If the answer is no, then sort in descending order based on the similarity reference value, and use the behavior reference image information corresponding to the similarity reference value ranked first as the selected reference image information.

[0010] Optionally, the method for generating the behavior reference sound information and the image detection result information includes: S31: Determine the detection position reference behavior feature based on the image detection position point; S32: Determine the detection time reference behavior feature based on the detection time point; S33: Determine the detection comprehensive reference behavior feature based on the detection position reference behavior feature and the detection time reference behavior feature; S34: Determine whether the image behavior feature belongs to the detection comprehensive reference behavior feature; S35: If the answer is yes, then generate the image reference sound information based on the image behavior feature and the detection comprehensive reference behavior feature, use the image reference sound information as the behavior reference sound information, and output the preset normal result information of the image behavior as the image detection result information; S36: If the answer is no, then determine the abnormal reference sound information based on the image behavior feature, and use the abnormal reference sound information as the behavior reference sound information; S37: Determine the abnormal result information of the image behavior based on the image behavior feature and the detection comprehensive reference behavior feature, and use the abnormal result information of the image behavior as the image detection result information.

[0011] Optionally, the method for generating the image reference sound information includes: S351: Determine the comprehensive reference sound information based on the detection comprehensive reference behavior feature; S352: Determine the behavior position prediction condition information based on the image detection position point; S353: Determine the prediction condition satisfaction value based on the behavior position prediction condition information and the image detection information; S354: Determine whether the prediction condition satisfaction value is greater than the preset prediction condition reference value; S355: If the answer is yes, then generate the single behavior associated sound information based on the image behavior feature and the prediction condition satisfaction value, and combine the comprehensive reference sound information and the single behavior associated sound information as the image reference sound information; S356: If the answer is no, then use the comprehensive reference sound information as the image reference sound information.

[0012] Optionally, the method for generating the single-action associated sound information includes: S3551: Calculate the difference between the estimated condition satisfaction value and a preset estimated condition reference value as the condition satisfaction deviation value; S3552: Determine the associated selection count value based on the condition satisfaction deviation value; S3553: Determine the associated behavior feature and the associated degree value based on the image behavior feature; S3554: Sort in descending order based on the associated degree value, and select the associated behavior feature corresponding to the sorting result of the associated selection count value as the selected behavior feature; S3555: Determine the associated selection sound information based on the selected behavior feature; S3556: Combine based on the associated selection sound information to form the single-action associated sound information.

[0013] Optionally, the method for generating the sound detection result information includes: S41: Determine the sound detection text information and the sound detection intensity value based on the sound detection information; S42: Determine the reference sound text information and the reference sound intensity value based on the behavior reference sound information; S43: Determine the text coincidence information based on the sound detection text information and the reference sound text information; S44: Determine the sound intensity deviation value based on the sound detection intensity value and the reference sound intensity value; S45: Generate the coincidence intensity comprehensive result information based on the text coincidence information and the sound intensity deviation value, and use the coincidence intensity comprehensive result information as the sound detection result information.

[0014] Optionally, the method for generating the coincidence intensity comprehensive result information includes: S451: Retrieve the text coincidence rate and the text coincidence position based on the text coincidence information; S452: Determine the position requirement coincidence interval based on the text coincidence position; S453: Determine the intensity influence value based on the sound intensity deviation value; S454: Determine the requirement adjustment coincidence interval based on the intensity influence value and the position requirement coincidence interval; S455: Determine the coincidence adjustment result information based on the text coincidence rate and the requirement adjustment coincidence interval, and use the coincidence adjustment result information as the coincidence intensity comprehensive result information.

[0015] In a second aspect, the present invention provides an edge - side multi - modal abnormal behavior detection system, adopting the following technical solutions: An edge - side multi - modal abnormal behavior detection system, comprising: An acquisition module, configured to acquire image detection information and sound detection information; A memory, storing a program for implementing an edge - side multi - modal abnormal behavior detection method as described in any one of the first aspect; A processor, configured to load and execute the program stored in the memory.

[0016] In summary, the present invention includes at least one of the following beneficial technical effects: 1. By collecting image detection information and sound detection information, analyzing and generating image detection result information and sound detection result information to determine abnormal behavior detection result information and output it, thereby reducing misjudgment or missed judgment existing in pure image recognition through sound detection and analysis, and further improving the accuracy of abnormal behavior detection results; 2. By retrieving the image detection position point and detection time point through the image detection information, determining the behavior reference image information through the image detection position point, thereby determining the image similarity information and retrieving the similar position point and similarity degree value, generating a similarity reference value through the detection time point, similar position point and similarity degree value, and generating a selected reference image information through the similarity reference value to determine the similar behavior characteristics and use them as image behavior characteristics, thereby improving the accuracy of the obtained image behavior characteristics; 3. By determining the detection comprehensive reference behavior characteristics through the image detection position point and detection time point, and when the image behavior characteristics belong to the detection comprehensive reference behavior characteristics, generating image reference sound information through the image behavior characteristics and the detection comprehensive reference behavior characteristics and using it as the behavior reference sound information, and outputting the preset normal result information of the image behavior as the image detection result information. When it does not belong, determining the abnormal reference sound information through the image behavior characteristics and using it as the behavior reference sound information, and determining the abnormal result information of the image behavior through the image behavior characteristics and the detection comprehensive reference behavior characteristics and using it as the image detection result information, thereby improving the accuracy of the obtained behavior reference sound information and image detection result information. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is the flowchart of the method for edge - side multi - modal abnormal behavior detection according to an embodiment of the present invention; Figure 2 is the flowchart of the method for generating image behavior characteristics according to an embodiment of the present invention; Figure 3 is the flowchart of the method for generating similarity reference values according to an embodiment of the present invention; Figure 4It is a flowchart of a method for generating reference image information in an embodiment of the present invention; Figure 5 It is a flowchart of a method for generating behavioral reference sound information and image detection result information in an embodiment of the present invention. Detailed implementation manners

[0018] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

[0019] An edge - side multi - modal abnormal behavior detection method collects image detection information and sound detection information, and performs comprehensive analysis based on the collected location and time, thereby generating image detection result information and sound detection result information to determine and output the abnormal behavior detection result information, so as to reduce misjudgment or missed judgment in pure image recognition through sound detection and analysis, and further improve the accuracy of the abnormal behavior detection result.

[0020] Refer to Figure 1 , an embodiment of the present invention discloses an edge - side multi - modal abnormal behavior detection method, which includes: S1: Collect image detection information and sound detection information.

[0021] Among them, the image detection information refers to the detected image information, and the sound detection information refers to the detected sound information.

[0022] The image detection information and the sound detection information are obtained through a preset detection terminal. The detection terminal includes a camera for detecting images and a sound sensor for detecting sounds, and the detection terminal is pre - installed at the location where abnormal behavior detection is required.

[0023] S2: Generate image behavior features based on the image detection information.

[0024] Among them, the image behavior features refer to the features corresponding to the action behaviors of the people in the detected images. By analyzing the image detection information, image behavior features are generated for convenient subsequent use. The specific generation steps of the image behavior features refer to S21 to S27.

[0025] S3: Generate behavioral reference sound information and image detection result information based on the image behavior features.

[0026] Among them, the behavioral reference sound information refers to the reference sound information normally emitted when a person performs an action, and the image detection result information refers to the result information corresponding to the detection of abnormal behaviors of the people in the detected images.

[0027] By analyzing the image behavior features, behavioral reference sound information and image detection result information are generated for convenient subsequent use.

[0028] The specific generation steps of the behavior-based sound information and the image detection result information refer to S31 to S37.

[0029] S4: Generate sound detection result information based on the sound detection information and the behavior-based sound information.

[0030] The sound detection result information refers to the result information corresponding to the abnormal behavior detection based on the collected sound data.

[0031] By analyzing the sound detection information and the behavior-based sound information, the sound detection result information is generated, which is convenient for subsequent use.

[0032] The specific generation steps of the sound detection result information refer to S41 to S45.

[0033] S5: Determine the abnormal behavior detection result information based on the image detection result information and the sound detection result information, and output the abnormal behavior detection result information.

[0034] The abnormal behavior detection result information refers to the comprehensive result information corresponding to the abnormal behavior detection.

[0035] By combining the image detection result information and the sound detection result information as the abnormal behavior detection result information and outputting it, the false judgment or missed judgment in the simple image recognition is reduced through sound detection and analysis, thereby improving the accuracy of the abnormal behavior detection result.

[0036] For example, when the image detection result information is abnormal and the sound detection result information is also abnormal, the abnormal behavior detection result information includes the abnormal situation of the image and the abnormal situation of the sound at this time. When the image detection result information is abnormal and the sound detection result information is not abnormal, the abnormal behavior detection result information is the abnormal situation of the image at this time. When the image detection result information is not abnormal and the sound detection result information is abnormal, the abnormal behavior detection result information is the abnormal situation of the sound at this time.

[0037] In step S2, in order to further ensure the rationality of the image behavior characteristics, it is necessary to perform further separate analysis and calculation on the image behavior characteristics, which are specifically described in detail through the following steps.

[0038] Refer to Figure 2 , the generation method of the image behavior characteristics includes the following steps: S21: Retrieve the image detection position points and detection time points based on the image detection information.

[0039] Among them, the image detection position point refers to the position point where image detection is performed, and the image detection position point is detected and obtained through a position sensor preset on the detection terminal. The detection time point refers to the time point when image detection is performed, and the detection time point is obtained by querying a time database, which stores time in real time.

[0040] S22: Determine the behavior reference image information based on the image detection position point.

[0041] Among them, the behavior reference image information refers to the reference image information corresponding to the behavior when the person at the detection position passes through the image detection position point. Different image detection position points correspond to different behavior reference image information.

[0042] By inputting the image detection position point into a preset behavior reference image database to match and obtain the behavior reference image information, it is convenient for subsequent use.

[0043] The behavior reference image database pre-stores a comparison table of different image detection position points and the corresponding behavior reference image information, and the behavior reference image database is obtained through pre-input.

[0044] S23: Determine the image similarity information based on the image detection information and the behavior reference image information.

[0045] Among them, the image similarity information refers to the position in the image and the similarity situation information when the images are similar.

[0046] By comparing the image detection information with the behavior reference image information, and combining the positions where there is similarity and the situation of the similar positions accounting for the overall position of the image in the image detection information as the image similarity information, it is convenient for subsequent use.

[0047] S24: Retrieve the similar position points and the similarity degree values based on the image similarity information.

[0048] Among them, the similar position points refer to the position points in the image when the images are similar, and the similarity degree value refers to the similarity situation in the image when the images are similar. The image similarity information includes the similar position points and the similarity degree values.

[0049] Retrieve the similar position points and the similarity degree values through the image similarity information, which is convenient for subsequent use.

[0050] S25: Generate a similarity reference value based on the detection time point, the similar position points, and the similarity degree values.

[0051] Among them, the similarity reference value refers to the reference value corresponding to the selection of the reference image according to the similarity situation.

[0052] By analyzing the detection time points, similar position points, and similarity degree values, a similar reference value is generated for convenient subsequent use. The specific generation steps of the similar reference value refer to S251 to S257.

[0053] S26: Generate selection reference image information based on the similar reference value.

[0054] Among them, the selection reference image information refers to the image information corresponding to the selection of the reference image.

[0055] By analyzing the similar reference value, selection reference image information is generated for convenient subsequent use. The specific generation steps of the selection reference image information refer to S261 to S266.

[0056] S27: Determine the similar behavior characteristics based on the selection reference image information and use the similar behavior characteristics as the image behavior characteristics.

[0057] Among them, the similar behavior characteristics refer to the behavior characteristics corresponding to the selected reference image. Different selection reference image information corresponds to different similar behavior characteristics.

[0058] By inputting the selection reference image information into a preset selection reference image database to match and obtain the similar behavior characteristics, and using the similar behavior characteristics as the image behavior characteristics, the accuracy of the obtained image behavior characteristics is improved.

[0059] The selection reference image database pre-stores a comparison table of different selection reference image information and the corresponding similar behavior characteristics, and the selection reference image database is obtained through pre-input.

[0060] In step S25, in order to further ensure the rationality of the similar reference value, further separate analysis and calculation of the similar reference value are required, which is specifically described in detail through the following steps.

[0061] Refer to Figure 3 , the generation method of the similar reference value includes the following steps: S251: Determine the position count value based on the similar position points.

[0062] Among them, the position count value refers to the count value corresponding to the similar position points.

[0063] By counting the similar position points and using the counting result as the position count value for convenient subsequent use.

[0064] S252: Determine the similar average value based on the similarity degree value and the position count value.

[0065] Among them, the similar average value refers to the average value of the similarity degree values corresponding to the number of the position count value.

[0066] By calculating the sum value of the similarity degree values, and then calculating the quotient value between the sum value and the number of position values as the similarity average value, it is convenient for subsequent use.

[0067] S253: Determine the number of time base values based on the detection time point.

[0068] Among them, the number of time base values refers to the minimum number of similarity values that can be tolerated at the detection time point. Different detection time points correspond to different numbers of time base values.

[0069] By inputting the detection time point into the preset time base number database to match and obtain the number of time base values, it is convenient for subsequent use.

[0070] The time base number database pre-stores a comparison table of different detection time points and the corresponding number of time base values, and the time base number database is obtained through pre-input.

[0071] S254: Determine whether the number of position values is greater than the number of time base values. If it is, execute S255; if not, execute S256.

[0072] Among them, by judging whether the number of position values is greater than the number of time base values, it is thus judged whether the similarity average value can be directly adopted.

[0073] S255: Determine the average reference value based on the similarity average value, and use the average reference value as the similarity reference value.

[0074] Among them, the average reference value refers to the reference value for reference based on the similarity average value, and different similarity average values correspond to different average reference values.

[0075] When the number of position values is greater than the number of time base values, it means that the similarity average value can be directly adopted at this time. Therefore, by inputting the similarity average value into the preset average reference database to match and obtain the average reference value, and using the average reference value as the similarity reference value, the accuracy of the obtained similarity reference value is improved.

[0076] The average reference database pre-stores a comparison table of different similarity average values and the corresponding average reference values, and the average reference database is obtained through pre-input.

[0077] S256: Calculate the difference between the number of position values and the number of time base values as the number deviation value.

[0078] Among them, the number deviation value refers to the deviation value corresponding to the deviation of the number.

[0079] When the value of the position quantity is not greater than the value of the time reference quantity, it indicates that the similar average value cannot be directly adopted at this time. Therefore, the difference between the value of the position quantity and the value of the time reference quantity is calculated and used as the quantity deviation value for subsequent use.

[0080] S257: Determine the quantity deviation reference value based on the quantity deviation value and the similar average value, and use the quantity deviation reference value as the similar reference value.

[0081] Among them, the quantity deviation reference value refers to the reference value corresponding to the reference after adjusting the similar average value according to the quantity deviation.

[0082] By inputting the quantity deviation value into the preset quantity deviation influence database to match and obtain the quantity deviation influence value, calculating the product value between the quantity deviation influence value and the similar average value as the average adjustment value, then inputting the average adjustment value into the preset quantity deviation reference database to match and obtain the quantity deviation reference value, and using the quantity deviation reference value as the similar reference value, the accuracy of the obtained similar reference value is improved.

[0083] In step S26, in order to further ensure the rationality of selecting the reference image information, it is necessary to perform further separate analysis and calculation on the selected reference image information, which is specifically described in detail through the following steps.

[0084] Refer to Figure 4 , the generation method of the reference image information includes the following steps: S261: Determine whether there is a similar reference value greater than the preset similar reference benchmark value. If yes, execute S262; if no, execute S266.

[0085] Among them, the similar reference benchmark value refers to the minimum reference value allowed during selection, and the similar reference benchmark value is obtained through pre-input.

[0086] By judging whether there is a similar reference value greater than the preset similar reference benchmark value, it is judged whether it is necessary to select the similar reference value and then make a further selection.

[0087] S262: Use the similar reference value corresponding to the value greater than the preset similar reference benchmark value as the selection reference value, and use the behavior reference image information corresponding to the selection reference value as the selection initial image information.

[0088] Among them, when there is a similar reference value greater than the preset similar reference benchmark value, it indicates that it is necessary to select the similar reference value and then make a further selection at this time. Therefore, the selection reference value and the selection initial image information are defined respectively for subsequent use.

[0089] S263: Determine the initial image related time point based on the selected initial image information.

[0090] Among them, the initial image related time point refers to the time point corresponding to the existence of a correlation in the selected initial image information. Different selected initial image information corresponds to different initial image related time points.

[0091] By inputting the selected initial image information into a preset initial image related time database to match and obtain the initial image related time point, it is convenient for subsequent use.

[0092] The initial image related time database pre-stores a comparison table of different selected initial image information and the corresponding initial image related time points. The initial image related time database is obtained through pre-input.

[0093] S264: Calculate the difference between the initial image related time point and the detection time point and use it as the interval time value.

[0094] Among them, the interval time value refers to the interval time between the initial image related time point and the detection time point.

[0095] By calculating the difference between the initial image related time point and the detection time point and using it as the interval time value, it is convenient for subsequent use.

[0096] S265: Sort the interval time values from smallest to largest, and use the selected initial image information corresponding to the interval time value ranked first as the selected reference image information.

[0097] Among them, by sorting the interval time values from smallest to largest and using the selected initial image information corresponding to the interval time value ranked first as the selected reference image information, the accuracy of the obtained selected reference image information is improved.

[0098] S266: Sort the similarity reference values from largest to smallest, and use the behavior reference image information corresponding to the similarity reference value ranked first as the selected reference image information.

[0099] Among them, when there is no similarity reference value greater than the preset similarity reference value, it means that there is no need to select the similarity reference value and then further select at this time. Therefore, by sorting the similarity reference values from largest to smallest and using the behavior reference image information corresponding to the similarity reference value ranked first as the selected reference image information, the accuracy of the obtained selected reference image information is improved.

[0100] In step S3, in order to further ensure the rationality of the behavior benchmark sound information and the image detection result information, it is necessary to perform further separate analysis and calculation on the behavior benchmark sound information and the image detection result information, which will be specifically described in detail through the following steps.

[0101] Referring to Figure 5 , the generation method of the behavior benchmark sound information and the image detection result information includes the following steps: S31: Determine the detection position reference behavior feature based on the image detection position point.

[0102] Among them, the detection position reference behavior feature refers to the reference behavior feature corresponding to the action of the person at the image detection position point. Different image detection position points correspond to different detection position reference behavior features.

[0103] By inputting the image detection position point into the preset detection position reference database to match and obtain the detection position reference behavior feature, it is convenient for subsequent use.

[0104] The detection position reference database pre-stores a comparison table of different image detection position points and the corresponding detection position reference behavior features, and the detection position reference database is obtained through pre-input.

[0105] S32: Determine the detection time reference behavior feature based on the detection time point.

[0106] Among them, the detection time reference behavior feature refers to the reference behavior feature corresponding to the action of the person at the detection time point. Different detection time points correspond to different detection time reference behavior features.

[0107] By inputting the detection time point into the preset detection time reference database to match and obtain the detection time reference behavior feature, it is convenient for subsequent use.

[0108] The detection time reference database pre-stores different detection time points and the corresponding detection time reference behavior features, and the detection time reference database is obtained through pre-input.

[0109] S33: Determine the detection comprehensive reference behavior feature based on the detection position reference behavior feature and the detection time reference behavior feature.

[0110] Among them, the detection comprehensive reference behavior feature refers to the reference behavior feature corresponding to the action of the person at the detection time point and the image detection position point.

[0111] By comparing the detection position reference behavior feature and the detection time reference behavior feature, and taking the overlapping reference behavior features as the detection comprehensive reference behavior feature, it is convenient for subsequent use.

[0112] S34: Determine whether the image behavior feature belongs to the detection comprehensive reference behavior feature. If yes, execute S35; if no, execute S36.

[0113] Among them, by judging whether the image behavior feature belongs to the detection comprehensive reference behavior feature, it is thus judged whether there is an abnormality in the human behavior in the image.

[0114] S35: Generate image reference sound information based on the image behavior feature and the detection comprehensive reference behavior feature, and use the image reference sound information as the behavior reference sound information, and output the preset normal image behavior result information as the image detection result information.

[0115] Among them, the image reference sound information refers to the reference sound information corresponding to the normal human behavior in the image. The normal image behavior result information refers to the result information corresponding to the normal human behavior in the image, and the normal image behavior result information is obtained through pre-input.

[0116] When the image behavior feature belongs to the detection comprehensive reference behavior feature, it indicates that there is no abnormality in the human behavior in the image at this time. Therefore, by analyzing the image behavior feature and the detection comprehensive reference behavior feature, the image reference sound information is generated, and the image reference sound information is used as the behavior reference sound information, and the preset normal image behavior result information is output as the image detection result information, so as to improve the accuracy of the obtained behavior reference sound information and the image detection result information.

[0117] S36: Determine the abnormal reference sound information based on the image behavior feature, and use the abnormal reference sound information as the behavior reference sound information.

[0118] Among them, the abnormal reference sound information refers to the reference sound information corresponding to the abnormal human behavior in the image. Different image behavior features correspond to different abnormal reference sound information.

[0119] When the image behavior feature does not belong to the detection comprehensive reference behavior feature, it indicates that there is an abnormality in the human behavior in the image at this time. Therefore, by inputting the image behavior feature into the preset abnormal reference sound database to match and obtain the abnormal reference sound information, and using the abnormal reference sound information as the behavior reference sound information, the accuracy of the obtained behavior reference sound information is improved.

[0120] The abnormal reference sound database pre-stores a comparison table of image behavior features and the corresponding abnormal reference sound information, and the abnormal reference sound database is obtained through pre-input.

[0121] S37: Determine the image behavior anomaly result information based on the image behavior features and the detection comprehensive reference behavior features, and use the image behavior anomaly result information as the image detection result information.

[0122] Among them, the image behavior anomaly result information refers to the result information corresponding to the abnormal behavior of the person in the image.

[0123] By comparing the image behavior features with the detection comprehensive reference behavior features, and taking the behavior features with differences as the image behavior anomaly result information, and using the image behavior anomaly result information as the image detection result information, the accuracy of the obtained image detection result information can be improved.

[0124] In step S35, in order to further ensure the rationality of the image reference sound information, it is necessary to perform a further separate analysis and calculation on the image reference sound information, which is specifically described in detail through the following steps.

[0125] The generation method of the image reference sound information includes the following steps: S351: Determine the comprehensive reference sound information based on the detection comprehensive reference behavior features.

[0126] Among them, the comprehensive reference sound information refers to the reference sound information corresponding to the actions of the person at the detection time point and the image detection position point. Different detection comprehensive reference behavior features correspond to different comprehensive reference sound information.

[0127] By inputting the detection comprehensive reference behavior features into the preset comprehensive reference sound database to match and obtain the comprehensive reference sound information, it is convenient for subsequent use.

[0128] The comprehensive reference sound database pre-stores a comparison table of different detection comprehensive reference behavior features and the corresponding comprehensive reference sound information, and the comprehensive reference sound database is obtained through pre-input.

[0129] S352: Determine the behavior position prediction condition information based on the image detection position point.

[0130] Among them, the behavior position prediction condition information refers to the condition feature information corresponding to the abnormal behavior predicted when the person performs an action at the image detection position point. Different image detection position points correspond to different behavior position prediction condition information.

[0131] By inputting the image detection position point into the preset behavior position prediction condition database to match and obtain the behavior position prediction condition information, it is convenient for subsequent use.

[0132] The behavior position estimation condition database pre-stores a comparison table of different image detection position points and the corresponding behavior position estimation condition information, and the behavior position estimation condition database is obtained through pre-input.

[0133] S353: Determine the estimation condition satisfaction value based on the behavior position estimation condition information and the image detection information.

[0134] Among them, the estimation condition satisfaction value refers to the degree value corresponding to the case where there is a consistent situation between the detected image and the estimation condition feature.

[0135] By comparing the behavior position estimation condition information with the image detection information, the ratio of the features in the detected image that are compared to be consistent with the estimation condition features to the total features is used as the estimation condition satisfaction value, which is convenient for subsequent use.

[0136] S354: Determine whether the estimation condition satisfaction value is greater than the preset estimation condition reference value. If yes, execute S355; if no, execute S356.

[0137] Among them, the estimation condition reference value refers to the maximum reference value corresponding to directly adopting the comprehensive reference sound information, and the estimation condition reference value is obtained through pre-input.

[0138] By judging whether the estimation condition satisfaction value is greater than the preset estimation condition reference value, it is thus judged whether the comprehensive reference sound information can be directly adopted.

[0139] S355: Generate single-behavior associated sound information based on the image behavior feature and the estimation condition satisfaction value, and combine the comprehensive reference sound information with the single-behavior associated sound information and use it as the image reference sound information.

[0140] Among them, the single-behavior associated sound information refers to the associated sound information required for the behavior according to the estimation condition.

[0141] When the estimation condition satisfaction value is greater than the preset estimation condition reference value, it means that the comprehensive reference sound information cannot be directly adopted at this time. Therefore, by analyzing the image behavior feature and the estimation condition satisfaction value, single-behavior associated sound information is generated, and then the comprehensive reference sound information is combined with the single-behavior associated sound information and used as the image reference sound information, thereby improving the accuracy of the obtained image reference sound information.

[0142] The specific generation steps of the single-behavior associated sound information refer to S3551 to S3556.

[0143] S356: Use the comprehensive reference sound information as the image reference sound information.

[0144] Among them, when the estimated condition satisfaction value is not greater than the preset estimated condition reference value, it indicates that the comprehensive reference sound information can be directly adopted at this time. Therefore, the comprehensive reference sound information is used as the image reference sound information, thereby improving the accuracy of the obtained image reference sound information.

[0145] In step S355, in order to further ensure the rationality of the single-behavior associated sound information, it is necessary to perform further separate analysis and calculation on the single-behavior associated sound information, which will be specifically described in detail through the following steps.

[0146] The method for generating the single-behavior associated sound information includes the following steps: S3551: Calculate the difference between the estimated condition satisfaction value and the preset estimated condition reference value and use it as the condition satisfaction deviation value.

[0147] Among them, the condition satisfaction deviation value refers to the deviation value corresponding to the situation where there is a deviation in the condition satisfaction.

[0148] By calculating the difference between the estimated condition satisfaction value and the preset estimated condition reference value and using it as the condition satisfaction deviation value, it is convenient for subsequent use.

[0149] S3552: Determine the associated selection number value based on the condition satisfaction deviation value.

[0150] Among them, the associated selection number value refers to the number value corresponding to the selection. Different condition satisfaction deviation values correspond to different associated selection number values.

[0151] By inputting the condition satisfaction deviation value into the preset associated selection number database to match and obtain the associated selection number value, it is convenient for subsequent use.

[0152] The associated selection number database pre-stores a comparison table of different condition satisfaction deviation values and the corresponding associated selection number values, and the associated selection number database is obtained through pre-input.

[0153] S3553: Determine the associated behavior feature and the associated degree value based on the image behavior feature.

[0154] Among them, the associated behavior feature refers to the behavior feature corresponding to the sound in the associated situation, and the associated degree value refers to the degree value that needs to be associated when there is a sound in the associated situation. Different image behavior features correspond to different associated behavior features and associated degree values.

[0155] By inputting the image behavior feature into the preset associated behavior database to match and obtain the associated behavior feature and the associated degree value, it is convenient for subsequent use.

[0156] The associated behavior database pre-stores different image behavior features and their corresponding associated behavior features and associated degree values. The associated behavior database is obtained through pre-input.

[0157] S3554: Sort in descending order based on the associated degree value, and select the associated behavior feature corresponding to the sorting result of the associated selection number value as the selected behavior feature.

[0158] Among them, the selected behavior feature refers to the behavior feature corresponding after selection.

[0159] By sorting the associated degree values in descending order, selecting the same number and the top-ranked sorting results corresponding to the associated selection number value from the sorting results, and taking the associated behavior feature corresponding to the selected sorting result as the selected behavior feature, it is convenient for subsequent use.

[0160] S3555: Determine the associated selection sound information based on the selected behavior feature.

[0161] Among them, the associated selection sound information refers to the sound information that the selected associated behavior feature needs to be associated with. Different selected behavior features correspond to different associated selection sound information.

[0162] By inputting the selected behavior feature into the preset associated selection sound database to match and obtain the associated selection sound information, it is convenient for subsequent use.

[0163] The associated selection sound database pre-stores a comparison table of different selected behavior features and their corresponding associated selection sound information. The associated selection sound database is obtained through pre-input.

[0164] S3556: Combine based on the associated selection sound information to form a single behavior associated sound information.

[0165] Among them, by combining each associated selection sound information and taking the combined sound information as the single behavior associated sound information, the accuracy of the obtained single behavior associated sound information is improved.

[0166] In step S4, in order to further ensure the rationality of the sound detection result information, it is necessary to perform further separate analysis and calculation on the sound detection result information, which is specifically described in detail through the following steps.

[0167] The generation method of the sound detection result information includes the following steps: S41: Determine the sound detection text information and the sound detection intensity value based on the sound detection information.

[0168] Among them, the voice detection text information refers to the text information corresponding to the detected voice, and the voice detection intensity value refers to the intensity value corresponding to the detected voice.

[0169] By performing text recognition on the voice detection information, using the recognized text as the voice detection text information, and retrieving the voice intensity of the voice detection information as the voice detection intensity value, it is convenient for subsequent use.

[0170] S42: Determine the reference voice text information and the reference voice intensity value based on the behavior reference voice information.

[0171] Among them, the reference voice text information refers to the text information corresponding to the behavior under normal circumstances, and the reference voice intensity value refers to the intensity value corresponding to the behavior under normal circumstances. Different behavior reference voice information corresponds to different reference voice text information and reference voice intensity values.

[0172] By inputting the behavior reference voice information into a preset behavior reference voice database to match and obtain the reference voice text information and the reference voice intensity value, it is convenient for subsequent use.

[0173] The behavior reference voice database pre-stores different behavior reference voice information and the corresponding reference voice text information and reference voice intensity values, and the behavior reference voice database is obtained through pre-input.

[0174] S43: Determine the text overlap information based on the voice detection text information and the reference voice text information.

[0175] Among them, the text overlap information refers to the overlap situation information corresponding to the existence of text overlap.

[0176] By comparing the voice detection text information and the reference voice text information, and using the text situation with overlap after comparison as the text overlap information, it is convenient for subsequent use.

[0177] S44: Determine the voice intensity deviation value based on the voice detection intensity value and the reference voice intensity value.

[0178] Among them, the voice intensity deviation value refers to the deviation value corresponding to the existence of voice intensity deviation.

[0179] By calculating the difference between the voice detection intensity value and the reference voice intensity value and using it as the voice intensity deviation value, it is convenient for subsequent use.

[0180] S45: Generate the combined overlap intensity result information based on the text overlap information and the voice intensity deviation value, and use the combined overlap intensity result information as the voice detection result information.

[0181] Among them, the comprehensive result information of the coincidence intensity refers to the result information corresponding to the detection and comparison based on the coincidence situation of the text and the sound intensity.

[0182] By analyzing the text coincidence information and the deviation value of the sound intensity, the comprehensive result information of the coincidence intensity is generated, and the comprehensive result information of the coincidence intensity is used as the sound detection result information, so as to improve the accuracy of the obtained sound detection result information.

[0183] In step S45, in order to further ensure the rationality of the comprehensive result information of the coincidence intensity, it is necessary to perform further separate analysis and calculation on the comprehensive result information of the coincidence intensity, which is specifically described in detail through the following steps.

[0184] The generation method of the comprehensive result information of the coincidence intensity includes the following steps: S451: Retrieve the text coincidence rate and the text coincidence position based on the text coincidence information.

[0185] Among them, the text coincidence rate refers to the coincidence rate corresponding to the existence of text coincidence, and the text coincidence position refers to the position in the text where the text coincidence exists. The text coincidence information includes the text coincidence rate and the text coincidence position.

[0186] Retrieving the text coincidence rate and the text coincidence position through the text coincidence information facilitates subsequent use.

[0187] S452: Determine the position requirement coincidence interval based on the text coincidence position.

[0188] Among them, the position requirement coincidence interval refers to the initial interval where the coincidence rate is required according to the text position when there is coincidence. Different text coincidence positions correspond to different position requirement coincidence intervals.

[0189] By inputting the text coincidence position into the preset position requirement coincidence database to match and obtain the position requirement coincidence interval, it facilitates subsequent use.

[0190] The position requirement coincidence database pre-stores different text coincidence positions and the corresponding position requirement coincidence intervals, and the position requirement coincidence database is obtained through pre-input.

[0191] S453: Determine the intensity influence value based on the sound intensity deviation value.

[0192] Among them, the intensity influence value refers to the influence degree value corresponding to the influence of the intensity deviation on the interval where the coincidence rate is required.

[0193] Different sound intensity deviation values correspond to different intensity influence values. The intensity influence value is used to adjust the position requirement coincidence interval.

[0194] By inputting the sound intensity deviation value into a preset intensity influence database to obtain the intensity influence value, it is convenient for subsequent use.

[0195] The intensity influence database pre-stores a comparison table of different sound intensity deviation values and the corresponding intensity influence values, and the intensity influence database is obtained through pre-input.

[0196] S454: Determine the demand adjustment coincidence interval based on the coincidence interval of the intensity influence value and the position demand.

[0197] Among them, the demand adjustment coincidence interval refers to the corresponding interval after adjusting the position demand coincidence interval.

[0198] By calculating the product values of the two end values of the coincidence interval of the intensity influence value and the position demand, and using the product values corresponding to the two end values as the new interval end values to form the demand adjustment coincidence interval, it is convenient for subsequent use.

[0199] S455: Determine the coincidence adjustment result information based on the text coincidence rate and the demand adjustment coincidence interval, and use the coincidence adjustment result information as the coincidence intensity comprehensive result information.

[0200] Among them, the coincidence adjustment result information refers to the result information determined based on the demand adjustment coincidence interval.

[0201] By judging whether the text coincidence rate is within the demand adjustment coincidence interval, when the text coincidence rate is within the demand adjustment coincidence interval, the text coincidence rate is input into a preset text coincidence abnormal result database to obtain the text coincidence result information, and the text coincidence result information is used as the coincidence adjustment result information.

[0202] When the text coincidence rate is not within the demand adjustment coincidence interval, the preset text coincidence normal result information is output as the coincidence adjustment result information. Then, the coincidence adjustment result information is used as the coincidence intensity comprehensive result information, thereby improving the accuracy of the obtained coincidence intensity comprehensive result information.

[0203] The text coincidence abnormal result database pre-stores a comparison table of different text coincidence rates and the corresponding text coincidence result information, and the text coincidence abnormal result database is obtained through pre-input.

[0204] Based on the same inventive concept, an edge-side multimodal abnormal behavior detection system provided by an embodiment of the present invention includes: An acquisition module, configured to acquire image detection information and sound detection information; A memory, storing a program for implementing an edge-side multimodal abnormal behavior detection method as described above; A processor, loading and executing the program stored in the memory.

[0205] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be allocated to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. For the specific working processes of the system, device, and unit described above, reference can be made to the corresponding processes in the foregoing method embodiments, which will not be elaborated herein.

[0206] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the idea of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, several improvements and refinements made without departing from the principle of the present invention should also be regarded as within the protection scope of the present invention.

Claims

1. An edge-side multimodal abnormal behavior detection method, characterized in that, Including: S1: Collect image detection information and sound detection information; S2: Generate image behavior features based on the image detection information; S3: Generate behavior reference sound information and image detection result information based on the image behavior features; S4: Generate sound detection result information based on the sound detection information and the behavior reference sound information; S5: Determine abnormal behavior detection result information based on the image detection result information and the sound detection result information, and output the abnormal behavior detection result information.

2. The method for detecting multi-modal abnormal behaviors at the edge end according to claim 1, wherein, The method for generating the image behavior features includes: S21: Retrieve image detection position points and detection time points based on the image detection information; S22: Determine behavior reference image information based on the image detection position points; S23: Determine image similarity information based on the image detection information and the behavior reference image information; S24: Retrieve similar position points and similarity degree values based on the image similarity information; S25: Generate a similarity reference value based on the detection time point, the similar position points, and the similarity degree values; S26: Generate selected reference image information based on the similarity reference value; S27: Determine similar behavior features based on the selected reference image information, and use the similar behavior features as the image behavior features.

3. The edge - side multimodal abnormal behavior detection method according to claim 2, characterized in that, The method for generating the similarity reference value includes: S251: Determine the number of position values based on the similar position points; S252: Determine the similar average value based on the similarity degree value and the number of position values; S253: Determine the time reference number value based on the detection time point; S254: Determine whether the number of position values is greater than the time reference number value; S255: If yes, determine the average reference value based on the similar average value, and use the average reference value as the similarity reference value; S256: If no, calculate the difference between the number of position values and the time reference number value and use it as the number deviation value; S257: Determine the number deviation reference value based on the number deviation value and the similar average value, and use the number deviation reference value as the similarity reference value.

4. The edge - side multimodal abnormal behavior detection method according to claim 3, wherein, The method for generating the selected reference image information includes: S261: Determine whether there is a similarity reference value greater than a preset similarity reference value; S262: If yes, use the similarity reference value greater than the preset similarity reference value as the selection reference value, and use the behavior reference image information corresponding to the selection reference value as the selected initial image information; S263: Determine the initial image related time point based on the selected initial image information; S264: Calculate the difference between the initial image related time point and the detection time point and use it as the interval time value; S265: Sort the interval time values from smallest to largest, and use the selected initial image information corresponding to the first sorted interval time value as the selected reference image information; S266: If no, sort the similarity reference values from largest to smallest, and use the behavior reference image information corresponding to the first sorted similarity reference value as the selected reference image information.

5. The edge - side multimodal abnormal behavior detection method according to claim 3, wherein, The method for generating the behavior reference sound information and the image detection result information includes: S31: Determine the detection position reference behavior feature based on the image detection position point; S32: Determine the detection time reference behavior feature based on the detection time point; S33: Determine the detection comprehensive reference behavior feature based on the detection position reference behavior feature and the detection time reference behavior feature; S34: Determine whether the image behavior feature belongs to the detection comprehensive reference behavior feature; S35: If so, generate the image reference sound information based on the image behavior feature and the detection comprehensive reference behavior feature, use the image reference sound information as the behavior reference sound information, and output the preset normal result information of the image behavior as the image detection result information; S36: If not, determine the abnormal reference sound information based on the image behavior feature, and use the abnormal reference sound information as the behavior reference sound information; S37: Determine the abnormal result information of the image behavior based on the image behavior feature and the detection comprehensive reference behavior feature, and use the abnormal result information of the image behavior as the image detection result information.

6. The edge-side multi-modal abnormal behavior detection method according to claim 5, wherein The method for generating the image reference sound information includes: S351: Determine the comprehensive reference sound information based on the detection comprehensive reference behavior feature; S352: Determine the behavior position prediction condition information based on the image detection position point; S353: Determine the prediction condition satisfaction value based on the behavior position prediction condition information and the image detection information; S354: Determine whether the prediction condition satisfaction value is greater than the preset prediction condition reference value; S355: If so, generate the single behavior associated sound information based on the image behavior feature and the prediction condition satisfaction value, and combine the comprehensive reference sound information and the single behavior associated sound information as the image reference sound information; S356: If not, use the comprehensive reference sound information as the image reference sound information.

7. The edge - side multimodal abnormal behavior detection method according to claim 6, wherein The method for generating the single behavior associated sound information includes: S3551: Calculate the difference between the prediction condition satisfaction value and the preset prediction condition reference value as the condition satisfaction deviation value; S3552: Determine the associated selection number value based on the condition satisfaction deviation value; S3553: Determine the associated behavior feature and the associated degree value based on the image behavior feature; S3554: Sort based on the associated degree value from large to small, and select the associated behavior feature corresponding to the sorting result of the associated selection number value as the selected behavior feature; S3555: Determine the associated selection sound information based on the selected behavior feature; S3556: Combine the associated selection sound information to form the single behavior associated sound information.

8. The edge-side multimodal abnormal behavior detection method according to claim 1, characterized in that The method for generating the sound detection result information includes: S41: Determine the sound detection text information and the sound detection intensity value based on the sound detection information; S42: Determine the reference sound text information and the reference sound intensity value based on the behavior reference sound information; S43: Determine text coincidence information based on the text information detected from the sound and the reference sound text information; S44: Determine the sound intensity deviation value based on the detected sound intensity value and the reference sound intensity value; S45: Generate coincidence intensity comprehensive result information based on the text coincidence information and the sound intensity deviation value, and use the coincidence intensity comprehensive result information as the sound detection result information.

9. The edge - side multimodal abnormal behavior detection method according to claim 8, characterized in that, The method for generating the coincidence intensity comprehensive result information includes: S451: Retrieve the text coincidence rate and the text coincidence position based on the text coincidence information; S452: Determine the position requirement coincidence interval based on the text coincidence position; S453: Determine the intensity influence value based on the sound intensity deviation value; S454: Determine the requirement adjustment coincidence interval based on the intensity influence value and the position requirement coincidence interval; S455: Determine the coincidence adjustment result information based on the text coincidence rate and the requirement adjustment coincidence interval, and use the coincidence adjustment result information as the coincidence intensity comprehensive result information.

10. An edge-side multimodal abnormal behavior detection system, characterized in that, Comprising: An acquisition module for acquiring image detection information and sound detection information; A memory storing a program for implementing an edge-side multimodal abnormal behavior detection method as described in any one of claims 1 to 9; A processor for loading and executing the program stored in the memory.

Images