Abnormal event pre-judging method and device

Abstract

The embodiment of the invention provides an abnormal event pre-judgment method and device, and the method comprises the steps: carrying out the model training through receiving learning materials of a business platform, and obtaining a face recognition model, an article recognition model, a behavior recognition model, and a sound recognition model; the method comprises the following steps: receiving environment images recorded by a plurality of cameras, inputting the environment images into a face recognition model, an article recognition model and a behavior recognition model, respectively determining abnormal image risk weight values corresponding to the cameras, receiving environment sounds recorded by the plurality of cameras, inputting the environment sounds into a sound recognition model, and identifying the abnormal image risk weight values corresponding to the cameras. Respectively determining an abnormal sound risk weight value corresponding to each camera; the environment noise shot by the noise source camera is received, if the environment noise is larger than the preset noise threshold value, the comprehensive risk value of the noise source camera is calculated, and the corresponding warning information is generated according to the judgment result of the comprehensive risk value, and the response efficiency and flexibility of the abnormal event can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method and device for predicting abnormal events. Background Art

[0002] Abnormal problems often occur in public enclosed places, such as buses, trains, and railway stations, especially public security problems and some behaviors that endanger public safety. These problems not only affect public order, but may also pose a threat to public safety. Common abnormal problems include quarrels, fights, brawls caused by disputes, as well as safety hazards such as theft, carrying contraband, violence, and terrorist incidents.

[0003] In order to prevent and deal with these abnormal problems, existing technical means rely on manual monitoring and post-event processing, including setting up cameras in public places such as subways, train stations, and car compartments. Once the above abnormal events occur, post-event processing can be carried out based on the video images recorded by the cameras. However, this method faces problems such as slow response speed, limited coverage, and high labor costs, and it is impossible to respond in time when abnormal events are in progress. The post-event processing method alone also increases the losses of abnormal events. For example, for theft, the best time to stop theft is to stop it when the theft continues. If the theft cannot be monitored and stopped when it continues, a lot of manpower and resources will be wasted in the subsequent recovery of the stolen goods; for another example, for voyeurism, if it cannot be discovered and handled in time when the voyeurism is being filmed, the subsequent uploading of the voyeuristic images to the Internet will also waste a lot of public resources.

[0004] Therefore, the existing methods cannot meet the requirements of abnormal event response efficiency, and a method is urgently needed to increase the response flexibility of abnormal events and improve the response efficiency of abnormal events. Summary of the invention

[0005] In view of the problems in the prior art, the present application provides a method and device for predicting abnormal events, which can improve the efficiency and flexibility of responding to abnormal events.

[0006] In order to solve at least one of the above problems, the present application provides the following technical solutions: In a first aspect, the present application provides a method for predicting abnormal events, comprising: Receive learning materials from the business platform, perform model training operations according to the learning materials, and respectively determine corresponding face recognition models, object recognition models, behavior recognition models, and voice recognition models, wherein the learning materials include face recognition pictures, object recognition pictures, behavior recognition pictures, and voice corpus information; Receive the environmental images captured by the multiple cameras, input the environmental images into the face recognition model, the object recognition model and the behavior recognition model, respectively determine the facial attribute information, object type information and behavior pattern information corresponding to each camera, perform weighted calculation operations on the facial attribute information, the object type information and the behavior pattern information according to preset abnormal event weights, respectively determine the abnormal image risk weight value corresponding to each camera, receive the environmental sounds captured by the multiple cameras, input the environmental sounds into the sound recognition model, perform a matching operation based on the recognition results output by the sound recognition model and a preset abnormal negative vocabulary corpus, and perform a weighted calculation operation based on the negative semantics obtained after the matching operation and the preset semantic weights, respectively determine the abnormal sound risk weight value corresponding to each camera; Receive the environmental noise recorded by the first camera, and determine whether the environmental noise is greater than a preset noise threshold; if so, perform a weight accumulation operation according to the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera to determine the corresponding first comprehensive risk value; determine whether the first comprehensive risk value is greater than the preset first risk threshold, and generate corresponding warning information according to the judgment result after the judgment operation; send the warning information to the business platform, so that the business platform performs an abnormal response according to the warning information, wherein the warning information includes alarm information, early warning information and prompt information, and the first camera is the camera closest to the noise source among the multiple cameras.

[0007] Furthermore, the method further comprises: According to a preset time interval, polling is performed to determine whether the comprehensive risk value corresponding to each camera is greater than a preset second risk threshold, wherein the comprehensive risk value corresponding to each camera is determined by performing a weighted accumulation operation based on the abnormal image risk weight value corresponding to each camera and the abnormal sound risk weight value corresponding to each camera; If it is greater, an alarm information generation operation is performed, and the alarm information is sent to the business platform, so that the business platform performs an abnormal response according to the alarm information; If not, the comprehensive risk values ​​corresponding to each camera are accumulated to determine whether the comprehensive risk values ​​corresponding to all cameras obtained after the accumulation operation are greater than the preset third risk threshold. If greater than, an alarm information generation operation is performed, and the alarm information is sent to the business platform so that the business platform responds to the abnormality according to the alarm information. Furthermore, the model training operation is performed according to the learning data to respectively determine the corresponding face recognition model, object recognition model, behavior recognition model and sound recognition model, including: Performing a model training operation on a preset Fast RCNN network according to the face recognition images in the learning material to determine a corresponding face recognition model, wherein the face recognition model includes a face detection module, a face normalization module, a face attribute recognition module, a face quality assessment module, and a face tracking module, and the face recognition model is used to recognize faces and analyze people's negative emotions to obtain face attribute information; Performing a model training operation on a preset convolutional neural network according to the object recognition images in the learning material to determine a corresponding object recognition model, wherein the object recognition model is used to identify objects that endanger environmental safety; Performing a model training operation on a preset Crowd Counting algorithm according to the behavior recognition pictures in the learning material to determine a corresponding behavior recognition model, wherein the behavior recognition model is used to detect crowd gathering and behaviors that endanger environmental safety; A model training operation is performed on a preset semantic recognition initial model according to the sound corpus information in the learning material to determine a corresponding sound recognition model, wherein the sound recognition model is used to convert sound into semantic information.

[0008] Furthermore, performing a weighted calculation operation on the face attribute information, the object type information, and the behavior pattern information according to a preset abnormal event weight to respectively determine the abnormal image risk weight value corresponding to each camera includes: Performing a weighting operation on the facial attribute information, the item type information, and the behavior pattern information according to the preset abnormal event weights, and determining the facial attribute information weight, the item type information weight, and the behavior pattern information weight corresponding to each camera; A weight accumulation operation is performed according to the facial attribute information weight, the item type information weight, and the behavior pattern information weight, to respectively determine the abnormal image risk weight value corresponding to each camera.

[0009] Furthermore, the matching operation is performed based on the recognition result output by the sound recognition model and the preset abnormal negative vocabulary corpus, and a weighted calculation operation is performed based on the negative semantics obtained after the matching operation and the preset semantic weight, to respectively determine the abnormal sound risk weight value corresponding to each camera, including: Performing a word segmentation operation on the recognition result output by the sound recognition model according to a preset word segmentation algorithm to determine the corresponding word segmentation vocabulary; The segmented vocabulary and the preset abnormal negative vocabulary corpus are matched, and a weighted calculation operation is performed according to the negative semantics obtained after the matching operation and the preset semantic weight, so as to determine the abnormal sound risk weight value corresponding to each camera respectively.

[0010] Further, the determining whether the environmental noise is greater than a preset noise threshold, if greater than, performing a weighted accumulation operation according to the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera to determine the corresponding first comprehensive risk value, includes: Determine whether the environmental noise is greater than a preset noise threshold, and if so, determine whether the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera exist at the same time; If both exist, a weighted accumulation operation is performed on the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera, and a multiplication operation is performed on the accumulated result after the weighted accumulation operation to determine the corresponding first comprehensive risk value; If they do not exist at the same time, the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera are weighted and accumulated to determine the corresponding first comprehensive risk value.

[0011] Furthermore, the determining whether the first comprehensive risk value is greater than a preset first risk threshold and generating corresponding warning information according to the determination result after the determination operation includes: Determine whether the first comprehensive risk value is greater than a preset first risk threshold, and if so, generate a warning message to determine the corresponding warning message; If it is less than, determine whether it is greater than half of the preset first risk threshold. If it is greater, perform a warning information generation operation to determine the corresponding early warning information. If it is less than, perform a warning information generation operation to determine the corresponding prompt information.

[0012] In a second aspect, the present application provides an abnormal event prediction device, comprising: A multi-model training module, used to receive learning materials from the business platform, perform model training operations according to the learning materials, and respectively determine corresponding face recognition models, object recognition models, behavior recognition models, and voice recognition models, wherein the learning materials include face recognition pictures, object recognition pictures, behavior recognition pictures, and voice corpus information; A risk weight calculation module, for receiving environmental images captured by multiple cameras, inputting the environmental images into the face recognition model, the object recognition model and the behavior recognition model, respectively determining the facial attribute information, object type information and behavior pattern information corresponding to each camera, performing weighted calculation operations on the facial attribute information, the object type information and the behavior pattern information according to preset abnormal event weights, respectively determining the abnormal image risk weight value corresponding to each camera, receiving environmental sounds captured by multiple cameras, inputting the environmental sounds into the sound recognition model, performing a matching operation based on the recognition results output by the sound recognition model and a preset abnormal negative vocabulary corpus, and performing a weighted calculation operation based on the negative semantics obtained after the matching operation and the preset semantic weights, respectively determining the abnormal sound risk weight value corresponding to each camera; A warning information generation module is used to receive the environmental noise recorded by the first camera, determine whether the environmental noise is greater than a preset noise threshold, and if so, perform a weight accumulation operation according to the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera to determine the corresponding first comprehensive risk value, determine whether the first comprehensive risk value is greater than the preset first risk threshold, and generate corresponding warning information according to the judgment result after the judgment operation, and send the warning information to the business platform so that the business platform performs an abnormal response according to the warning information, wherein the warning information includes alarm information, early warning information and prompt information, and the first camera is the camera closest to the noise source among the multiple preset cameras.

[0013] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the abnormal event prediction method when executing the program.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the abnormal event prediction method.

[0015] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which implements the steps of the abnormal event prediction method when executed by a processor.

[0016] It can be seen from the above technical scheme that the present application provides a method and device for predicting abnormal events, which performs model training by receiving learning materials from a business platform, and determines a face recognition model, an object recognition model, a behavior recognition model, and a sound recognition model respectively; receives environmental images recorded by multiple cameras, inputs the environmental images into the face recognition model, the object recognition model, and the behavior recognition model, and determines the abnormal image risk weight value corresponding to each camera respectively; receives environmental sounds recorded by multiple cameras, inputs the environmental sounds into the sound recognition model, and determines the abnormal sound risk weight value corresponding to each camera respectively; receives environmental noise recorded by a noise source camera, and if the environmental noise is greater than a preset noise threshold, calculates the comprehensive risk value of the noise source camera, and generates corresponding warning information based on the comprehensive risk value judgment result, thereby improving the response efficiency and flexibility of abnormal events. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is one of the flow charts of the abnormal event prediction method in the embodiment of the present application; Figure 2 This is a second flow chart of the abnormal event prediction method in the embodiment of the present application; Figure 3 The third flowchart of the abnormal event prediction method in the embodiment of the present application; Figure 4 This is a fourth flow chart of the abnormal event prediction method in the embodiment of the present application; Figure 5 This is a fifth flow chart of the abnormal event prediction method in the embodiment of the present application; Figure 6 This is a sixth flow chart of the abnormal event prediction method in the embodiment of the present application; Figure 7 FIG7 is a flow chart of the abnormal event prediction method in the embodiment of the present application; Figure 8 is a structural diagram of an abnormal event prediction device in an embodiment of the present application; Fig. 9 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0019] Reference numerals: Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0022] Considering the problem that the existing abnormal event processing methods cannot meet the requirements for efficiency and flexibility. The present application provides a method and device for predicting abnormal events, which receives the learning materials of the business platform for model training, and respectively determines the face recognition model, the object recognition model, the behavior recognition model and the sound recognition model; receives the environmental images recorded by multiple cameras, inputs the environmental images into the face recognition model, the object recognition model and the behavior recognition model, and respectively determines the abnormal image risk weight value corresponding to each camera; receives the environmental sounds recorded by multiple cameras, inputs the environmental sounds into the sound recognition model, and respectively determines the abnormal sound risk weight value corresponding to each camera; receives the environmental noise recorded by the noise source camera, and if the environmental noise is greater than the preset noise threshold, calculates the comprehensive risk value of the noise source camera, and generates corresponding warning information according to the comprehensive risk value judgment result, thereby improving the response efficiency and flexibility of abnormal events.

[0023] In order to improve the response efficiency and flexibility of abnormal events, the present application provides an embodiment of an abnormal event prediction method, see Figure 1 The abnormal event prediction method specifically includes the following contents: Step S101: receiving learning materials from the business platform, performing model training operations according to the learning materials, and respectively determining corresponding face recognition models, object recognition models, behavior recognition models, and voice recognition models, wherein the learning materials include face recognition pictures, object recognition pictures, behavior recognition pictures, and voice corpus information; Optionally, in this embodiment, the purpose of this step is to configure a pre-trained algorithm model on the edge computing platform, and the edge computing platform receives the algorithm adjustment image sent by the business platform to perform self-learning to identify abnormal behavior.

[0024] Optionally, in this embodiment, in this step, the learning materials of the business platform include two categories: image materials and sound materials. It can be understood that the learning materials of the business platform are also the training sets used by the edge computing platform to train the model.

[0025] Specifically, image data are divided into face images, object images, and behavior images.

[0026] More specifically, face images are mainly based on face recognition. On the basis of training face detection images, image labels of face attributes are added, including analysis of facial expressions (such as fear, anger, anxiety, etc.), gender, age, glasses (sunglasses), masks, hats, etc.

[0027] More specifically, object images mainly refer to objects that endanger environmental safety, especially public security, such as controlled knives, guns, etc. The category and location of each prohibited item (for example, a gun location box or a bounding box for a controlled knife) need to be clearly indicated in the object image.

[0028] More specifically, behavioral images use a behavioral feature library that contains a variety of labeled bad behavior data to compare actions and scenes in images, such as theft, voyeurism, etc.

[0029] More specifically, the audio data is connected to iFlytek's speech recognition training set using an interface to convert speech into text, laying the foundation for subsequent negative vocabulary analysis.

[0030] Optionally, in this embodiment, the above learning materials are used to train corresponding models respectively.

[0031] Specifically, face images are used to train the face recognition algorithm Fast RCNN, which consists of a face detection module, a face normalization module, a face attribute module, a face quality module, and a face tracking module.

[0032] More specifically, the face detection module first detects whether there is a face in the image. Through the features in the image, Fast RCNN can accurately extract the face area from the background, calibrate the face position in the image, and crop it for subsequent processing; secondly, the face normalization module normalizes the detected face image, adjusts the image size, ensures that the face is in a standardized coordinate system, reduces the influence of external factors such as different lighting and angles, and unifies it to a fixed size and proportion. This process helps to improve the accuracy of subsequent emotion analysis and attribute recognition; next, the face attribute module performs attribute recognition on the normalized face image, uses convolutional neural network (CNN) for multi-task learning, and simultaneously performs expression classification (such as fear, anger, anxiety), accessory recognition (such as sunglasses, masks, hats), etc., to analyze the attributes of the face. Then, based on the specific results output by the face attribute module, the face quality module determines whether further image restoration or discarding is needed, and based on the image quality assessment model, determines whether the image is clear and whether there is too much noise or blur. Finally, the face tracking module uses a face tracking algorithm (such as the KLT tracking algorithm) based on the correlation between video frames to keep tracking the target face.

[0033] In terms of effectiveness, by analyzing the captured images and recognized facial attributes, the system can identify potential negative emotions such as fear, anger, anxiety, etc., as well as the wearing of sunglasses, masks, and hats. By subsequently adding weights to these situations, the probability of abnormal events occurring can be evaluated.

[0034] Specifically, convolutional neural networks are trained with object images, and contraband (such as guns, controlled knives, etc.) are detected through image recognition technology. A data set containing images of various contraband (such as guns, knives, etc.) is used to train the target detection model. The contraband images are labeled through the training model, and learning is done to learn how to identify contraband in complex environments.

[0035] Specifically, in the behavior recognition model, crowd gathering behavior is realized through the crowd counting algorithm. For behaviors such as stealing, secretly photographing, spitting, throwing garbage, and physical conflicts, the time dimension is added to the traditional behavior feature library to capture the dynamic nature of the behavior. By analyzing multiple images within 1-2 seconds (for example, stealing, secretly photographing, etc.), a "time series set" of behavior features is formed, thereby increasing the accuracy of behavior recognition.

[0036] For example, using camera data combined with a crowd counting algorithm, the number of people in an area is analyzed in real time. When the number of people exceeds the set threshold, the system triggers a "people gathering" event. When a people gathering event is detected, the behavior analysis system is started to compare the currently captured image set (a time series set containing multiple images) with the behavior images in the feature library. Image similarity metrics (such as cosine similarity, Euclidean distance, etc.) are used to determine whether the current behavior matches the known behaviors in the feature library. Based on the comparison results, the corresponding behavior type (such as theft, sneak photography, etc.) is classified. If the behavior matches the pattern in the feature library, it is marked as a related behavior.

[0037] In terms of effectiveness, behavioral analysis will be more accurate in areas where people gather. By leveraging the time dimension, the system can accurately identify specific behavioral patterns, especially in cases where there are a large number of people gathered, to avoid missed detections.

[0038] Specifically, the speech recognition model is directly connected to iFlytek, which transfers the speech file to the calling interface and returns the recognition result.

[0039] Step S102: receiving the environmental images captured by the multiple cameras, inputting the environmental images into the face recognition model, the object recognition model and the behavior recognition model, respectively determining the facial attribute information, object type information and behavior pattern information corresponding to each camera, performing weighted calculation operations on the facial attribute information, the object type information and the behavior pattern information according to preset abnormal event weights, respectively determining the abnormal image risk weight value corresponding to each camera, receiving the environmental sounds captured by the multiple cameras, inputting the environmental sounds into the sound recognition model, performing a matching operation based on the recognition results output by the sound recognition model and a preset abnormal negative vocabulary corpus, and performing a weighted calculation operation based on the negative semantics obtained after the matching operation and the preset semantic weights, respectively determining the abnormal sound risk weight value corresponding to each camera; Optionally, in this embodiment, the purpose of this step is for the edge computing platform to receive the image and voice information recorded by the camera in the environment, analyze it through the model trained by the above step S101, obtain the analysis results, and assign weights to the analysis results according to the probability of prone public security incidents, and finally obtain the image risk weight value and the sound risk weight value.

[0040] Optionally, in this embodiment, all cameras in the environment transmit environmental images to the edge computing platform, and the edge computing platform performs calculations simultaneously. The face recognition model, object recognition model, and behavior recognition model respectively calculate the output of each camera's image abnormal event prediction results, which will be explained below using one camera as an example.

[0041] Specifically, the face recognition model outputs face attribute information, including emotional information (fear, anger, anxiety, etc.) and accessory information (hats, sunglasses, masks, etc.).

[0042] Specifically, the object recognition model outputs object type information, including (guns, controlled knives, etc.).

[0043] Specifically, the behavior recognition model outputs behavior pattern information, including (secret photography, theft, robbery, etc.).

[0044] Optionally, after obtaining the prediction results output by the above model, weights are assigned to the above abnormal results according to the probability of abnormal events. The sub-items of each major item (such as facial expressions, hats, sunglasses, and masks in facial attribute analysis) have weight ratios. The weight of the major item is calculated based on the weights of the sub-items, and then the major items also have weight ratios, and the comprehensive weight is calculated based on the weight ratios between the major items.

[0045] For example: (1) Facial attribute analysis, identifying specific emotions (such as fear, anger, and anxiety) and assigning corresponding weight values ​​to each emotion.

[0046] Negative emotions: Identify fearful expressions (weight 1), angry expressions (weight 2), and anxious expressions (weight 1).

[0047] Wearing condition: Identify wearing sunglasses (weight 0.5), wearing a mask (weight 0.5), and wearing a hat (weight 0.5).

[0048] Assume that in the captured image, there is a person showing an angry expression and wearing a mask at the same time. Then, the comprehensive weight of the facial attributes of this person is calculated as follows: Angry emoticon weight: 2 Mask wearing weight: 0.5 The combined weight of facial attributes = 2 (anger) + 0.5 (mask) = 2.5 (2) Item type analysis Identify guns (weight 10) and controlled knives (weight 5).

[0049] Assuming that no contraband is found in the captured image, then the comprehensive weight of contraband is: Contraband Weight: 0 (3) Behavior recognition analysis Through the crowd counting algorithm, it is identified that the number of people gathered exceeds a certain threshold (assuming the threshold is 5 people and the weight is 2).

[0050] Assume that in the captured image, there are 6 people gathered at the front door area. Then, the comprehensive weight of the gathering of people is: Personnel gathering weight: 2 Identify theft (weight 3), peeping (weight 3), and robbery (weight 5).

[0051] Suppose that in the captured image, there is a person stealing. Then, the comprehensive weight of the action is: Theft weight: 3 In this way, the weight of the facial attribute is 2.5, the weight of the object type is 0, the weight of the behavior recognition is 5, and the abnormal image risk weight value is 7.5.

[0052] Optionally, in this embodiment, all cameras in the environment transmit ambient sound to the edge computing platform, and the edge computing platform performs calculations at the same time. The sound recognition model calculates and obtains the sound abnormal event prediction results of each camera outputted by it, which will be explained below using one camera as an example.

[0053] Specifically, use a third-party speech recognition platform (such as iFlytek) to convert speech into text, match the converted text with a negative emotion vocabulary, and identify whether it contains emotionally intense words (such as "kill", "swear", etc.). Classify emotionally intense words contained in the text, such as anger, threat, intimidation, etc.

[0054] After identifying negative emotional words, one way is to assign different risk weights according to the emotional type, such as anger: weight 2, threat / violence: weight 5. Another way is to assign a weight of 1 to each negative word. Finally, the total abnormal sound risk weight value is the accumulation of each small sound risk.

[0055] For example, According to the first method, if the word "violence" appears in the environmental scenario, no matter how many times it appears, the weight value is 5. If no other negative words appear, the weight value is 0, and the abnormal sound risk weight value is 5+0=5.

[0056] According to the second method, in the environment scenario, assuming that in the last 10 seconds, the voice recognition system recognizes 5 negative words, then regardless of the type of negative words, the comprehensive weight of the sound is 5.

[0057] After the above step S102, we calculated the abnormal image risk weight value and abnormal sound risk weight value of each camera through the edge computing platform, laying the foundation for subsequent timely response to risks.

[0058] Step S103: Receive the environmental noise recorded by the first camera, and determine whether the environmental noise is greater than a preset noise threshold; if so, perform a weight accumulation operation according to the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera to determine the corresponding first comprehensive risk value; determine whether the first comprehensive risk value is greater than the preset first risk threshold; and generate corresponding warning information according to the judgment result after the judgment operation, and send the warning information to the business platform so that the business platform performs an abnormal response according to the warning information, wherein the warning information includes alarm information, early warning information and prompt information, and the first camera is the camera closest to the noise source among the multiple cameras.

[0059] Optionally, in this embodiment, the purpose of this step is to build a risk response algorithm for timely responding to environmental abnormality risks.

[0060] Specifically, most abnormal security incidents start with a quarrel, followed by a quarrel or a fight. Therefore, a method is defined to detect environmental noise based on the noise sensor configured on the camera. The edge computing platform receives the environmental noise and determines whether the noise is greater than 60 decibels (the sound of a quarrel is greater than 60 decibels). If it is greater, the camera closest to the noise is first located, and the abnormal image risk weight value and abnormal sound risk weight value of the camera in the last 10 seconds are obtained (at this time, the edge computing platform has calculated the abnormal image risk weight value and abnormal sound risk weight value of each camera). At the same time, a judgment is made. If there is both a voice risk weight value X and a sound risk weight value Y, then the calculated weight is (X+Y) 2. Since there are both sound risk and image recognition risk, the risk must be high, so we multiply it. If there is only image risk or only sound risk, the comprehensive risk weight is (X+Y).

[0061] If the cumulative value of the comprehensive risk weight is greater than 10, an alarm is generated and sent to the business platform, along with alarm information (alarm picture, alarm information, and alarm point).

[0062] If the cumulative value of the comprehensive risk weight is less than 10 but greater than 5, an early warning message is generated and sent to the business platform, including early warning information (early warning picture, early warning information, and early warning points).

[0063] If the comprehensive risk weight is less than 5, a voice prompt will be sent to the camera (eg: keep quiet in public places, etc.).

[0064] For example, suppose there is an edge computing box on a bus. The bus is equipped with two cameras, one for the front door and one for the back door. The edge box uses RTSP (real-time communication protocol) to get the video stream from the camera and take a picture every 1 second.

[0065] First, the edge box analyzes the facial attributes, object types, and behavior patterns of the video images every second in real time, and performs weighted accumulation of the identified subcategories (such as the subcategories illustrated in step S102) to obtain the corresponding major category weight values, and then accumulates the above three major category weight values ​​to obtain the abnormal image risk weight value; the edge box performs semantic recognition of the sound in real time, and compares the recognized sentences with the negative vocabulary library. If negative words or semantics appear, the identified risk types are accumulated according to the predetermined weight value accumulation method to obtain the abnormal sound risk weight value.

[0066] Then, when an abnormal environmental event occurs, it is usually accompanied by noise. The camera noise sensor identifies the noise, the edge box obtains the camera noise data, locates the nearest camera (assuming it is the front door), obtains the abnormal image risk weight value and abnormal sound risk weight value of the front door camera in the edge box, and accumulates them. If both have values, the accumulated value multiples are processed to obtain the comprehensive risk weight value, and it is determined whether the comprehensive risk weight value is greater than 10. If it is greater, an alarm message is generated and sent to the business platform. If the weight accumulation value is less than 10 and greater than 5, an early warning sending platform is generated. If the weight is less than 5 and the early warning threshold is not reached, the box sends a prompt voice file to the camera (eg: keep quiet in public places, harmony is the most valuable, classmates, anger is not only harmful to the body but also easy to age, etc.). If a person sees a public security incident such as a violation of the law on the bus and wants to call the police, he can just yell at a voice exceeding 60 decibels to trigger the early warning strategy, which is convenient and efficient.

[0067] This example demonstrates how this embodiment deploys a multifunctional model and weights the predictions output by the model to build an edge computing platform, through which environmental anomalies can be quickly determined and corresponding warning information can be sent, thereby improving the efficiency of abnormal response.

[0068] From the above description, it can be seen that the abnormal event prediction method provided in the embodiment of the present application can perform model training by receiving learning materials from the business platform, and determine the face recognition model, object recognition model, behavior recognition model and sound recognition model respectively; receive environmental images recorded by multiple cameras, input the environmental images into the face recognition model, object recognition model and behavior recognition model, and determine the abnormal image risk weight value corresponding to each camera respectively; receive environmental sounds recorded by multiple cameras, input the environmental sounds into the sound recognition model, and determine the abnormal sound risk weight value corresponding to each camera respectively; receive the environmental noise recorded by the noise source camera, if the environmental noise is greater than the preset noise threshold, calculate the comprehensive risk value of the noise source camera, and generate corresponding warning information based on the comprehensive risk value judgment result, thereby improving the response efficiency and flexibility of abnormal events.

[0069] In one embodiment of the abnormal event prediction method of the present application, see Figure 2 , and can also include the following: Step S201: According to a preset time interval, polling is performed to determine whether the comprehensive risk value corresponding to each camera is greater than a preset second risk threshold, wherein the comprehensive risk value corresponding to each camera is determined by weighted accumulation operation based on the abnormal image risk weight value corresponding to each camera and the abnormal sound risk weight value corresponding to each camera; Step S202: If it is greater than, an alarm information generation operation is performed, and the alarm information is sent to the business platform, so that the business platform performs an abnormal response according to the alarm information; Step S203: If it is not greater than, then the comprehensive risk value corresponding to each camera is accumulated, and it is determined whether the comprehensive risk value corresponding to all cameras obtained after the accumulation operation is greater than the preset third risk threshold; if it is greater, an alarm information generation operation is performed, and the alarm information is sent to the business platform, so that the business platform responds to the exception according to the alarm information.

[0070] Optionally, in this embodiment, this step is a supplementary step of the above step S103. Step S103 provides a way of responding to environmental anomalies, and this step is another way. When environmental anomalies do not generate noise, we timely discover environmental anomalies based on the scheduled polling task.

[0071] Specifically, the edge box sets a scheduled polling task to scan the comprehensive risk weights of all cameras connected within 10 seconds (i.e., the cumulative value of the abnormal image risk weight value and the abnormal sound risk weight value). If the cumulative weight value of a single camera is greater than 10, a single camera alarm is generated to send to the platform. If it is less than 10, the cumulative weight of all cameras connected to the edge box is judged. If the weight value is greater than 5, N (N represents the number of working cameras connected to this edge box), a combined alarm is generated to send to the platform. If the weight value is less than 10 or 5 N, the edge box will not report warning information to the platform, but can send prompt voice messages directly to the camera, such as "Please keep your finances safe and be careful not to lose them".

[0072] For example, suppose a robbery occurs on a bus and no noise is generated. At this time, the camera detects a hat (0.5) and a mask (0.5) on the face, a robbery (5) on the behavior, a controlled knife (5) on the object, and a negative word (2) on the sound. The comprehensive risk weight is 13. At this time, an alarm message for the camera will be generated, and the alarm image, alarm message, and alarm point will be sent to the business platform. The platform can obtain the real-time point coordinates of the alarm camera (the camera is installed on the bus or train and will move in real time) to serve as a basis for the next step of response and processing.

[0073] For example, suppose a theft occurs on a bus and no noise is generated. At this time, the front door camera detects a hat (0.5) and a mask (0.5), and the behavior detects theft (3). The overall risk weight is 4. At this time, the edge computing platform calculates the weighted cumulative value of the front door and back door cameras. Assuming that the back door camera detects an angry expression (2), the overall risk weight is 2, then the weighted cumulative value is 6, 6 < 10 (5 2) At this time, the edge computing platform does not interact with the business platform, but sends the built-in voice file to the camera (the camera that captured the action is closest to the action) through the intercom function, playing the pre-made voice file ("Please keep your finances safe and prevent loss", "Be careful when you go out", etc.). If a person sees a law-breaking or other public security incident on the bus and wants to call the police, he can simply shout at a voice exceeding 60 decibels to trigger the early warning strategy, which is convenient and efficient. At this time, the real-time positioning of the vehicle will be uploaded to the platform, laying the foundation for subsequent timely processing.

[0074] Through step S203, this embodiment supplements another environmental abnormality response method, increasing the flexibility of responding to environmental abnormalities.

[0075] In one embodiment of the abnormal event prediction method of the present application, see Figure 3 , and can also include the following: Step S301: performing a model training operation on a preset Fast RCNN network according to the face recognition pictures in the learning material to determine a corresponding face recognition model, wherein the face recognition model includes a face detection module, a face normalization module, a face attribute recognition module, a face quality assessment module and a face tracking module, and the face recognition model is used to recognize faces and analyze people's negative emotions to obtain face attribute information; Step S302: performing a model training operation on a preset convolutional neural network according to the object recognition images in the learning material to determine a corresponding object recognition model, wherein the object recognition model is used to identify objects that endanger environmental safety; Step S303: performing a model training operation on a preset Crowd Counting algorithm according to the behavior recognition pictures in the learning material to determine a corresponding behavior recognition model, wherein the behavior recognition model is used to detect crowd gathering and behaviors that endanger environmental safety; Step S304: performing a model training operation on a preset semantic recognition initial model according to the sound corpus information in the learning material to determine a corresponding sound recognition model, wherein the sound recognition model is used to convert sound into semantic information.

[0076] Optionally, the face recognition model uses the Fast RCNN algorithm, which consists of a face detection module, a face normalization module, a face attribute module, a face quality module, and a face tracking module.

[0077] More specifically, the face detection module first detects whether there is a face in the image. Through the features in the image, Fast RCNN can accurately extract the face area from the background, calibrate the face position in the image, and crop it for subsequent processing; secondly, the face normalization module normalizes the detected face image, adjusts the image size, ensures that the face is in a standardized coordinate system, reduces the influence of external factors such as different lighting and angles, and unifies it to a fixed size and proportion. This process helps to improve the accuracy of subsequent emotion analysis and attribute recognition; next, the face attribute module performs attribute recognition on the normalized face image, uses convolutional neural network (CNN) for multi-task learning, and simultaneously performs expression classification (such as fear, anger, anxiety), accessory recognition (such as sunglasses, masks, hats), etc., to analyze the attributes of the face. Then, based on the specific results output by the face attribute module, the face quality module determines whether further image restoration or discarding is needed, and based on the image quality assessment model, determines whether the image is clear and whether there is too much noise or blur. Finally, the face tracking module uses a face tracking algorithm (such as the KLT tracking algorithm) based on the correlation between video frames to keep tracking the target face.

[0078] In terms of effectiveness, by analyzing the captured images and recognized facial attributes, the system can identify potential negative emotions such as fear, anger, anxiety, etc., as well as the wearing of sunglasses, masks, and hats. By subsequently adding weights to these situations, the probability of abnormal events occurring can be evaluated.

[0079] Optionally, the object recognition model, the training set contains a large number of images labeled with prohibited items (such as guns, controlled knives, etc.). The platform or third-party platform will send these high-quality images through the interface for learning.

[0080] Use a convolutional neural network (CNN) model to detect objects in images and train the model to identify different types of contraband. During the training process, the model optimizes parameters by minimizing the loss function so that the model can more accurately identify objects in the image. The loss function usually includes classification loss and positioning loss. Use the validation set to test the model and evaluate its accuracy, recall rate, and F1 value to ensure that the model can effectively identify contraband in the actual environment. After the model is trained and deployed, the edge box will analyze the captured image to identify whether there are contraband.

[0081] Optional, behavior recognition model, crowd gathering behavior is realized through crowd counting algorithm, such as stealing, secretly taking photos, spitting, throwing garbage, physical conflict, etc., on the basis of the traditional behavior feature library, to capture the dynamic nature of the behavior, add the time dimension. By analyzing multiple images within 1-2 seconds (for example, stealing, spitting, etc.), a "time series set" of behavior features is formed, thereby increasing the accuracy of behavior recognition. In terms of effect, in areas where people gather, behavior analysis will be more accurate. Using the characteristics of the time dimension, the system can accurately identify specific behavior patterns, especially in the case of a large number of people gathering, to avoid missed detection.

[0082] Optionally, the sound recognition model can be directly connected to iFlytek to transfer the voice file to the calling interface and return the recognition result.

[0083] Through step S304, this embodiment constructs four intelligent models for face detection, behavior detection, object detection and sound detection, laying a foundation for subsequent identification of environmental anomalies.

[0084] In one embodiment of the abnormal event prediction method of the present application, see Figure 4 , and can also include the following: Step S401: performing a weighting operation on the facial attribute information, the object type information, and the behavior pattern information according to a preset abnormal event weight, and determining the facial attribute information weight, the object type information weight, and the behavior pattern information weight corresponding to each camera; Step S402: performing a weight accumulation operation according to the face attribute information weight, the object type information weight, and the behavior pattern information weight, and determining the abnormal image risk weight value corresponding to each camera respectively.

[0085] Optionally, in this embodiment, weights are assigned to the above-mentioned abnormal results according to the probability of abnormal events that are prone to occur. The sub-items of each major item (such as facial expressions, hats, sunglasses, and masks in facial attribute analysis) have weight ratios, and the weights of the major items are calculated based on the weights of the sub-items. Then, there are also weight ratios between the major items, and a comprehensive weight calculation is performed based on the weight ratios between the major items.

[0086] For example: (1) Facial attribute analysis, identifying specific emotions (such as fear, anger, and anxiety) and assigning corresponding weight values ​​to each emotion.

[0087] Negative emotions: Identify fearful expressions (weight 1), angry expressions (weight 2), and anxious expressions (weight 1).

[0088] Wearing condition: Identify wearing sunglasses (weight 0.5), wearing a mask (weight 0.5), and wearing a hat (weight 0.5).

[0089] Assume that in the captured image, there is a person showing an angry expression and wearing a mask at the same time. Then, the comprehensive weight of the facial attributes of this person is calculated as follows: Angry emoticon weight: 2 Mask wearing weight: 0.5 The combined weight of facial attributes = 2 (anger) + 0.5 (mask) = 2.5 (2) Item type analysis Identify guns (weight 10) and controlled knives (weight 5).

[0090] Assuming that no contraband is found in the captured image, then the comprehensive weight of contraband is: Contraband Weight: 0 (3) Behavior recognition analysis Through the crowd counting algorithm, it is identified that the number of people gathered exceeds a certain threshold (assuming the threshold is 5 people and the weight is 2).

[0091] Assume that in the captured image, there are 6 people gathered at the front door area. Then, the comprehensive weight of the gathering of people is: Personnel gathering weight: 2 Identify theft (weight 3), peeping (weight 3), and robbery (weight 5).

[0092] Suppose that in the captured image, there is a person stealing. Then, the comprehensive weight of the action is: Theft weight: 3 In this way, the weight of the facial attribute is 2.5, the weight of the object type is 0, the weight of the behavior recognition is 5, and the abnormal image risk weight value is 7.5.

[0093] Through step S402, this embodiment successfully calculates the abnormal image risk weight value of each camera, laying a solid foundation for the subsequent calculation of the comprehensive risk weight and identifying environmental anomalies and providing timely response.

[0094] In one embodiment of the abnormal event prediction method of the present application, see Figure 5 , and can also include the following: Step S501: performing a word segmentation operation on the recognition result output by the sound recognition model according to a preset word segmentation algorithm to determine the corresponding word segmentation vocabulary; Step S502: Matching the segmented vocabulary with a preset abnormal negative vocabulary corpus, and performing a weighted calculation operation based on the negative semantics obtained after the matching operation and the preset semantic weight, to determine the abnormal sound risk weight value corresponding to each camera.

[0095] Optionally, in this embodiment, a third-party speech recognition platform (such as iFlytek) is used to convert speech into text, and the converted text is matched with a negative emotion word library to identify whether it contains emotionally intense words (such as "kill", "swear", etc.). Emotionally intense words contained in the text are classified, such as anger, threat, intimidation, etc.

[0096] After identifying negative emotional words, one way is to assign different risk weights according to the emotional type, such as anger: weight 2, threat / violence: weight 5. Another way is to assign a weight of 1 to each negative word. Finally, the total abnormal sound risk weight value is the accumulation of each small sound risk.

[0097] For example, According to the first method, if "violence" related words appear in the environmental scenario, no matter how many times they appear, the weight value is 5. If no other negative words appear, the weight value is 0, and the abnormal sound risk weight value is 5+0=5.

[0098] According to the second method, in the environment scenario, assuming that in the last 10 seconds, the voice recognition system recognizes 5 negative words, then regardless of the type of negative words, the comprehensive weight of the sound is 5.

[0099] Through step S502, this embodiment successfully calculates the abnormal sound risk weight value of each camera, laying a solid foundation for the subsequent calculation of the comprehensive risk weight and identifying environmental anomalies and providing timely response.

[0100] In one embodiment of the abnormal event prediction method of the present application, see Figure 6 , and can also include the following: Step S601: determining whether the environmental noise is greater than a preset noise threshold, and if so, determining whether an abnormal image risk weight value corresponding to the first camera and an abnormal sound risk weight value corresponding to the first camera exist at the same time; Step S602: If both exist, performing a weight accumulation operation on the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera, and performing a multiplication operation on the accumulated result after the weight accumulation operation to determine the corresponding first comprehensive risk value; Step S603: If they do not exist at the same time, a weighted accumulation operation is performed on the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera to determine the corresponding first comprehensive risk value.

[0101] Optionally, in this embodiment, the noise sensor configured based on the camera is used to detect the environmental noise. The edge computing platform receives the environmental noise and determines whether the noise is greater than 60 decibels (the sound of a quarrel is greater than 60 decibels). If it is greater, the camera closest to the noise is first located, and the abnormal image risk weight value and abnormal sound risk weight value of the camera in the last 10 seconds are obtained (at this time, the edge computing platform has calculated the abnormal image risk weight value and abnormal sound risk weight value of each camera). At the same time, a judgment is made. If there is both an image risk weight value X and a sound risk weight value Y, the calculated weight is (X+Y). 2. Since there are both sound risk and image recognition risk, the risk must be high, so we multiply it. If there is only image risk X, or only sound risk Y, the combined risk weight is (X+Y).

[0102] Through step S603, this embodiment successfully obtains the comprehensive risk weight, laying the foundation for subsequent risk response based on the comprehensive risk weight value.

[0103] In one embodiment of the abnormal event prediction method of the present application, see Figure 7 , and can also include the following: Step S701: determining whether the first comprehensive risk value is greater than a preset first risk threshold, and if so, performing a warning information generation operation to determine corresponding warning information; Step S702: If it is less than, determine whether it is greater than half of the preset first risk threshold. If it is greater than, perform a warning information generation operation to determine the corresponding early warning information. If it is less than, perform a warning information generation operation to determine the corresponding prompt information.

[0104] Optionally, in this embodiment, the comprehensive risk value of the above step S603 is judged and corresponding corresponding measures are defined.

[0105] Specifically, if the comprehensive risk value is greater than 10, an alarm is generated and sent to the business platform, along with alarm information (alarm picture, alarm information, and alarm point).

[0106] Specifically, if the comprehensive risk value is less than 10 and greater than 5, an early warning message is generated and sent to the business platform, including early warning information (early warning picture, early warning information, and early warning points).

[0107] Specifically, if the comprehensive risk value is less than 5, a prompt voice will be sent to the camera (eg: keep quiet in public places, etc.).

[0108] Through step S702, this embodiment successfully performs real-time response based on comprehensive environmental risks, thereby improving the efficiency of abnormal event response.

[0109] In order to improve the response efficiency and flexibility of abnormal events, the present application provides an embodiment of an abnormal event prediction device for implementing all or part of the abnormal event prediction method, see Figure 8 The abnormal event prediction device specifically includes the following contents: The multi-model training module 10 is used to receive the learning materials of the business platform, perform model training operations according to the learning materials, and respectively determine the corresponding face recognition model, object recognition model, behavior recognition model and sound recognition model, wherein the learning materials include face recognition pictures, object recognition pictures, behavior recognition pictures and sound corpus information; The risk weight calculation module 20 is used to receive the environmental images captured by the multiple cameras, input the environmental images into the face recognition model, the object recognition model and the behavior recognition model, respectively determine the facial attribute information, object type information and behavior pattern information corresponding to each camera, perform weighted calculation operations on the facial attribute information, the object type information and the behavior pattern information according to preset abnormal event weights, respectively determine the abnormal image risk weight value corresponding to each camera, receive the environmental sounds captured by the multiple cameras, input the environmental sounds into the sound recognition model, perform a matching operation based on the recognition results output by the sound recognition model and a preset abnormal negative vocabulary corpus, and perform a weighted calculation operation based on the negative semantics obtained after the matching operation and the preset semantic weights, respectively determine the abnormal sound risk weight value corresponding to each camera; The warning information generation module 30 is used to receive the environmental noise recorded by the first camera, and determine whether the environmental noise is greater than a preset noise threshold. If greater than, a weight accumulation operation is performed according to the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera to determine the corresponding first comprehensive risk value, and determine whether the first comprehensive risk value is greater than the preset first risk threshold. According to the judgment result after the judgment operation, a corresponding warning message is generated, and the warning message is sent to the business platform so that the business platform responds abnormally according to the warning message, wherein the warning information includes alarm information, early warning information and prompt information, and the first camera is the camera closest to the noise source among the multiple cameras.

[0110] From the above description, it can be seen that the abnormal event prediction device provided in the embodiment of the present application can perform model training by receiving learning materials from the business platform, and determine the face recognition model, object recognition model, behavior recognition model and sound recognition model respectively; receive environmental images recorded by multiple cameras, input the environmental images into the face recognition model, object recognition model and behavior recognition model, and determine the abnormal image risk weight value corresponding to each camera respectively, receive environmental sounds recorded by multiple cameras, input the environmental sounds into the sound recognition model, and determine the abnormal sound risk weight value corresponding to each camera respectively; receive environmental noise recorded by the noise source camera, if the environmental noise is greater than the preset noise threshold, calculate the comprehensive risk value of the noise source camera, and generate corresponding warning information based on the comprehensive risk value judgment result, thereby improving the response efficiency and flexibility of abnormal events.

[0111] From a hardware perspective, in order to improve the efficiency and flexibility of responding to abnormal events, the present application provides an embodiment of an electronic device for implementing all or part of the content of the abnormal event prediction method, and the electronic device specifically includes the following content: Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the abnormal event prediction method and the core business system, user terminal and related database and other related devices; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., and the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the abnormal event prediction method in the embodiment, and the embodiment of the abnormal event prediction method, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0112] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0113] In practical applications, part of the abnormal event prediction method can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0114] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0115] Fig. 9 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig. 9 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0116] In one embodiment, the abnormal event prediction method function may be integrated into the central processing unit 9100. The central processing unit 9100 may be configured to perform the following control: Step S101: receiving learning materials from the business platform, performing model training operations according to the learning materials, and respectively determining corresponding face recognition models, object recognition models, behavior recognition models, and voice recognition models, wherein the learning materials include face recognition pictures, object recognition pictures, behavior recognition pictures, and voice corpus information; Step S102: receiving the environmental images captured by the multiple cameras, inputting the environmental images into the face recognition model, the object recognition model and the behavior recognition model, respectively determining the facial attribute information, object type information and behavior pattern information corresponding to each camera, performing weighted calculation operations on the facial attribute information, the object type information and the behavior pattern information according to preset abnormal event weights, respectively determining the abnormal image risk weight value corresponding to each camera, receiving the environmental sounds captured by the multiple cameras, inputting the environmental sounds into the sound recognition model, performing a matching operation based on the recognition results output by the sound recognition model and a preset abnormal negative vocabulary corpus, and performing a weighted calculation operation based on the negative semantics obtained after the matching operation and the preset semantic weights, respectively determining the abnormal sound risk weight value corresponding to each camera; Step S103: Receive the environmental noise recorded by the first camera, and determine whether the environmental noise is greater than a preset noise threshold; if so, perform a weight accumulation operation according to the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera to determine the corresponding first comprehensive risk value; determine whether the first comprehensive risk value is greater than the preset first risk threshold; and generate corresponding warning information according to the judgment result after the judgment operation, and send the warning information to the business platform so that the business platform performs an abnormal response according to the warning information, wherein the warning information includes alarm information, early warning information and prompt information, and the first camera is the camera closest to the noise source among the multiple cameras.

[0117] From the above description, it can be seen that the electronic device provided in the embodiment of the present application performs model training by receiving learning materials from the business platform, and determines the face recognition model, object recognition model, behavior recognition model and sound recognition model respectively; receives environmental images recorded by multiple cameras, inputs the environmental images into the face recognition model, the object recognition model and the behavior recognition model, and determines the abnormal image risk weight value corresponding to each camera respectively; receives environmental sounds recorded by multiple cameras, inputs the environmental sounds into the sound recognition model, and determines the abnormal sound risk weight value corresponding to each camera respectively; receives environmental noise recorded by the noise source camera, and if the environmental noise is greater than the preset noise threshold, calculates the comprehensive risk value of the noise source camera, and generates corresponding warning information based on the comprehensive risk value judgment result, thereby improving the response efficiency and flexibility of abnormal events.

[0118] In another embodiment, the abnormal event prediction method can be configured separately from the central processing unit 9100. For example, the abnormal event prediction method can be configured as a chip connected to the central processing unit 9100, and the function of the abnormal event prediction method is realized through the control of the central processing unit.

[0119] like Fig. 9 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig. 9 In addition, the electronic device 9600 may also include Fig. 9 For components not shown, reference may be made to the prior art.

[0120] like Fig. 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0121] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0122] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0123] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0124] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0125] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0126] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless local area network module, etc. The communication module 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the machine through the microphone 9132, and the sound stored on the machine can be played through the speaker 9131.

[0127] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the abnormal event prediction method in the above embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps of the abnormal event prediction method in the above embodiment, where the execution subject is a server or a client, are implemented. For example, when the processor executes the computer program, the following steps are implemented: Step S101: receiving learning materials from the business platform, performing model training operations according to the learning materials, and respectively determining corresponding face recognition models, object recognition models, behavior recognition models, and voice recognition models, wherein the learning materials include face recognition pictures, object recognition pictures, behavior recognition pictures, and voice corpus information; Step S102: receiving the environmental images captured by the multiple cameras, inputting the environmental images into the face recognition model, the object recognition model and the behavior recognition model, respectively determining the facial attribute information, object type information and behavior pattern information corresponding to each camera, performing weighted calculation operations on the facial attribute information, the object type information and the behavior pattern information according to preset abnormal event weights, respectively determining the abnormal image risk weight value corresponding to each camera, receiving the environmental sounds captured by the multiple cameras, inputting the environmental sounds into the sound recognition model, performing a matching operation based on the recognition results output by the sound recognition model and a preset abnormal negative vocabulary corpus, and performing a weighted calculation operation based on the negative semantics obtained after the matching operation and the preset semantic weights, respectively determining the abnormal sound risk weight value corresponding to each camera; Step S103: Receive the environmental noise recorded by the first camera, and determine whether the environmental noise is greater than a preset noise threshold; if so, perform a weight accumulation operation according to the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera to determine the corresponding first comprehensive risk value; determine whether the first comprehensive risk value is greater than the preset first risk threshold; and generate corresponding warning information according to the judgment result after the judgment operation, and send the warning information to the business platform so that the business platform performs an abnormal response according to the warning information, wherein the warning information includes alarm information, early warning information and prompt information, and the first camera is the camera closest to the noise source among the multiple cameras.

[0128] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application performs model training by receiving learning materials from the business platform, and determines the face recognition model, object recognition model, behavior recognition model and sound recognition model respectively; receives environmental images recorded by multiple cameras, inputs the environmental images into the face recognition model, the object recognition model and the behavior recognition model, and determines the abnormal image risk weight value corresponding to each camera respectively; receives environmental sounds recorded by multiple cameras, inputs the environmental sounds into the sound recognition model, and determines the abnormal sound risk weight value corresponding to each camera respectively; receives environmental noise recorded by the noise source camera, and if the environmental noise is greater than the preset noise threshold, calculates the comprehensive risk value of the noise source camera, and generates corresponding warning information based on the comprehensive risk value judgment result, thereby improving the response efficiency and flexibility of abnormal events.

[0129] The embodiments of the present application also provide a computer program product capable of implementing all the steps of the abnormal event prediction method in the above embodiments, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the abnormal event prediction method are implemented. For example, the computer program / instruction implements the following steps: Step S101: receiving learning materials from the business platform, performing model training operations according to the learning materials, and respectively determining corresponding face recognition models, object recognition models, behavior recognition models, and voice recognition models, wherein the learning materials include face recognition pictures, object recognition pictures, behavior recognition pictures, and voice corpus information; Step S102: receiving the environmental images captured by the multiple cameras, inputting the environmental images into the face recognition model, the object recognition model and the behavior recognition model, respectively determining the facial attribute information, object type information and behavior pattern information corresponding to each camera, performing weighted calculation operations on the facial attribute information, the object type information and the behavior pattern information according to preset abnormal event weights, respectively determining the abnormal image risk weight value corresponding to each camera, receiving the environmental sounds captured by the multiple cameras, inputting the environmental sounds into the sound recognition model, performing a matching operation based on the recognition results output by the sound recognition model and a preset abnormal negative vocabulary corpus, and performing a weighted calculation operation based on the negative semantics obtained after the matching operation and the preset semantic weights, respectively determining the abnormal sound risk weight value corresponding to each camera; Step S103: Receive the environmental noise recorded by the first camera, and determine whether the environmental noise is greater than a preset noise threshold; if so, perform a weight accumulation operation according to the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera to determine the corresponding first comprehensive risk value; determine whether the first comprehensive risk value is greater than the preset first risk threshold; and generate corresponding warning information according to the judgment result after the judgment operation, and send the warning information to the business platform so that the business platform performs an abnormal response according to the warning information, wherein the warning information includes alarm information, early warning information and prompt information, and the first camera is the camera closest to the noise source among the multiple cameras.

[0130] From the above description, it can be seen that the computer program product provided in the embodiment of the present application performs model training by receiving learning materials from the business platform, and determines the face recognition model, object recognition model, behavior recognition model and sound recognition model respectively; receives environmental images recorded by multiple cameras, inputs the environmental images into the face recognition model, the object recognition model and the behavior recognition model, and determines the abnormal image risk weight value corresponding to each camera respectively; receives environmental sounds recorded by multiple cameras, inputs the environmental sounds into the sound recognition model, and determines the abnormal sound risk weight value corresponding to each camera respectively; receives environmental noise recorded by the noise source camera, and if the environmental noise is greater than the preset noise threshold, calculates the comprehensive risk value of the noise source camera, and generates corresponding warning information based on the comprehensive risk value judgment result, thereby improving the response efficiency and flexibility of abnormal events.

[0131] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0133] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0135] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for predicting abnormal events, characterized in that: Applied to an edge computing platform, the edge computing platform is communicatively connected with a preset business platform and a plurality of preset cameras, wherein the plurality of cameras are configured with a voice device and a noise sensor for interacting with the environment, the method comprising: Receive learning materials from the business platform, perform model training operations according to the learning materials, and respectively determine corresponding face recognition models, object recognition models, behavior recognition models, and voice recognition models, wherein the learning materials include face recognition pictures, object recognition pictures, behavior recognition pictures, and voice corpus information; Receive the environmental images captured by the multiple cameras, input the environmental images into the face recognition model, the object recognition model and the behavior recognition model, respectively determine the facial attribute information, object type information and behavior pattern information corresponding to each camera, perform weighted calculation operations on the facial attribute information, the object type information and the behavior pattern information according to preset abnormal event weights, respectively determine the abnormal image risk weight value corresponding to each camera, receive the environmental sounds captured by the multiple cameras, input the environmental sounds into the sound recognition model, perform a matching operation based on the recognition results output by the sound recognition model and a preset abnormal negative vocabulary corpus, and perform a weighted calculation operation based on the negative semantics obtained after the matching operation and the preset semantic weights, respectively determine the abnormal sound risk weight value corresponding to each camera; Receive the environmental noise recorded by the first camera, and determine whether the environmental noise is greater than a preset noise threshold; if so, perform a weight accumulation operation according to the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera to determine the corresponding first comprehensive risk value; determine whether the first comprehensive risk value is greater than the preset first risk threshold, and generate corresponding warning information according to the judgment result after the judgment operation; send the warning information to the business platform, so that the business platform performs an abnormal response according to the warning information, wherein the warning information includes alarm information, early warning information and prompt information, and the first camera is the camera closest to the noise source among the multiple cameras.

2. The abnormal event prediction method according to claim 1, characterized in that: The method further comprises: According to a preset time interval, polling is performed to determine whether the comprehensive risk value corresponding to each camera is greater than a preset second risk threshold, wherein the comprehensive risk value corresponding to each camera is determined by performing a weighted accumulation operation based on the abnormal image risk weight value corresponding to each camera and the abnormal sound risk weight value corresponding to each camera; If it is greater, an alarm information generation operation is performed, and the alarm information is sent to the business platform, so that the business platform performs an abnormal response according to the alarm information; If not, the comprehensive risk values ​​corresponding to each camera are accumulated to determine whether the comprehensive risk values ​​corresponding to all cameras obtained after the accumulation operation are greater than the preset third risk threshold; if so, an alarm information generation operation is performed and the alarm information is sent to the business platform so that the business platform responds to the exception according to the alarm information.

3. The abnormal event prediction method according to claim 1, characterized in that: The model training operation is performed according to the learning data to respectively determine the corresponding face recognition model, object recognition model, behavior recognition model and sound recognition model, including: Performing a model training operation on a preset Fast RCNN network according to the face recognition images in the learning material to determine a corresponding face recognition model, wherein the face recognition model includes a face detection module, a face normalization module, a face attribute recognition module, a face quality assessment module, and a face tracking module, and the face recognition model is used to recognize faces and analyze people's negative emotions to obtain face attribute information; Performing a model training operation on a preset convolutional neural network according to the object recognition images in the learning material to determine a corresponding object recognition model, wherein the object recognition model is used to identify objects that endanger environmental safety; Performing a model training operation on a preset Crowd Counting algorithm according to the behavior recognition pictures in the learning material to determine a corresponding behavior recognition model, wherein the behavior recognition model is used to detect crowd gathering and behaviors that endanger environmental safety; A model training operation is performed on a preset semantic recognition initial model according to the sound corpus information in the learning material to determine a corresponding sound recognition model, wherein the sound recognition model is used to convert sound into semantic information.

4. The abnormal event prediction method according to claim 1, characterized in that: The step of performing a weighted calculation operation on the face attribute information, the object type information, and the behavior pattern information according to the preset abnormal event weights to respectively determine the abnormal image risk weight value corresponding to each camera includes: Performing a weighting operation on the facial attribute information, the item type information, and the behavior pattern information according to the preset abnormal event weights, and determining the facial attribute information weight, the item type information weight, and the behavior pattern information weight corresponding to each camera; A weight accumulation operation is performed according to the facial attribute information weight, the item type information weight, and the behavior pattern information weight, to respectively determine the abnormal image risk weight value corresponding to each camera.

5. The abnormal event prediction method according to claim 1, characterized in that: The matching operation is performed based on the recognition result output by the sound recognition model and the preset abnormal negative vocabulary corpus, and the weighted calculation operation is performed based on the negative semantics obtained after the matching operation and the preset semantic weight, to respectively determine the abnormal sound risk weight value corresponding to each camera, including: Performing a word segmentation operation on the recognition result output by the sound recognition model according to a preset word segmentation algorithm to determine the corresponding word segmentation vocabulary; The segmented vocabulary and the preset abnormal negative vocabulary corpus are matched, and a weighted calculation operation is performed according to the negative semantics obtained after the matching operation and the preset semantic weight, so as to determine the abnormal sound risk weight value corresponding to each camera respectively.

6. The abnormal event prediction method according to claim 1, characterized in that: The determining whether the environmental noise is greater than a preset noise threshold, and if so, performing a weight accumulation operation according to the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera to determine the corresponding first comprehensive risk value, includes: Determine whether the environmental noise is greater than a preset noise threshold, and if so, determine whether the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera exist at the same time; If both exist, a weighted accumulation operation is performed on the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera, and a multiplication operation is performed on the accumulated result after the weighted accumulation operation to determine the corresponding first comprehensive risk value; If they do not exist at the same time, the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera are weighted and accumulated to determine the corresponding first comprehensive risk value.

7. The abnormal event prediction method according to claim 1, characterized in that: The determining whether the first comprehensive risk value is greater than a preset first risk threshold, and generating corresponding warning information according to the determination result after the determination operation, includes: Determine whether the first comprehensive risk value is greater than a preset first risk threshold, and if so, generate a warning message to determine the corresponding warning message; If it is less than, determine whether it is greater than half of the preset first risk threshold. If it is greater, perform a warning information generation operation to determine the corresponding early warning information. If it is less than, perform a warning information generation operation to determine the corresponding prompt information.

8. An abnormal event prediction device, characterized in that: The device comprises: A multi-model training module, used to receive learning materials from the business platform, perform model training operations according to the learning materials, and respectively determine corresponding face recognition models, object recognition models, behavior recognition models, and voice recognition models, wherein the learning materials include face recognition pictures, object recognition pictures, behavior recognition pictures, and voice corpus information; A risk weight calculation module, for receiving environmental images captured by multiple cameras, inputting the environmental images into the face recognition model, the object recognition model and the behavior recognition model, respectively determining the facial attribute information, object type information and behavior pattern information corresponding to each camera, performing weighted calculation operations on the facial attribute information, the object type information and the behavior pattern information according to preset abnormal event weights, respectively determining the abnormal image risk weight value corresponding to each camera, receiving environmental sounds captured by multiple cameras, inputting the environmental sounds into the sound recognition model, performing a matching operation based on the recognition results output by the sound recognition model and a preset abnormal negative vocabulary corpus, and performing a weighted calculation operation based on the negative semantics obtained after the matching operation and the preset semantic weights, respectively determining the abnormal sound risk weight value corresponding to each camera; A warning information generation module is used to receive the environmental noise recorded by the first camera, determine whether the environmental noise is greater than a preset noise threshold, and if so, perform a weight accumulation operation according to the abnormal image risk weight value corresponding to the first camera and the abnormal sound risk weight value corresponding to the first camera to determine the corresponding first comprehensive risk value, determine whether the first comprehensive risk value is greater than the preset first risk threshold, and generate corresponding warning information according to the judgment result after the judgment operation, and send the warning information to the business platform so that the business platform performs an abnormal response according to the warning information, wherein the warning information includes alarm information, early warning information and prompt information, and the first camera is the camera closest to the noise source among the preset multiple cameras.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the abnormal event prediction method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the abnormal event prediction method according to any one of claims 1 to 7 are implemented.

Images