A multimedia monitoring method, device and storage medium based on visual detection

Abstract

The present application discloses a multimedia monitoring method, device and storage medium based on visual detection. The method includes: obtaining a first image sequence captured by a camera from a camera; extracting a target video frame containing a target object from the first image sequence, and sorting the target video frames according to the time sequence to form a second image sequence containing the target object; inputting the target video frames in the second image sequence into a pre-trained emotion recognition model in sequence, determining the emotion vectors corresponding to the target objects in each target video frame, so that the emotion vectors corresponding to the multiple target video frames constitute an emotion vector sequence, wherein the multiple elements in the emotion vector respectively indicate the probability values ​​corresponding to different types of emotions; sliding the emotion vector sequence using a sliding window, and constructing an emotion matrix corresponding to the target object using the emotion vectors contained in the sliding window; and determining whether the target object is abnormal according to the emotion matrix.

Description

Technical Field

[0001] The present application relates to the field of security technology, and in particular to a multimedia monitoring method, device and storage medium based on visual detection. Background Art

[0002] Multimedia monitoring systems based on visual detection have now been widely used in the security field. For example, by laying cameras in a community, videos from various locations in the community can be collected in real time. Then, by deploying edge devices to connect to each camera, the video collected by each camera can be obtained in real time. The edge device can then be used to analyze the video collected by each camera, and the video that may have anomalies can be uploaded to the cloud server.

[0003] In addition, with the help of deep learning-based recognition models (such as face recognition models) deployed on edge devices, edge devices can identify strangers entering the community, and by identifying the same stranger in videos captured by cameras deployed at different locations, the behavior trajectory of strangers in the community can be tracked.

[0004] In addition, with the help of the deep learning-based recognition model deployed on the edge device, the emotional state of the stranger can be analyzed and identified in real time, so as to determine whether the stranger is abnormal and whether it will bring security risks to the community based on the results of the analysis and identification.

[0005] At present, the recognition model based on deep learning is usually a neural network-based classifier, which receives an image containing a target object (such as a stranger) and determines the probability values ​​of multiple different emotions corresponding to the target object based on the image. The multiple emotions can be, for example, eight basic emotions such as happiness, trust, fear, surprise, sadness, disgust, anger and expectation. Therefore, when the probability of a certain emotion is the highest, it is determined that the target object in the image is in a certain emotion. Therefore, it is further determined whether the target object is abnormal based on the determined emotion.

[0006] However, this method requires that the probability value corresponding to a certain emotion output by the classifier is significantly higher than the probability values ​​of other emotions in order to more accurately identify the target object's emotions. However, in many cases, surveillance cameras can only capture a general image of the target object rather than a close-up of the target object, and before the target object clearly reveals abnormal emotions, its emotional expression is not very obvious. Therefore, in this case, the probability values ​​corresponding to different emotions contained in the emotional information output by the classifier are not much different. For example, the difference between the probability values ​​corresponding to the emotion with the largest probability value and the emotion with the second highest probability value is less than a predetermined threshold. In this case, the target object's emotions cannot be effectively identified, and it is impossible to determine whether the target object is abnormal.

[0007] With respect to the technical problem in the prior art mentioned above that when the emotion of the target object cannot be determined by the classification-based emotion recognition model, it is impossible to determine whether the target object is abnormal. No effective solution has been proposed so far. Summary of the invention

[0008] The embodiments of the present disclosure provide a multimedia monitoring method, device and storage medium based on visual detection, so as to at least solve the technical problem existing in the prior art that when the emotion of the target object cannot be determined by a classification-based emotion recognition model, it is impossible to determine whether the target object is abnormal.

[0009] According to one aspect of an embodiment of the present disclosure, a multimedia monitoring method based on visual detection is provided, including: acquiring a first image sequence captured by a camera from a camera; extracting target video frames containing a target object from the first image sequence, and sorting the target video frames according to time order to form a second image sequence containing the target object; inputting the target video frames in the second image sequence into a pre-trained emotion recognition model in sequence, determining emotion vectors corresponding to the target objects in each target video frame, so that the emotion vectors corresponding to multiple target video frames constitute an emotion vector sequence, wherein multiple elements in the emotion vector respectively indicate probability values ​​corresponding to different types of emotions; sliding a sliding window in the emotion vector sequence, and constructing an emotion matrix corresponding to the target object using the emotion vectors contained in the sliding window; and determining whether the target object is abnormal according to the emotion matrix.

[0010] According to another aspect of the embodiments of the present disclosure, a storage medium is further provided. The storage medium includes a stored program, wherein the above method is executed by a processor when the program is running.

[0011] According to another aspect of the embodiment of the present disclosure, a multimedia monitoring device based on visual detection is also provided, including: a first image sequence acquisition module, used to acquire a first image sequence captured by a camera from a camera; a second image sequence construction module, used to extract target video frames containing a target object from the first image sequence, and sort the target video frames according to time order to form a second image sequence containing the target object; an emotion vector determination module, used to input the target video frames in the second image sequence into a pre-trained emotion recognition model in sequence, and determine the emotion vectors corresponding to the target objects in each target video frame, so that the emotion vectors corresponding to multiple target video frames constitute an emotion vector sequence, wherein multiple elements in the emotion vector respectively indicate probability values ​​corresponding to different types of emotions; an emotion matrix determination module, used to slide the emotion vector sequence using a sliding window, and use the emotion vectors contained in the sliding window to construct an emotion matrix corresponding to the target object; and an abnormality determination module, used to determine whether the target object is abnormal according to the emotion matrix.

[0012] According to another aspect of the embodiment of the present disclosure, a multimedia monitoring device based on visual detection is also provided, including: a processor; and a memory, connected to the processor, for providing the processor with instructions for processing the following processing steps: obtaining a first image sequence captured by a camera from a camera; extracting target video frames containing a target object from the first image sequence, and sorting the target video frames according to time order to form a second image sequence containing the target object; inputting the target video frames in the second image sequence into a pre-trained emotion recognition model in sequence, determining the emotion vectors corresponding to the target objects in each target video frame, so that the emotion vectors corresponding to multiple target video frames constitute an emotion vector sequence, wherein multiple elements in the emotion vector respectively indicate probability values ​​corresponding to different types of emotions; using a sliding window to slide in the emotion vector sequence, and using the emotion vectors contained in the sliding window to construct an emotion matrix corresponding to the target object; and determining whether the target object is abnormal based on the emotion matrix.

[0013] In the disclosed embodiment, the target video frames containing the target object are sequentially input into the emotion recognition model, so as to determine the emotion vectors corresponding to the target object in each target video frame, and then the emotion vectors corresponding to each target video frame constitute an emotion vector sequence. The emotion vector sequence can be regarded as multi-channel time series information, and the time series information of each channel reflects the change of the corresponding emotion at different times. Then, a sliding window of a predetermined width is used to slide on the vector sequence, so that an emotion matrix of a corresponding size can be defined by the sliding window. Then, it is determined whether the target object is abnormal according to the emotion matrix. Since the emotion matrix contains the change information of different emotions of the target object over time, even if the numerical values ​​(such as probability values) corresponding to each emotion in the emotion vector are not much different and cannot be classified, the information of the emotion matrix can be used to predict the abnormal situation of the target object. Thus, the technical problem that the target object cannot be determined whether it is abnormal in the prior art when the emotion of the target object cannot be determined by the classification-based emotion recognition model is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present application. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0015] Figure 1 is a hardware structure block diagram of a computing device for implementing the method according to Embodiment 1 of the present disclosure;

[0016] Figure 2 is a schematic diagram of a multimedia monitoring system based on visual detection according to Embodiment 1 of the present disclosure;

[0017] Figure 3 is a flowchart of a multimedia monitoring method based on visual detection according to the first aspect of Embodiment 1 of the present disclosure;

[0018] Figure 4 is a schematic diagram of an edge device acquiring a first image sequence according to the first aspect of Embodiment 1 of the present disclosure;

[0019] Figure 5 is a schematic diagram of constructing a second image sequence by an edge device according to the first aspect of Embodiment 1 of the present disclosure;

[0020] Figure 6 is a schematic diagram of an edge device according to the first aspect of Embodiment 1 of the present disclosure generating an emotion vector sequence based on a second image sequence using an emotion recognition model;

[0021] Figure 7 is a schematic diagram of an emotion recognition model according to the first aspect of Embodiment 1 of the present disclosure;

[0022] Figure 8 is a schematic diagram of an edge device sliding a sliding window on an emotion vector sequence according to the first aspect of Embodiment 1 of the present disclosure;

[0023] Fig. 9 is a schematic diagram of a multimedia monitoring device based on visual detection according to Embodiment 2 of the present disclosure; and

[0024] Fig.10 It is a schematic diagram of a multimedia monitoring device based on visual detection according to Example 3 of the present disclosure. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0027] Example 1

[0028] According to this embodiment, a method embodiment of a multimedia monitoring method based on visual detection is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] The method embodiment provided in this embodiment can be executed in a server or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computing device for implementing a multimedia monitoring method based on visual detection. Figure 1 As shown, the computing device may include one or more processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory for storing data, a transmission device for communication functions, and an input / output interface. The memory, the transmission device, and the input / output interface are connected to the processor via a bus. In addition, it may also include: a display, a keyboard, and a cursor control device connected to the input / output interface. A person skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0030] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computing device. As involved in the embodiments of the present disclosure, the data processing circuitry acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0031] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the multimedia monitoring method based on visual detection in the embodiment of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the multimedia monitoring method based on visual detection of the above-mentioned application is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computing device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computing device. In one example, the transmission device includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0033] The display may be, for example, a touch screen liquid crystal display (LCD) that may enable a user to interact with a user interface of the computing device.

[0034] It should be noted that, in some optional embodiments, the above Figure 1 The computing device shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computing devices described above.

[0035] Figure 2is a schematic diagram of a multimedia monitoring system based on visual detection according to this embodiment. Figure 2 As shown, the system includes: a cloud server 100 and multiple edge devices 200. The edge devices 200 are respectively deployed in corresponding cells 300 and are communicatively connected with multiple cameras arranged in the corresponding cells 300. The cloud server 100 communicates with each edge device 200 via the Internet.

[0036] Thus, the edge device 200 can obtain the surveillance video collected by the camera of the corresponding community, and use the deep learning-based emotion recognition model deployed on the edge device 200 to determine the emotional information of the target object (stranger) in the surveillance video. The emotional information can be represented in the form of a vector, for example, so as to include the probability values ​​of different types of emotions. In this embodiment, the different types of emotions can be, for example, eight basic emotions such as happiness, trust, fear, surprise, sadness, disgust, anger and expectation. Thus, the edge device 200 can determine the emotional information corresponding to the target object in each video frame containing the target object in the surveillance video.

[0037] The edge device 200 can then determine whether the target object is abnormal based on the emotion information, and upload the video of the abnormal target object to the cloud server 100.

[0038] It should be noted that the edge device 200 and the server 100 in the system can both be applicable to the hardware structure described above.

[0039] Under the above operating environment, according to the first aspect of this embodiment, a multimedia monitoring method based on visual detection is provided, the method comprising: Figure 2 The edge device implementation shown in . Figure 3 A schematic diagram showing the process of the method is shown in FIG. Figure 3 As shown, the method includes:

[0040] S302: Acquire a first image sequence captured by the camera from the camera;

[0041] S304: extracting target video frames containing the target object from the first image sequence, and sorting the target video frames according to a time sequence to form a second image sequence containing the target object;

[0042] S306: inputting the target video frames in the second image sequence into a pre-trained emotion recognition model in sequence, determining the emotion vectors corresponding to the target objects in the respective target video frames, so that the emotion vectors corresponding to the plurality of target video frames constitute an emotion vector sequence, wherein the plurality of elements in the emotion vectors respectively indicate probability values ​​corresponding to different types of emotions;

[0043] S308: Slide the emotion vector sequence using a sliding window, and construct an emotion matrix corresponding to the target object using the emotion vectors included in the sliding window; and

[0044] S310: Determine whether the target object is abnormal according to the emotion matrix.

[0045] Specifically, refer to Figure 4 As shown, in a cell, multiple cameras transmit captured image sequences to the edge device 200, so that the edge device obtains the captured image sequence from each camera, that is, the first image sequence (S302).

[0046] Then, when the edge device 200 detects a stranger in the first image sequence, the stranger is taken as the target object and a target video frame containing the target object is extracted from the first image sequence. Figure 4 As shown, since the first image sequence is an image sequence acquired by the edge device 200 from multiple cameras respectively. Figure 5 As shown, the first image sequences captured by each camera may include a target video frame containing the target object. The edge device 200 can, for example, use a known target detection model, such as Faster-Rcnn or YOLOv5, to extract the target video frame containing the target object from the multiple first image sequences. This will not be repeated here.

[0047] Thus, the edge device 200 extracts target video frames from each first image sequence in time sequence, and sorts the target video frames extracted from each first image sequence in time sequence, thereby forming a second image sequence (S304).

[0048] Then, refer to Figure 6 As shown, the edge device 200 sequentially inputs the target video frames in the second image sequence into the pre-trained emotion recognition model to determine the emotion vectors corresponding to the target video frames, and the emotion vectors corresponding to the target video frames sequentially constitute an emotion vector sequence.

[0049] Among them, reference Figure 7 As shown, the emotion recognition model, for example, first cuts out an image region containing a target object (i.e., the region of the detection frame corresponding to the target object) from a target video frame, and then inputs the image region into a classification model based on a convolutional neural network. Thus, an emotion vector corresponding to the target object in the target video frame is output. Specifically, the emotion vector can be expressed as:

[0050] Q=[q1,q2,q3,q4,q5,q6,...,q m ] T .

[0051] Among them, q1~q m For example, it may be probability values ​​corresponding to m different emotions. In this embodiment, it may be probability values ​​corresponding to 8 different basic emotions. Of course, m may also take other different values ​​according to actual needs.

[0052] Therefore, after the edge device 200 sequentially inputs the second image sequence into the emotion recognition model, it sequentially outputs the corresponding emotion vectors, thereby forming the following Figure 6 The emotion vector sequence (S306) is shown.

[0053] Then, refer to Figure 8 As shown, the edge device 200 slides on the emotion vector sequence using a preset sliding window (e.g., a sliding window with a width of n) to determine a series of emotion matrices containing n emotion vectors. In other words, the data defined by each sliding window is an m×n matrix (i.e., an emotion matrix). Thus, each time the sliding window slides one step on the emotion vector sequence, an m×n matrix is ​​defined (S308).

[0054] Each emotion matrix can be regarded as a multi-channel (i.e., m channels) emotion sequence data. That is, each row of the emotion matrix can be regarded as a time series data of an emotion fluctuating over time. Therefore, each row of the emotion matrix can be regarded as a multi-channel emotion sequence data composed of time series data of different emotions.

[0055] Then, the edge device 200 further determines whether the target object is abnormal based on the emotion matrix (S310). The specific method of the edge device determining whether the target object is abnormal based on the emotion matrix will be described below.

[0056] As described in the background technology, the existing technology needs to more accurately identify the target object's emotions when the probability value corresponding to a certain emotion output by the classifier is significantly higher than the probability values ​​of other emotions. However, in many cases, the surveillance camera can only capture ordinary images of the target object instead of close-ups of the target object, and before the target object obviously reveals abnormal emotions, its emotional expression is not very obvious. Therefore, in this case, the probability values ​​corresponding to different emotions contained in the emotional information output by the classifier are not much different. For example, the difference between the probability values ​​corresponding to the emotion with the largest probability value and the emotion with the second highest probability value is less than a predetermined threshold. In this case, the target object's emotions cannot be effectively identified, and it is impossible to determine whether the target object is abnormal.

[0057] In view of this, the present embodiment sequentially inputs the target video frames containing the target object into the emotion recognition model, thereby determining the emotion vectors corresponding to the target object in each target video frame, and then the emotion vectors corresponding to each target video frame constitute an emotion vector sequence. The emotion vector sequence can be regarded as multi-channel time series information, and the time series information of each channel reflects the change of the corresponding emotion at different times. Then, a sliding window of a predetermined width is used to slide on the vector sequence, so that an emotion matrix of a corresponding size can be defined by the sliding window. Then, it is determined whether the target object is abnormal based on the emotion matrix. Since the emotion matrix contains the change information of different emotions of the target object over time, even if the numerical values ​​(such as probability values) corresponding to each emotion in the emotion vector are not much different and cannot be classified, the information of the emotion matrix can be used to predict the abnormal situation of the target object. Thus, the technical problem that the target object cannot be determined whether it is abnormal in the prior art when the emotion of the target object cannot be determined by the classification-based emotion recognition model is solved.

[0058] Optionally, the operation of extracting target video frames containing the target object from the first image sequence and sorting the target video frames according to time sequence to form a second image sequence containing the target object includes: extracting target video frames containing the target object from multiple first image sequences corresponding to multiple cameras respectively; and sorting the target video frames according to time sequence to form a second image sequence containing the target object.

[0059] As mentioned above, reference Figure 4 As shown, the first image sequence may be a plurality of first image sequences acquired by the edge device 200 from a plurality of cameras respectively. Figure 5 As shown, the first image sequences captured by each camera may include a target video frame containing the target object. Thus, the edge device 200 extracts the target video frames from each first image sequence in chronological order, and sorts the target video frames extracted from each first image sequence in chronological order, thereby forming a second image sequence.

[0060] In this way, even if the target object is moving in the community and a single camera cannot capture a complete image sequence of the target object, the target video frame can be extracted from the image sequence captured by multiple cameras to obtain an image sequence that contains the target object and is continuous and complete in time, which facilitates the analysis of the target object.

[0061] Optionally, the operation of determining whether the target object is abnormal based on the emotion matrix includes: determining a covariance matrix reflecting the correlation between different emotions of the target object based on the emotion matrix and the transposed matrix of the emotion matrix; performing feature extraction based on the covariance matrix to determine an emotion feature vector reflecting the emotional characteristics of the target object; and determining whether the target object is abnormal based on the emotion feature vector.

[0062] Specifically, the edge device 200 defines the emotion matrix at each position on the emotion vector sequence through a sliding window:

[0063] That is, Q∈R m×n .

[0064] where q i,j (i=1~m, j=1~n) represents the emotion information (i.e., probability value) of the i-th type of emotion of the target object at the j-th moment in the emotion vector sequence defined by the emotion matrix. Thus, each row of data q in the emotion matrix i,1 ~q i,n Represents the data of a channel, reflecting the fluctuation of the emotional information of the i-th emotional type at time 1 to n.

[0065] Then, the edge device 200 calculates the covariance matrix of the emotion matrix Q according to the following formula:

[0066] where P∈R m×m .

[0067] In which, let R = QQ T ,Right now:

[0068]

[0069] If Q is expressed as Among them, Q i =[q i,1 q i,2 ...q i,m ],

[0070] Then for each element r in the matrix R i,j (i=1~m, j=1~m), we have:

[0071] r i,j =Q i Q j T .

[0072] Therefore, each element in the matrix R reflects the correlation between the fluctuations of different types of emotional information. Therefore, the covariance matrix Q can also reflect the correlation between different types of emotional changes.

[0073] Then, the edge device performs feature extraction based on the covariance matrix P to determine an emotional feature vector F reflecting the emotional features of the target object, and determines whether the target object is abnormal based on the emotional feature vector F. The specific determination method will be described in detail below.

[0074] Therefore, the technical solution of this embodiment extracts the emotional feature vector from the covariance matrix that can reflect the correlation between different types of emotional fluctuations, so that information reflecting whether the target object is abnormal can be extracted based on the correlation between different types of emotional fluctuations. In this way, when the emotional information of various types of emotions is not very different, the information used to determine whether the target object is abnormal can be accurately captured, which is conducive to accurately predicting the state of the target object.

[0075] Optionally, the operation of performing feature extraction based on the covariance matrix to determine an emotion feature vector reflecting the emotion characteristics of the target object includes: using a pre-trained projection matrix to project the covariance matrix into a corresponding emotion feature vector.

[0076] Specifically, after the edge device 200 calculates the covariance matrix P corresponding to the emotion matrix Q, it can use the pre-trained projection matrix W∈R m×2L , project the covariance matrix P into the corresponding emotion feature vector F∈R 2L×1 .

[0077] Specifically, the emotion feature vector F can be calculated by the following formula:

[0078] F=diag(W T PW).

[0079] Where diag() means taking diagonal elements; F = [f1,f2,f3,...,f 2L ].

[0080] Optionally, the method also includes: providing a sample set for training the projection matrix, wherein the sample set includes a plurality of first emotion matrix samples corresponding to abnormal target objects and a plurality of second emotion matrix samples corresponding to normal target objects; determining a first covariance matrix mean of the covariance matrix of the first emotion matrix samples, and determining a second covariance matrix mean of the covariance matrix of the second emotion matrix samples; and determining the projection matrix based on the first covariance matrix mean and the second covariance matrix mean.

[0081] Specifically, the projection matrix W can be trained as follows:

[0082] First, prepare a sample set for training the projection matrix. The sample set includes a plurality of first emotion matrix samples {Qa1, Qa2, Qa3, ...}, where Qa1, Qa2, Qa3, ... are all ∈ R m×n , and corresponds to a normal target object; in addition, the sample set includes a plurality of second emotion matrix samples {Qb1, Qb2, Qb3, ...}, where Qb1, Qb2, Qb3, ... are all ∈ R m×n , and corresponds to the target object of the exception.

[0083] Then, the covariance matrices of the multiple first emotion matrix samples Qa1, Qa2, Qa3, ... are calculated, and the average values ​​of the covariance matrices are calculated to obtain the first covariance matrix mean Pa. And, the covariance matrices of the multiple second emotion matrix samples Qb1, Qb2, Qb3, ... are calculated, and the average values ​​of the covariance matrices are calculated to obtain the second covariance matrix mean Pb.

[0084] Then, the projection matrix W can be determined using the following formula:

[0085]

[0086] Optionally, the operation of determining whether the target object is abnormal based on the emotion feature vector includes: inputting the emotion feature vector into a pre-trained binary classification model, and using the binary classification model to determine whether the target object is abnormal.

[0087] Specifically, the edge device 200 calculates the emotion feature vector F = [f1, f2, f3, ..., f 2L ], the emotion feature vector is input into a pre-trained binary classification model, so as to use the binary classification model to determine whether the target object is abnormal. Specifically, the binary classification model can adopt a logistic regression model, as shown below:

[0088] g=k0+k1×f1+k2×f2+......+k 2L ×f 2L

[0089]

[0090] Among them, g is the linear regression output result used for the logistic regression model, h is the probability that the target object is abnormal, and when h is greater than 0.5, the target object is judged to be abnormal.

[0091] Therefore, according to the method described above, the sliding window is slid on the emotion vector sequence, so that each sliding step will define an emotion matrix, so that the edge device 200 can analyze the emotion matrix defined by each sliding step of the sliding window to determine whether the target object is abnormal.

[0092] Optionally, the method further includes: in a case where the target object is determined to be abnormal, sending the second image sequence to a cloud server.

[0093] Specifically, when the edge device 200 determines that the target object is abnormal, it sends the second image sequence to the cloud server 100 through the Internet. Thus, the cloud server 100 can analyze and record the target object with abnormality in time.

[0094] In addition, reference Figure 1 As shown, according to the second aspect of this embodiment, a storage medium is provided, wherein the storage medium includes a stored program, wherein when the program is run, a processor executes any one of the above methods.

[0095] Thus, according to this embodiment, the target video frames containing the target object are sequentially input into the emotion recognition model, so as to determine the emotion vectors corresponding to the target object in each target video frame, and then the emotion vectors corresponding to each target video frame constitute an emotion vector sequence. The emotion vector sequence can be regarded as multi-channel time series information, and the time series information of each channel reflects the change of the corresponding emotion at different times. Then, a sliding window of a predetermined width is used to slide on the vector sequence, so that an emotion matrix of a corresponding size can be defined by the sliding window. Then, it is determined whether the target object is abnormal according to the emotion matrix. Since the emotion matrix contains the change information of different emotions of the target object over time, even if the numerical values ​​(such as probability values) corresponding to each emotion in the emotion vector are not much different and cannot be classified, the abnormal situation of the target object can be predicted using the information of the emotion matrix. Thus, the technical problem that the target object cannot be determined whether it is abnormal in the prior art when the emotion of the target object cannot be determined by the classification-based emotion recognition model is solved.

[0096] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0098] Example 2

[0099] Fig. 9 A multimedia monitoring device 900 based on visual detection according to this embodiment is shown, and the device 900 corresponds to the method according to the first aspect of embodiment 1. Fig. 9 As shown, the device 900 includes: a first image sequence acquisition module 910, which is used to acquire a first image sequence captured by a camera from a camera; a second image sequence construction module 920, which is used to extract target video frames containing a target object from the first image sequence, and sort the target video frames according to the time sequence to form a second image sequence containing the target object; an emotion vector determination module 930, which is used to input the target video frames in the second image sequence into a pre-trained emotion recognition model in sequence, and determine the emotion vectors corresponding to the target objects in each target video frame, so that the emotion vectors corresponding to the multiple target video frames constitute an emotion vector sequence, wherein the multiple elements in the emotion vector respectively indicate the probability values ​​corresponding to different types of emotions; an emotion matrix determination module 940, which is used to slide in the emotion vector sequence using a sliding window, and construct an emotion matrix corresponding to the target object using the emotion vectors contained in the sliding window; and an abnormality determination module 950, which is used to determine whether the target object is abnormal according to the emotion matrix.

[0100] Optionally, the second image sequence construction module 920 includes: a target video frame extraction submodule, used to extract target video frames containing the target object from multiple first image sequences corresponding to multiple cameras respectively; and a second image sequence construction submodule, used to sort the target video frames according to time sequence to form a second image sequence containing the target object.

[0101] Optionally, the abnormality determination module 950 includes: a covariance matrix determination submodule, which is used to determine a covariance matrix reflecting the correlation between different emotions of the target object based on the emotion matrix and the transposed matrix of the emotion matrix; a feature extraction submodule, which is used to perform feature extraction based on the covariance matrix to determine an emotion feature vector reflecting the emotion characteristics of the target object; and an abnormality determination submodule, which is used to determine whether the target object is abnormal based on the emotion feature vector.

[0102] Optionally, the feature extraction submodule includes a projection unit, which is used to project the covariance matrix into a corresponding emotion feature vector using a pre-trained projection matrix.

[0103] Optionally, the abnormality determination submodule includes a binary classification unit, which is used to input the emotion feature vector into a pre-trained binary classification model, and use the binary classification model to determine whether the target object is abnormal.

[0104] Optionally, the device further includes a transmission module, for sending the second image sequence to a cloud server when the target object is determined to be abnormal.

[0105] Thus, according to this embodiment, the target video frames containing the target object are sequentially input into the emotion recognition model, so as to determine the emotion vectors corresponding to the target object in each target video frame, and then the emotion vectors corresponding to each target video frame constitute an emotion vector sequence. The emotion vector sequence can be regarded as multi-channel time series information, and the time series information of each channel reflects the change of the corresponding emotion at different times. Then, a sliding window of a predetermined width is used to slide on the vector sequence, so that an emotion matrix of a corresponding size can be defined by the sliding window. Then, it is determined whether the target object is abnormal according to the emotion matrix. Since the emotion matrix contains the change information of different emotions of the target object over time, even if the numerical values ​​(such as probability values) corresponding to each emotion in the emotion vector are not much different and cannot be classified, the abnormal situation of the target object can be predicted using the information of the emotion matrix. Thus, the technical problem that the target object cannot be determined whether it is abnormal in the prior art when the emotion of the target object cannot be determined by the classification-based emotion recognition model is solved.

[0106] Example 3

[0107] Fig.10 FIG. 1 shows a multimedia monitoring device 1000 based on visual detection according to this embodiment, and the device 1000 corresponds to the method described in the first aspect of Embodiment 1. Fig.10As shown, the device 1000 includes: a processor 1010; and a memory 1020, which is connected to the processor 1010 and is used to provide the processor 1010 with instructions for processing the following processing steps: obtaining a first image sequence captured by a camera from a camera; extracting target video frames containing a target object from the first image sequence, and sorting the target video frames according to a time sequence to form a second image sequence containing the target object; inputting the target video frames in the second image sequence into a pre-trained emotion recognition model in sequence, determining the emotion vectors corresponding to the target objects in each target video frame, so that the emotion vectors corresponding to the multiple target video frames constitute an emotion vector sequence, wherein the multiple elements in the emotion vector respectively indicate probability values ​​corresponding to different types of emotions; sliding a sliding window in the emotion vector sequence, and constructing an emotion matrix corresponding to the target object using the emotion vectors contained in the sliding window; and determining whether the target object is abnormal according to the emotion matrix.

[0108] Optionally, the operation of extracting target video frames containing the target object from the first image sequence and sorting the target video frames according to time sequence to form a second image sequence containing the target object includes: extracting target video frames containing the target object from multiple first image sequences corresponding to multiple cameras respectively; and sorting the target video frames according to time sequence to form a second image sequence containing the target object.

[0109] Optionally, the operation of determining whether the target object is abnormal based on the emotion matrix includes: determining a covariance matrix reflecting the correlation between different emotions of the target object based on the emotion matrix and the transposed matrix of the emotion matrix; performing feature extraction based on the covariance matrix to determine an emotion feature vector reflecting the emotional characteristics of the target object; and determining whether the target object is abnormal based on the emotion feature vector.

[0110] Optionally, the operation of performing feature extraction based on the covariance matrix to determine an emotion feature vector reflecting the emotion characteristics of the target object includes: using a pre-trained projection matrix to project the covariance matrix into a corresponding emotion feature vector.

[0111] Optionally, the operation of determining whether the target object is abnormal based on the emotion feature vector includes: inputting the emotion feature vector into a pre-trained binary classification model, and using the binary classification model to determine whether the target object is abnormal.

[0112] Optionally, the memory 1020 is further configured to provide the processor 1010 with instructions for processing the following processing steps: when it is determined that the target object is abnormal, sending the second image sequence to the cloud server.

[0113] Thus, according to this embodiment, the target video frames containing the target object are sequentially input into the emotion recognition model, so as to determine the emotion vectors corresponding to the target object in each target video frame, and then the emotion vectors corresponding to each target video frame constitute an emotion vector sequence. The emotion vector sequence can be regarded as multi-channel time series information, and the time series information of each channel reflects the change of the corresponding emotion at different times. Then, a sliding window of a predetermined width is used to slide on the vector sequence, so that an emotion matrix of a corresponding size can be defined by the sliding window. Then, it is determined whether the target object is abnormal according to the emotion matrix. Since the emotion matrix contains the change information of different emotions of the target object over time, even if the numerical values ​​(such as probability values) corresponding to each emotion in the emotion vector are not much different and cannot be classified, the abnormal situation of the target object can be predicted using the information of the emotion matrix. Thus, the technical problem that the target object cannot be determined whether it is abnormal in the prior art when the emotion of the target object cannot be determined by the classification-based emotion recognition model is solved.

[0114] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0115] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0117] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0118] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0119] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0120] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multimedia monitoring method based on visual detection, characterized in that: include: Acquire, from a camera, a first image sequence captured by the camera; Extracting target video frames containing the target object from the first image sequence, and sorting the target video frames according to a time sequence to form a second image sequence containing the target object; The target video frames in the second image sequence are sequentially input into a pre-trained emotion recognition model to determine an emotion vector corresponding to a target object in each of the target video frames, so that the emotion vectors corresponding to the multiple target video frames constitute an emotion vector sequence, wherein the multiple elements in the emotion vector respectively indicate probability values ​​corresponding to different types of emotions; Using a sliding window to slide on the emotion vector sequence, and using the emotion vectors contained in the sliding window to construct an emotion matrix corresponding to the target object; as well as Determine whether the target object is abnormal according to the emotion matrix, and wherein The operation of determining whether the target object is abnormal according to the emotion matrix includes: Determining a covariance matrix reflecting correlations between different emotions of the target object based on the emotion matrix and a transposed matrix of the emotion matrix; Projecting the covariance matrix using a pre-trained projection matrix to determine an emotional feature vector reflecting the emotional features of the target object; and The emotion feature vector is input into a pre-trained binary classification model, and the binary classification model is used to determine whether the target object is abnormal.

2. The method according to claim 1, characterized in that The operation of extracting target video frames containing the target object from the first image sequence and sorting the target video frames according to a time sequence to form a second image sequence containing the target object includes: Extracting a target video frame containing the target object from a plurality of the first image sequences respectively corresponding to the plurality of cameras; and The target video frames are sorted according to a time sequence to form a second image sequence containing the target object.

3. The method according to claim 1, characterized in that The operation of extracting features based on the covariance matrix and determining an emotion feature vector reflecting the emotion features of the target object includes: using a pre-trained projection matrix to project the covariance matrix into a corresponding emotion feature vector.

4. The method according to claim 3, characterized in that: Also includes: Providing a sample set for training a projection matrix, wherein the sample set includes a plurality of first emotion matrix samples corresponding to abnormal target objects and a plurality of second emotion matrix samples corresponding to normal target objects; determining a first covariance matrix mean of the covariance matrix of the first emotion matrix samples, and determining a second covariance matrix mean of the covariance matrix of the second emotion matrix samples; as well as The projection matrix is ​​determined according to the first covariance matrix mean and the second covariance matrix mean.

5. The method according to claim 1, characterized in that Also includes: When it is determined that the target object is abnormal, the second image sequence is sent to a cloud server.

6. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the processor executes the method according to any one of claims 1 to 5.

7. A multimedia monitoring device based on visual detection, characterized in that: include: A first image sequence acquisition module, used to acquire a first image sequence captured by the camera from the camera; a second image sequence construction module, configured to extract target video frames containing a target object from the first image sequence, and sort the target video frames according to a time sequence to form a second image sequence containing the target object; an emotion vector determination module, configured to sequentially input the target video frames in the second image sequence into a pre-trained emotion recognition model, and determine the emotion vectors corresponding to the target objects in the respective target video frames, so that the emotion vectors corresponding to the plurality of target video frames constitute an emotion vector sequence, wherein the plurality of elements in the emotion vectors respectively indicate probability values ​​corresponding to different types of emotions; An emotion matrix determination module, used to slide the emotion vector sequence using a sliding window, and construct an emotion matrix corresponding to the target object using the emotion vectors contained in the sliding window; as well as an abnormality determination module, used to determine whether the target object is abnormal according to the emotion matrix, and wherein The abnormality determination module includes: A covariance matrix determination submodule, used to determine a covariance matrix reflecting the correlation between different emotions of the target object based on the emotion matrix and the transposed matrix of the emotion matrix; a projection submodule, configured to project the covariance matrix using a pre-trained projection matrix to determine an emotion feature vector reflecting the emotion feature of the target object; and The classification submodule is used to input the emotion feature vector into a pre-trained binary classification model, and use the binary classification model to determine whether the target object is abnormal.

8. A multimedia monitoring device based on visual detection, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Acquire, from a camera, a first image sequence captured by the camera; Extracting target video frames containing the target object from the first image sequence, and sorting the target video frames according to a time sequence to form a second image sequence containing the target object; The target video frames in the second image sequence are sequentially input into a pre-trained emotion recognition model to determine an emotion vector corresponding to a target object in each of the target video frames, so that the emotion vectors corresponding to the multiple target video frames constitute an emotion vector sequence, wherein the multiple elements in the emotion vector respectively indicate probability values ​​corresponding to different types of emotions; Using a sliding window to slide on the emotion vector sequence, and using the emotion vectors contained in the sliding window to construct an emotion matrix corresponding to the target object; as well as Determine whether the target object is abnormal according to the emotion matrix, and wherein The operation of determining whether the target object is abnormal according to the emotion matrix includes: Determining a covariance matrix reflecting correlations between different emotions of the target object based on the emotion matrix and a transposed matrix of the emotion matrix; Projecting the covariance matrix using a pre-trained projection matrix to determine an emotional feature vector reflecting the emotional features of the target object; and The emotion feature vector is input into a pre-trained binary classification model, and the binary classification model is used to determine whether the target object is abnormal.

Images