Video behavior recognition method

Abstract

The invention provides a video behavior recognition method, which comprises the following steps: obtaining a video behavior cue word feature vector sequence according to a to-be-recognized target behavior; the video behavior cue word feature vector sequence is a first number of cue word features obtained by splitting the target behavior; calculating the similarity between the video behavior cue word feature vector sequence and the picture frame vector sequence by using the similarity matrix so as to detect a target behavior in the video; the picture frame vector sequence is a second number of picture frame features obtained by utilizing video frame extraction; the elements of the similarity matrix represent the similarity between each cue word feature and each picture frame feature. The invention further provides a video behavior recognition device and equipment and a medium. The target behavior description accuracy is improved and the detection accuracy is improved by describing a plurality of related prompt words and sentences of the target behavior; and through the similarity matrix, calculating the similarity between the video behavior cue word feature vector sequence and the picture frame vector sequence, and accurately detecting the target behavior.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and more specifically to a video behavior recognition method. Background Art

[0002] With the continuous development of intelligent monitoring, security systems and automated video analysis technology, video behavior recognition has become increasingly important in many fields. Traditional video analysis methods mainly rely on image processing technology combined with machine learning models for behavior recognition. However, relying solely on image information for behavior recognition faces a series of challenges, such as complex video content, frequent background changes, and fast and changeable character movements, which leads to certain limitations in video analysis systems.

[0003] In order to improve the accuracy and efficiency of video behavior recognition, deep learning technology that combines multimodal information (such as image information and text information) has gradually become a research hotspot in recent years. How to effectively match text information with image information and then perform video behavior recognition has become an urgent problem to be solved. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a video behavior recognition method that effectively combines text information and image information.

[0005] The present disclosure provides a video behavior recognition method, including: obtaining a video behavior prompt word feature vector sequence according to a target behavior to be recognized; the video behavior prompt word feature vector sequence is a first number of prompt word features obtained by splitting the target behavior; using a similarity matrix, calculating the similarity between the video behavior prompt word feature vector sequence and a picture frame vector sequence to determine whether the target behavior exists; the picture frame vector sequence is a second number of picture frame features obtained by extracting frames from a video stream; the elements of the similarity matrix represent the similarity between each prompt word feature and each picture frame feature.

[0006] According to an embodiment of the present disclosure, a video behavior prompt word feature vector sequence is obtained according to the target behavior to be identified, including using the target behavior to be identified to obtain behavior prompt sentences from a behavior prompt word description library to obtain a prompt word sentence list; the behavior prompt word description library is used to store behavior prompt sentences of the behavior; the behavior prompt sentence is used to describe part of the characteristics of the behavior; and using a text feature extractor to obtain a video behavior prompt word feature vector sequence according to the prompt word sentence list.

[0007] According to an embodiment of the present disclosure, using the target behavior to be identified, behavior prompt sentences are obtained from a behavior prompt word description library to obtain a prompt word sentence list, including: in response to the target behavior being crossing the road, the obtained behavior prompt sentence includes at least one of walking to the side of the road, looking left and right, being on the sidewalk, being in the middle of the road, and being across the road; in response to the target behavior being stealing, the obtained behavior prompt sentence includes at least one of looking left and right, approaching an item, the item disappearing, leaving the shelf, and leaving the scene.

[0008] According to an embodiment of the present disclosure, a similarity matrix is ​​used to calculate the similarity between a video behavior prompt word feature vector sequence and a picture frame vector sequence for detecting a target behavior in a video, including: for each video behavior prompt word feature vector, calculating the similarity value between the video behavior prompt word feature vector and the picture frame vector sequence; filling the similarity value into a matrix having a first number of rows and a second number of columns to obtain a similarity matrix; and using the similarity matrix to detect the target behavior in the video.

[0009] According to an embodiment of the present disclosure, a similarity matrix is ​​used to detect target behaviors in a video, including: obtaining the maximum similarity value of each column in the similarity matrix; calculating the sum of the maximum similarity values ​​of each column to obtain the similarity between a feature vector sequence of video behavior prompt words and a vector sequence of picture frames; and in response to the similarity exceeding a preset threshold, determining that the target behavior exists in the video.

[0010] According to an embodiment of the present disclosure, the method also includes using a similarity matrix to detect target behaviors in a video stream in real time, including: in response to receiving a real-time video stream, intercepting a current video segment of a preset time window length to obtain a real-time picture frame vector sequence; the picture frame vector sequence is a real-time picture frame feature obtained by extracting frames from the current video segment; using a similarity matrix, calculating the similarity between a video behavior prompt word feature vector sequence and a real-time picture frame vector sequence to determine whether the current video segment has a target behavior; the elements of the similarity matrix represent the similarity between each prompt word feature and each real-time picture frame feature; in response to the presence of unprocessed video segments in the real-time video stream, repeatedly executing the step of intercepting the current video segment of a preset time window length to obtain a real-time picture frame vector sequence.

[0011] According to an embodiment of the present disclosure, the similarity between a feature vector sequence of a video behavior prompt word and a picture frame vector sequence is calculated for detecting a target behavior in a video, and the method also includes: extracting a video segment of a preset time window length in the video, extracting frames to obtain a picture frame list; and using an image feature extractor to obtain a picture frame vector sequence according to the picture frame list.

[0012] The second aspect of the present disclosure provides a video behavior recognition device, which can be used to implement the above-mentioned video behavior recognition method, including: a prompt word module, used to obtain a video behavior prompt word feature vector sequence according to the target behavior to be recognized; the video behavior prompt word feature vector sequence is a first number of prompt word features obtained by splitting the target behavior; a similarity module is used to calculate the similarity between the video behavior prompt word feature vector sequence and the picture frame vector sequence using a similarity matrix for detecting the target behavior in the video; the picture frame vector sequence is a second number of picture frame features obtained by video frame extraction; the elements of the similarity matrix represent the similarity between each prompt word feature and each picture frame feature.

[0013] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned video behavior recognition method.

[0014] The fourth aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned video behavior recognition method.

[0015] According to the video behavior recognition method provided by the present disclosure, a sequence of prompt word vectors describing the same target behavior from multiple angles is obtained by splitting the target behavior. Since the similarity between the target behavior and the image information is fully considered, a similarity matrix is ​​obtained, which helps to express the semantic features of the target behavior in a more accurate natural language, making the model's understanding of the behavior more detailed. Therefore, it at least partially solves the technical problem of ignoring the semantic hierarchical description of the behavior, resulting in reduced recognition accuracy when facing complex scenes or rapidly changing dynamic environments, and achieves the technical effect of improving the accuracy of behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a video behavior recognition method according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 2 The structure block diagram of the video behavior recognition device according to the embodiment of the present disclosure is schematically shown;

[0018] Figure 3 A block diagram of an electronic device suitable for implementing a video behavior recognition method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0019] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0020] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0021] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0022] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0023] At present, with the rapid growth of the number of domestic cameras, the huge amount of video data generated every day not only poses a huge burden on human and material resources, but also leads to expensive costs. In response to this problem, this paper focuses on solving specific problems in the field of video behavior analysis. Based on the visual language large model technology and combined with multi-prompt word feature fusion, a video behavior analysis method is proposed to enhance the accuracy and robustness of video analysis.

[0024] Figure 1 The flowchart of the video behavior recognition method according to the embodiment of the present disclosure is schematically shown. Figure 1As shown, an embodiment of the present disclosure provides a video behavior recognition method, including: obtaining a video behavior prompt word feature vector sequence according to a target behavior to be recognized; the video behavior prompt word feature vector sequence is a first number of prompt word features obtained by splitting the target behavior; using a similarity matrix, calculating the similarity between the video behavior prompt word feature vector sequence and the picture frame vector sequence to determine whether the target behavior exists; the picture frame vector sequence is a second number of picture frame features obtained by extracting frames from a video stream; the elements of the similarity matrix represent the similarity between each prompt word feature and each picture frame feature.

[0025] Through the embodiments of the present disclosure, the target behavior is combined with descriptive prompt words, which helps to express the semantic features of the target behavior with more accurate natural language, making the model's understanding of the behavior more accurate; by defining a sequence of prompt word vectors, it is possible to flexibly respond to different target behaviors, describe each behavior in detail, and comprehensively examine the similarity between the video and the behavior when calculating the similarity; by combining the visual language large model and video image analysis technology, it is possible to more comprehensively understand and analyze the video content, and provide intelligent analysis services for the video analysis platform, because it not only relies on visual information, but also includes the understanding of the video description, which can improve the accuracy of behavior recognition. Since the system can identify behaviors more accurately, it can reduce false positives and false negatives, thereby reducing the manual operation costs of the monitoring system.

[0026] On the basis of the above embodiment, a video behavior prompt word feature vector sequence is obtained according to the target behavior to be identified, including using the target behavior to be identified to obtain behavior prompt sentences from a behavior prompt word description library to obtain a prompt word sentence list; the behavior prompt word description library is used to store behavior prompt sentences of the behavior; the behavior prompt sentence is used to describe part of the characteristics of the behavior; and using a text feature extractor to obtain a video behavior prompt word feature vector sequence according to the prompt word sentence list.

[0027] In this embodiment, a video behavior analysis prompt word vector sequence is constructed. According to the behavior recognition task, a behavior prompt word description library is defined, which consists of multiple language description sentences. The sentence content is the sub-event description of the target video event behavior, forming a prompt word sentence list. Assume that there are M, represented by T1, T2, ..., T M Using the text feature extractor in the visual language model CLIP (Contrastive Language-Image Pretraining), the natural language in the prompt word list is feature extracted and vectorized to obtain Ft1, Ft2, ..., Ft M , forming a video behavior analysis prompt word vector sequence.

[0028] It should be noted that CLIP is a large visual-language model. CLIP simultaneously trains a visual model (image understanding) and a language model (text understanding) through contrastive learning methods, so that the model can understand and process the relationship between images and text. This model can achieve cross-modal understanding and reasoning on visual and language tasks, such as image-text matching, image generation, and text description generation. CLIP processes image and text data at the same time, which is different from traditional image classification models. Traditional image classification models usually only focus on image data, while CLIP enhances the ability of image understanding by fusing text information, allowing it to understand and describe the content of the image. For example, CLIP can retrieve related images based on text descriptions, or given an image, generate a text describing it.

[0029] Through the embodiments of the present disclosure, by using a behavior prompt word description library, more behavior patterns can be easily expanded by simply adding new prompt statements to the library without retraining the entire model. The behavior prompt word description library provides a specific description for each target behavior, and can quickly and accurately generate a prompt word list to describe the target behavior to be identified in a hierarchical manner.

[0030] On the basis of the above embodiment, using the target behavior to be identified, behavior prompt sentences are obtained from the behavior prompt word description library to obtain a prompt word sentence list, including: in response to the target behavior being crossing the road, the obtained behavior prompt sentence includes at least one of walking to the side of the road, looking left and right, being on the sidewalk, being in the middle of the road, and being across the road; in response to the target behavior being stealing, the obtained behavior prompt sentence includes at least one of looking left and right, approaching an item, the item disappearing, leaving the shelf, and leaving the scene.

[0031] It should be noted that the behavior prompt word description library can optionally include other types of behavior recognition, such as entry, abnormal wandering and fall monitoring, illegal intrusion, etc., and the behavior can be recognized by simply breaking down the sub-behavior and adding it to the behavior prompt word description library. Breaking down the sub-behavior can be specifically described from multiple angles (such as action, expression, position), describing the behavior in the order of human actions, etc.

[0032] Through the embodiments of the present disclosure, by defining a behavior prompt word description library, the system can convert natural language descriptions into vector sequences, which helps the machine understand the semantic information of the video content; by refining the prompt statements for different target behaviors, the behavior characteristics can be defined more accurately and the recognition accuracy can be improved; and appropriate prompt statements can be selected according to different target behavior types, so that the model can handle a variety of different behavior recognition tasks.

[0033] On the basis of the above embodiment, a similarity matrix is ​​used to calculate the similarity between a video behavior prompt word feature vector sequence and a picture frame vector sequence for detecting a target behavior in a video, including: for each video behavior prompt word feature vector, calculating the similarity value between the video behavior prompt word feature vector and the picture frame vector sequence; filling the similarity value into a matrix having a first number of rows and a second number of columns to obtain a similarity matrix; and using the similarity matrix to detect the target behavior in the video.

[0034] In this embodiment, the cosine similarity is used to calculate the feature similarity to calculate the similarity matrix between the picture frame vector sequence and the prompt word vector sequence:

[0035]

[0036] Among them, Fi i Represents the i-th element from the picture frame vector sequence; Ft j represents the jth element from the prompt word vector sequence; similarity(i, j) represents the similarity between the i-th element from the video picture group sequence vector sequence and the j-th element from the video behavior analysis prompt word vector sequence.

[0037] By calculating the similarity of each vector in the video behavior prompt word feature vector sequence and the picture frame vector sequence, we can obtain the similarity of each image feature vector (N) and each text feature vector (M) describing the video event. These similarity values ​​are filled into an M×N matrix, where the element in the i-th row and j-th column of the matrix represents Fi i and Ft. j The similarity between .

[0038] Through the embodiments of the present disclosure, the target behavior is detected by calculating the similarity between the feature vector sequence of the video behavior prompt word and the image frame vector sequence, and the matching degree between the video content and the predefined behavior can be quantified. This method can improve the accuracy of the analysis and help identify specific behaviors in the video. Storing similarity values ​​in matrix form can intuitively show the matching degree between each prompt word and the image frame, which is convenient for subsequent analysis and processing. For example, the column represents the matching degree between the image and each prompt word, and the row represents the matching degree between the prompt word and each image.

[0039] On the basis of the above embodiment, a similarity matrix is ​​used to detect the target behavior in the video, including: obtaining the maximum similarity value of each column in the similarity matrix; calculating the sum of the maximum similarity values ​​of each column to obtain the similarity between the video behavior prompt word feature vector sequence and the picture frame vector sequence; in response to the similarity exceeding a preset threshold, determining that the target behavior exists in the video.

[0040] In this embodiment, the similarity matrix score is calculated. Each column in the matrix has M similarity values. By finding the maximum value among the M similarity values, the maximum similarity value between the current image feature vector and the M text feature vectors can be obtained, and the text description that best matches the current image can be obtained. The maximum similarity value of each column is obtained in turn, and the final similarity matrix score is obtained by summing up, and the similarity between the current video clip and the target video event is obtained.

[0041] Through the embodiments of the present disclosure, the elements of the similarity matrix represent the similarity between each prompt word feature and each picture frame feature, wherein each column represents the similarity between the frame and all text feature vectors respectively. Since each text feature vector describes the target behavior from a certain angle or a certain stage, the sum of the columns represents the matching degree between the frame and the target behavior. By summing up the maximum similarity value, the target behavior detection is quantified, and the target behavior can be detected efficiently and responded to in a timely manner; by summing up the maximum value to integrate multiple similarities, the noise or error caused by single angle judgment can be solved, and the robustness of recognition can be improved.

[0042] On the basis of the above embodiments, the method also includes using a similarity matrix to detect target behaviors in a video stream in real time, including: in response to receiving a real-time video stream, intercepting a current video segment of a preset time window length to obtain a real-time picture frame vector sequence; the picture frame vector sequence is a real-time picture frame feature obtained by extracting frames from the current video segment; using a similarity matrix, calculating the similarity between a video behavior prompt word feature vector sequence and a real-time picture frame vector sequence to determine whether the current video segment has a target behavior; the elements of the similarity matrix represent the similarity between each prompt word feature and each real-time picture frame feature; in response to the presence of unprocessed video segments in the real-time video stream, repeatedly executing the step of intercepting the current video segment of a preset time window length to obtain a real-time picture frame vector sequence.

[0043] In this embodiment, the video stream data sliding window analysis. The image sequence in the video stream is vectorized, and the picture frame vector sequence of the fixed time window length L is selected in sequence according to the time interval step t, and the similarity calculation is repeated to obtain the current video sliding window segment behavior similarity matrix score. When the similarity matrix score exceeds the set threshold, the video segment is determined to be the defined target video behavior, and the video behavior analysis effect is achieved.

[0044] Through the embodiments of the present disclosure, the performance of video behavior analysis is improved through improved algorithms and model structures. For example, by using a variable sliding window mechanism, the system can adaptively adjust the window size according to the characteristics of the data stream, thereby improving efficiency and accuracy. By applying a sliding window on the video stream, it is possible to process real-time video streams, ensuring that the system can respond immediately to the occurrence of target behaviors during video playback, which is suitable for monitoring and security applications in dynamic environments; only video clips within a preset time window are processed, saving computing resources and improving the response speed of detection.

[0045] Based on the above embodiment, the similarity between the video behavior prompt word feature vector sequence and the picture frame vector sequence is calculated for detecting the target behavior in the video, which also includes: extracting video segments of a preset time window length in the video, extracting frames to obtain a picture frame list; using an image feature extractor to obtain a picture frame vector sequence according to the picture frame list.

[0046] In this embodiment, extracting a video clip fixed time window length L picture frame vector sequence includes: extracting a frame picture sequence from the video stream at a fixed time interval T, assuming there are N frames, represented by I1, I2, ..., I N Using the image feature extractor in the visual language model CLIP, the images in the image sequence list are vectorized to extract features and obtain Fi1, Fi2, ..., Fi N , forming a sequence of picture frame vectors.

[0047] Through the embodiments of the present disclosure, the consistency and repeatability of video analysis can be ensured by extracting a sequence of picture frame vectors with a fixed time window length, and the errors caused by inconsistent frame selection can be reduced by extracting frames at fixed time intervals. Through the frame extraction operation, the features of different time segments can be extracted, and the video content can be analyzed more carefully, while avoiding the processing of redundant information and improving the recognition accuracy.

[0048] Based on the above video behavior recognition method, the present disclosure also provides a video behavior recognition device. Figure 2 The device is described in detail.

[0049] Figure 2 The structure block diagram of the video behavior recognition device according to an embodiment of the present disclosure is schematically shown.

[0050] like Figure 2 As shown, the video behavior recognition device of this embodiment includes a prompt word module and a similarity module.

[0051] The prompt word module is used to obtain a video behavior prompt word feature vector sequence according to the target behavior to be identified; the video behavior prompt word feature vector sequence is a first number of prompt word features obtained by splitting the target behavior. In one embodiment, the prompt word module can be used to perform the operation S1 described above, which will not be repeated here.

[0052] The similarity module is used to calculate the similarity between the video behavior prompt word feature vector sequence and the picture frame vector sequence using a similarity matrix to detect the target behavior in the video; the picture frame vector sequence is a second number of picture frame features obtained by video frame extraction; the elements of the similarity matrix represent the similarity between each prompt word feature and each picture frame feature. In one embodiment, the similarity module can be used to perform the operation S2 described above, which will not be repeated here.

[0053] Figure 3 A block diagram of an electronic device suitable for implementing a video behavior recognition method according to an embodiment of the present disclosure is schematically shown.

[0054] like Figure 3 As shown, the electronic device 300 according to an embodiment of the present disclosure includes a processor 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage part 308 into a random access memory (RAM) 303. The processor 301 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 301 may also include an onboard memory for caching purposes. The processor 301 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0055] In RAM 303, various programs and data required for the operation of electronic device 300 are stored. Processor 301, ROM 302 and RAM 303 are connected to each other through bus 304. Processor 301 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 302 and / or RAM 303. It should be noted that the program can also be stored in one or more memories other than ROM 302 and RAM 303. Processor 301 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0056] According to an embodiment of the present disclosure, the electronic device 300 may further include an input / output (I / O) interface 305, which is also connected to the bus 304. The electronic device 300 may further include one or more of the following components connected to the I / O interface 305: an input portion 306 including a keyboard, a mouse, etc.; an output portion 307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 308 including a hard disk, etc.; and a communication portion 309 including a network interface card such as a LAN card, a modem, etc. The communication portion 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage portion 308 as needed.

[0057] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0058] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

[0059] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0060] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0061] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0062] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A video behavior recognition method, characterized in that: include: According to the target behavior to be identified, a feature vector sequence of video behavior prompt words is obtained; The video behavior prompt word feature vector sequence is a first number of prompt word features obtained by splitting the target behavior; Using a similarity matrix, calculating the similarity between the feature vector sequence of the video behavior prompt word and the picture frame vector sequence for detecting the target behavior in the video; The picture frame vector sequence is a second number of picture frame features obtained by using the video frame extraction; The elements of the similarity matrix represent the similarity between each of the prompt word features and each of the picture frame features.

2. The method according to claim 1, wherein: According to the target behavior to be identified, the feature vector sequence of the video behavior prompt word is obtained, including Using the target behavior to be identified, obtaining behavior prompt sentences from a behavior prompt word description library to obtain a prompt word sentence list; the behavior prompt word description library is used to store behavior prompt sentences of behaviors; the behavior prompt sentences are used to describe some characteristics of the behaviors; A text feature extractor is used to obtain a video behavior prompt word feature vector sequence according to the prompt word sentence list.

3. The method according to claim 2, wherein: Using the target behavior to be identified, obtaining behavior prompt sentences from a behavior prompt word description library to obtain a prompt word sentence list includes: In response to the target behavior being crossing the road, the obtained behavior prompt sentence includes at least one of walking to the side of the road, looking left and right, being on the sidewalk, being in the middle of the road, and being across the road; In response to the target behavior being stealing, the obtained behavior prompt sentence includes at least one of looking around, approaching the object, the object disappearing, leaving the shelf, and leaving the scene.

4. The method according to claim 1, wherein: Using the similarity matrix, calculating the similarity between the video behavior prompt word feature vector sequence and the picture frame vector sequence for detecting the target behavior in the video, including: For each video behavior prompt word feature vector, calculate the similarity value between the video behavior prompt word feature vector and the picture frame vector sequence; Filling the similarity values ​​into a matrix having a first number of rows and a second number of columns to obtain the similarity matrix; The similarity matrix is ​​used to detect target behaviors in the video.

5. The method according to claim 3, wherein: Utilizing the similarity matrix, detecting the target behavior in the video includes: Obtaining the maximum similarity value of each column in the similarity matrix; The sum of the maximum similarity values ​​of each column is obtained to obtain the similarity between the feature vector sequence of the video behavior prompt word and the picture frame vector sequence; In response to the similarity exceeding a preset threshold, it is determined that the target behavior exists in the video.

6. The method according to claim 1, wherein: The method further includes, using a similarity matrix, detecting target behavior in a video stream in real time, including: In response to receiving a real-time video stream, a current video segment of a preset time window length is intercepted to obtain a real-time picture frame vector sequence; the picture frame vector sequence is a real-time picture frame feature obtained by extracting frames from the current video segment; Using a similarity matrix, the similarity between the video behavior prompt word feature vector sequence and the real-time picture frame vector sequence is calculated to determine whether the current video segment has the target behavior; the elements of the similarity matrix represent the similarity between each of the prompt word features and each of the real-time picture frame features; In response to the presence of unprocessed video segments in the real-time video stream, the step of cutting out the current video segment of a preset time window length to obtain a real-time picture frame vector sequence is repeatedly performed.

7. The method according to claim 1, wherein: Calculating the similarity between the feature vector sequence of the video behavior prompt word and the picture frame vector sequence for detecting the target behavior in the video, which also includes: Extract video clips of preset time window length from the video, extract frames to obtain a list of picture frames; An image feature extractor is used to obtain a picture frame vector sequence according to the picture frame list.

8. A video behavior recognition device, characterized in that: The device can be used to implement the method as claimed in any one of claims 1 to 7, including: A prompt word module, used to obtain a video behavior prompt word feature vector sequence according to the target behavior to be identified; the video behavior prompt word feature vector sequence is a first number of prompt word features obtained by splitting the target behavior; A similarity module is used to calculate the similarity between the video behavior prompt word feature vector sequence and the picture frame vector sequence using a similarity matrix to detect the target behavior in the video; the picture frame vector sequence is a second number of picture frame features obtained by extracting frames from the video; the elements of the similarity matrix represent the similarity between each of the prompt word features and each of the picture frame features.

9. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 7.

Images