Action recognition system

Abstract

[Problem] To provide an action recognition system capable of performing action recognition by using information on a natural language and the like without performing prior learning. [Solution] An action recognition system 100 according to the present invention comprises: an image acquisition unit 110 that acquires an image such as a moving image; a recognition target image detection and tracking unit 120 that detects a recognition target image of a person, an object, and the like from each frame of the acquired moving image, and tracks the same; a detection expression input unit 130 that inputs a detection expression that specifies an action to be detected; a language-image matching unit 150 that calculates the matching degree between the detected and tracked recognition target image and the language of the detection expression; and a determination unit 170 that determines the action in the recognition target image on the basis of the matching degree.

Description

Behavior Recognition System

[0001] The present invention relates to an activity recognition system that recognizes activities, and more particularly to activity recognition that uses natural language and does not require prior learning.

[0002] Action recognition is a technology that automatically identifies the movements and behaviors of humans and objects from video or sequential image data. It detects and classifies actions and movement patterns by analyzing continuous image data collected from cameras and sensors. Action recognition technology can be used, for example, to detect suspicious behavior in security monitoring systems and monitor people's movements in autonomous driving systems. Action recognition methods use machine learning algorithms, such as deep learning models and recurrent neural networks (RNNs), to classify actions using features extracted from image data.

[0003] For example, Patent Literature 1 discloses an action recognition device that extracts skeleton data from a video and recognizes actions in the video using the extracted skeleton data and a pre-constructed trained model. Patent Literature 2 also discloses an action recognition device that acquires skeletal information including the positions of multiple joints, changes weights assigned to each of the multiple joints depending on the type of partial action to be recognized, and recognizes partial actions based on feature amounts obtained from the positions of the multiple joints.

[0004] JP 2019-191981 A JP 2021-135898 A

[0005] The conventional behavior recognition devices disclosed in Patent Documents 1 and 2 require a learning model to be constructed in advance in order to recognize behaviors. As a result, for example, when a new behavior needs to be recognized or when a behavior to be recognized needs to be changed, the devices are unable to immediately recognize the behavior, and are therefore lacking in flexibility and convenience.

[0006] The present invention is intended to solve such conventional problems, and aims to provide an action recognition system that is capable of performing action recognition using information such as natural language without prior learning.

[0007] The behavior recognition system according to the present invention comprises an acquisition means for acquiring an image, a detection / tracking means for detecting a recognition target image from the acquired image and tracking the recognition target image, and a calculation means for calculating the degree of matching between specific information for identifying a search target and the recognition target image detected / tracked by the detection / tracking means.

[0008] In one aspect, the behavior recognition system further includes a discrimination means for discriminating the search target from the recognition target image based on the degree of matching. In one aspect, the identification information includes linguistic information of a sentence, character string, or keyword expressed in a natural language for identifying the search target, and the calculation means calculates the degree of matching between the linguistic information and the recognition target image. In one aspect, the linguistic information includes vector data in a natural language, and the calculation means calculates the degree of matching between vector data of the recognition target image detected / tracked from a video image and the vector data in the natural language. In one aspect, the identification information includes image information for identifying the search target, and the calculation means calculates the degree of matching between the image information and the recognition target image. In one aspect, the identification information includes audio information for identifying the search target, and the calculation means calculates the degree of matching between the audio information and the recognition target image. In one aspect, the behavior recognition system further includes a storage means for storing the recognition target image from the detection / tracking means, and the calculation means calculates the degree of matching between the recognition target image stored in the storage means. In one aspect, the behavior recognition system further includes a vector data storage means for storing vector data of a recognition target image detected / tracked from a video image, and the calculation means calculates a degree of match between vector data of a natural language for identifying a search target and the vector data of the recognition target image. In one aspect, the behavior recognition system further includes a calculation result storage means for storing a calculation result of the calculation means, and the discrimination means discriminates the search target based on the calculation result stored in the calculation result storage means. In one aspect, the behavior recognition system further includes an input means for inputting the specific information. In one aspect, the recognition target image includes a person, and the discrimination means discriminates the person's behavior and / or behavior attributes. In one aspect, the recognition target image includes a person and an object, and the discrimination means discriminates the person's behavior and / or behavior attributes including the object.

[0009] According to the present invention, the degree of matching between specific information for identifying the behavior to be searched for and the detected / tracked recognition target image is calculated, so that a user can search for behaviors matching the specific information from the recognition target image, for example, by simply inputting the specific information, without having to perform pre-learning for each behavior recognition. This makes it possible to improve the versatility, convenience, and cost-effectiveness of the behavior recognition system.

[0010] 1 is a block diagram showing the overall configuration of a behavior recognition system according to a first embodiment of the present invention. FIG. 1 is a diagram showing an example of an image of a person / object detected / tracked by the recognition target detection / tracking unit shown in FIG. 1. FIG. 2 is a diagram showing an example of an image of a person / object detected / tracked by the recognition target detection / tracking unit shown in FIG. 1. FIG. 3 is a diagram showing an example of an image of a person / object detected / tracked by the recognition target detection / tracking unit shown in FIG. 1. FIG. 4 is a diagram showing an example of a numerical value showing the degree of matching (correlation) between each image shown in FIG. 2 and FIG. 3 and text (language). FIG. 5 is a diagram showing an example of a numerical value showing the degree of matching (correlation) between each image shown in FIG. 4 and text (language). FIG. 6 is a data example showing the degree of matching (correlation) between an image detected / tracked from consecutive frames and a detected expression. FIG. 7 is a diagram showing an example of overlapping images detected / tracked from frames. FIG. 8 is a diagram showing an example of visualized display of behavior-recognized images and detected expressions. FIG. 9 is a diagram showing an example of the configuration of a behavior recognition system according to a second embodiment of the present invention. FIG. 10 is a diagram showing an example of the configuration of a behavior recognition system according to a third embodiment of the present invention. FIG. 11 is a diagram showing an example of vector data stored in a person / object vector storage unit. FIG. 12 is a diagram showing an example of the configuration of a behavior recognition system according to a fourth embodiment of the present invention. FIG. 13 is a diagram showing an example of the operation of an image acquisition unit according to a fifth embodiment of the present invention. Fig. 10 is a diagram showing a configuration example of a behavior recognition system according to a sixth embodiment of the present invention. Fig. 11 is a diagram showing an example of specifying a specific person according to a modified example of the sixth embodiment. Fig. 12 is a diagram showing a configuration example of a behavior recognition system according to a seventh embodiment of the present invention. Fig. 13 is a diagram showing a configuration example of a behavior recognition system according to an eighth embodiment of the present invention.

[0011] The activity recognition system of the present invention tracks activities captured in image data or video and enables recognition of activities defined by natural language or written expressions. The activity recognition system of the present invention can be applied to a variety of applications, such as surveillance systems, security systems, and autonomous driving systems.

[0012] The activity recognition system of the present invention is realized using hardware resources and software resources installed thereon, and is configured using a computer, computer system, electronic device, or electronic system including, for example, a central processing unit including a CPU, ALU, etc., an arithmetic processing unit, a memory unit, an input / output unit, an imaging unit, a display unit, a communication unit, a user interface unit, etc. The activity recognition system may also be a system in which one or more computer devices or electronic devices are connected via a network.

[0013] The memory unit includes, for example, a semiconductor storage device, a hard disk storage device, etc., and stores various data, programs, and software applications used for activity recognition. The input / output unit includes a user input device (keyboard, voice input, touch panel, etc.), a display, an audio output device, etc. The communication unit has a function for communicating with a network or external devices via wired or wireless communication means. The communication unit enables, for example, downloading data from a server connected to the network, uploading data to a server, or acquiring data from an imaging camera, sensor, etc. connected to the network. The arithmetic processing unit includes, for example, an image processing processor or a DSP for processing image data. The central processing unit executes the programs or software applications stored in the memory unit and controls activity recognition.

[0014] 1 is a block diagram showing the configuration of a behavior recognition system according to a first embodiment of the present invention. The behavior recognition system 100 of this embodiment includes an image acquisition unit 110 that acquires image data (including video and still images) of a recognition target, a recognition target image detection / tracking unit 120 that detects and tracks recognition target images (e.g., people or objects) captured in the images acquired by the image acquisition unit 110, a detection expression input unit 130 that inputs detection expressions for detecting behavior using natural language from the detected / tracked recognition target images, a detection expression list storage unit 140 that stores a list of detection expressions, a language-image matching unit 150 that calculates the degree of matching (correlation) between the language used as the detection expression input from the detection expression input unit 130 and the recognition target images detected / tracked by the recognition target image detection / tracking unit 120, a data storage unit 160 that stores the matching results of the language-image matching unit 150, a determination unit 170 that determines or distinguishes the behavior of people / objects captured in the images based on the data stored in the data storage unit 160, and an application unit 180 that uses the determination results of the determination unit 170.

[0015] The image acquisition unit 110 acquires, for example, image data captured by one or more imaging cameras (e.g., high-resolution cameras, RGB cameras, infrared cameras, stereo cameras), or image data provided from other electronic devices, servers, etc., without being particularly limited thereto. The image data is a moving image composed of consecutive frames (e.g., 30 or 60 frames per second), or a series of multiple still images. The size, resolution, type, etc., of the image data are, without being particularly limited thereto, for example, color images, infrared images, grayscale images, black and white images, etc.

[0016] The image data acquired by the image acquisition unit 110 is an image of a person or object to be recognized, and may be, for example, a video image captured by one or more fixed cameras capturing a certain space where a person or object may exist (such as a road, a building, a shopping mall, a restaurant, or the premises of a station or airport).

[0017] The recognition target image detection / tracking unit 120 receives image data (e.g., moving images) from the image acquisition unit 110, detects images of person A, person B, ..., person N, or object A, object B, ..., object N, which are recognition targets, from the image data, and tracks the images of these detected people and objects in consecutive frames on the time axis. The detection of people and objects as recognition target images is performed, for example, by image analysis of the features of the people and objects.

[0018] There are no particular limitations on the type, size, etc. of people or objects to be detected / tracked. For example, people include men, women, children, adults, elderly people, tall people, short people, hunched people, walking people, running people, stationary people, fallen people, people lying on their backs, people lying face down, people moving their hands or fingers, people holding something in their hands, people moving their feet, people eating, people drinking, people smoking, people wearing hats, people carrying luggage, people carrying a bag, people carrying shopping bags, etc.

[0019] In addition, objects include animals such as birds, dogs, and cats, cars, motorbikes, bicycles, roads, crosswalks, buildings, mountains, the sea, plates, cigarettes, bags, smartphones, hats, helmets, masks, specific stationary positions, and specific locations (for example, specific parts of structures that may deteriorate, such as apartment buildings, locations where cleaning is carried out, specific locations at intersections, specific locations on roads, specific locations in urban areas, and specific locations on cliffs and rivers).

[0020] 2 to 4 are examples of images of detected people and objects. In Fig. 2, [bird], [car], and [dog] are examples of objects. [h1] and [h2] are examples of people. [s1], [s2], [s3], and [s4] are examples of people, but these images also include cigarettes, masks, bags, etc.

[0021] The images [su1], [su2], and [su3] in Figure 3 are examples of people, and include cigarettes, fingers, smoke, etc. The images [w1], [w2], [w3], [w4], [w5], and [w6] are examples of people walking, and include cell phones, bags, hoodies, glasses, etc. The images [b1], [b2], [b3], [b4], [b5], [b6], [h1], [h2], [h3], [h4], [h5], [h6], [hb1], and [hb2] in Figure 4 are examples of people, and include day bags, backpacks, satchels, handbags, canes, etc.

[0022] Images of people or objects detected / tracked by the recognition target image detection / tracking unit 120 are sequentially provided to the language-image matching unit 150. The language-image matching unit 150 calculates the degree of matching between the image of the detected / tracked person or object and the detection expression (natural language) input from the detection expression input unit 130.

[0023] The detection expression input unit 130 receives a detection expression for specifying the behavior to be detected from a user or another electronic device, and provides the received detection expression to the language-image matching unit 150. The behavior to be detected should be interpreted broadly and includes not only the behavior of a person but also the behavior of an object, and the behavior of an object may include changes in the state or event of the object. The detection expression input unit 130 may be one that is directly input by the user via a user interface (e.g., a keyboard, voice recognition, a mobile terminal, or other device), or may be one that inputs a detection expression selected from a list of detection expressions stored in the list storage unit 140.

[0024] The detection expression input from the detection expression input unit 130 is a natural language for specifying the behavior to be detected or searched for, and this natural language can be arbitrarily defined by a user or the like. The natural language may be a sentence (text), or a single keyword or a combination of multiple keywords. Furthermore, when a sentence is input, the detection expression input unit 130 may perform a syntactic or semantic analysis of the sentence and extract a detection expression from the analysis results.

[0025] Specifically, the detection expression is determined depending on what kind of behavior or object is desired to be detected. In other words, the detection expression differs depending on what application the behavior recognition system 100 is used for. Furthermore, multiple detection expressions may be used to detect one behavior. In other words, using multiple detection expressions makes it possible to identify behavior from multiple perspectives.

[0026] For example, when using the behavior recognition system 100 for marketing, if it is desired to detect customer behaviors such as "a person looking at shelves" or "a person picking up a product," the detection expressions can be "a person looking at shelves" or "a person picking up a product." Of course, it is also possible to identify one behavior using multiple detection expressions. For example, "a person looking at shelves" can be identified using detection expressions such as "shelf," "woman," or "middle-aged," and "a person picking up a product" can be identified using detection expressions such as "product," "woman," or "reaching out." For example, when it is desired to detect a person's clothing or belongings as customer attributes, it is possible to use detection expressions such as "a man wearing short sleeves," "a child wearing long sleeves," or "a woman carrying a handbag." Furthermore, when it is desired to detect a person taking a photo or eating or drinking as a nuisance, it is possible to use detection expressions such as "a person taking a photo," "taking a photo," "camera," "a person eating or drinking," "eating and drinking," "fork," "spoon," or "cup."

[0027] Furthermore, when using the behavior recognition system 100 for security purposes, if one wants to detect "a person looking around" as a suspicious behavior, one can use detection expressions such as "a person looking around," "a person moving their head," "a person with an unsteady gaze direction," or "a person with unusual movements." If one wants to detect "backpack," "mask," "hat," "sneakers," or the like as suspicious attributes, one can use detection expressions such as "backpack," "person carrying a backpack," "mask," "person wearing a mask," "person wearing a hat," "sneakers," or "person wearing sneakers." If one wants to detect "smoking" as a nuisance behavior, one can use detection expressions such as "cigarette," "person smoking," or "smoker."

[0028] In this embodiment, the behaviors, attributes, and objects to be detected are specified using natural language. By using natural language, the user can arbitrarily and flexibly specify or express the behaviors to be detected, and can easily add or change the behaviors to be detected.

[0029] The language-image matching unit 150 calculates the degree of matching between the detected expression received from the detected expression input unit 130 and the image (person or object) received from the recognition target image detection / tracking unit 120. For example, for the images [bird], [car], and [dog] in FIG. 2, the degree of matching is high if the detected expressions received from the detected expression input unit 130 are "bird," "car," and "dog." For the images [h1] and [h2], the degree of matching is high if the detected expressions are, for example, "woman," "man," or "person standing still." For the image [s1], the degree of matching is high if the detected expressions are, for example, "man," "person smoking," or "person standing smoking." For the image [s2], the degree of matching is high if the detected expressions are, for example, "woman," "person walking," "person wearing a mask," "woman carrying a bag," "woman wearing a coat," or "woman holding a cup of drink." For the image [s3], the degree of matching is high if the detected expressions are, for example, "person smoking," "person smoking while walking," or "man wearing a jacket." In the case of the image [s4], for example, if the detected expressions are "woman," "person sitting in a chair," or "woman smoking a cigarette while sitting," the degree of matching will be high.

[0030] In one aspect, an AI learning model can be used to calculate the degree of matching. The learning model uses, for example, a DNN or the like to learn the relationship between image features and human behavior and objects. The detection expression input from the detection expression input unit 130 is compared with the human behavior output from the learning model, and the degree of matching between the two is calculated. In another aspect, an engine that can handle various data types such as language, images, and audio, such as multimodal AI, may be used to calculate image-language matching.

[0031] For example, when the image of [bird] in Fig. 2 is detected / tracked, the learning model recognizes the object detected from this image as a "bird" and outputs this recognition result. If the detected expression input from the detected expression input unit 130 is a "bird," a "small bird," a "bird perched on a branch," etc., the degree of matching will be high in comparison with the recognition result of the learning model, whereas if the detected expression is a "dog," a "man," a "pedestrian," etc., the degree of matching will be low.

[0032] For example, when the image [s2] in Fig. 2 is detected / tracked, the learning model recognizes this image as, for example, "a woman wearing a mask and walking with a bag" and outputs this recognition result. If the detected expression input from the detected expression input unit 130 is "mask," "woman," "pedestrian," "woman carrying a bag," etc., the degree of matching will be high in comparison with the recognition result of the learning model, while if the detected expression is "man," "person walking while smoking," "person standing still," "bird," "dog," etc., the degree of matching will be low.

[0033] In one aspect, the language-matching unit 150 can use such an AI learning model in a cloud service that provides external AI functions or in an external open source. That is, the language-matching unit 150 provides the image received from the recognition target detection / tracking unit 120 and the detection expression input from the detection expression input unit 130 to the external open source and receives the degree of matching therefrom. Alternatively, the language-matching unit 150 may be equipped with such an AI learning model.

[0034] Figure 5 shows the degree of matching between the images shown in Figures 2 and 3 and the detected expressions [A dog], [A car], [A bird], [A human], [A walking human], [A walker], [walking], [A smoking human], [A smoker], [smoking], [cigarette], [tobacco], and [A jumping human]. Figure 6 shows the degree of matching between the image shown in Figure 4 and the detected expressions [man], [male], [boy], [woman], [female], [girl], [backpack], [knapsack], [carry on back], [handbag], [have a bag], and [grip]. A higher value indicates a higher degree of matching. While many words are shown in this example, the following sentence style is also used: [A janitor, dressed in a yellow sleeveless T-shirt and black trousers, can wear elbow-length gloves or wear a mask or not, vacuuming the floor with a vacuum cleaner or takes tools inside a blue trolley.]

[0035] The language-image matching unit 150 outputs data indicating the degree of matching between the images detected / tracked in consecutive frames of the video and the language of the detected expression to the data storage unit 160, and the data storage unit 160 stores the received data. FIG. 7 shows an example of data output from the language-image matching unit 150. The data shows the degree of matching (correlation) between the images detected / tracked in consecutive frames and the detected expressions (in this example, [man], [woman], [a walking human], [a walker], [A smoking human], [A smoker], [backpack], [knapsack], and [handbag]). For example, the image detected / tracked in the frame at 13:00 has the highest degree of matching with the detected expression [handbag], and the image detected / tracked in the frame at 13:01 has the highest degree of matching with the detected expression [smoking human].

[0036] The determination unit 170 determines or extracts behavior recognition of a person or object that matches the detection expression from the data stored in the data storage unit 160. The determination rules of the determination unit 170 can be changed as appropriate depending on the type of behavior to be recognized, the attributes (resolution, size) of the image data, the usage mode of the application unit 180, and the determination accuracy.

[0037] In one aspect, the determination unit 170 identifies one or more detected expressions / images with a high degree of matching in each frame and recognizes the identified detected expressions / images as activity recognition. For example, in the example of FIG. 7 , from the image detected / tracked in the frame at 13:00, the "handbag" with the highest degree of matching is identified and recognized as activity recognition, and from the image detected / tracked in the frame at 13:01, the "smoking human" with the highest degree of matching is identified and recognized as activity recognition. Furthermore, the determination unit 170 may identify detected expressions / images with a higher degree of matching than a predetermined threshold, or may calculate an average value of the degree of matching in one frame and identify detected expressions / images with a higher degree of matching than the average value.

[0038] In one aspect, the determination unit 170 calculates the average value of the matching degree for the entire tracking time of the detected / tracked images or for a certain period of time, identifies the detected expression / image with the highest average value, and determines this as the recognized behavior.

[0039] In one aspect, the determination unit 170 may determine behavior recognition taking into account image size and the influence of noise. Regarding image size, the determination unit 170 determines the larger size of the detected / tracked image. That is, the determination unit 170 weights the images so that the matching degree of distant images is lower, or weights the images so that the matching degree of nearby images is higher. Also, as shown in FIG. 8 , when multiple images detected / tracked from a frame overlap, it is expected that noise is included in the images. Therefore, in order to suppress the influence of noise, the determination unit 170 weights the overlapping images so that the matching degree of the overlapping images is lower.

[0040] The result of the determination by the determination unit 170 is provided to the application unit 180, which outputs various notifications, warnings, etc. to the user based on the determination result. The manner of output is not particularly limited, but for example, as shown in Fig. 9, when a video is displayed on a display, the relationship between the detected expression / image identified by the determination unit 170 may be visualized. In this example, the image is shown in a rectangular frame, and the detected expression is displayed near the frame.

[0041] In one aspect, the application unit 180 outputs a summary of the attributes and behavior of people and objects, or notifies the appropriate parties when an emergency such as a fall, violence, or a suspicious person is detected.

[0042] When the application unit 180 is used for marketing, it is possible to compile and display customer behavior (e.g., customers looking at shelves / picking up products, etc.), or visualize and display it as a moving image, compile and display customer attributes (e.g., clothing / possessions, etc.), or visualize and display it as a moving image, or visualize and display nuisance behavior (e.g., photography / eating / drinking, etc.), or notify the appropriate person.

[0043] Furthermore, when the application unit 180 is used for security purposes, it is possible to visualize and display suspicious behavior (for example, looking around, looking away, etc.), visualize and display suspicious attributes (for example, backpacks / bags, masks / hat, sneakers / sports shoes), aggregate and display them, visualize and display nuisance behavior (for example, smoking, climbing fences, etc.), or notify the appropriate parties.

[0044] Furthermore, when the application unit 180 is used for management or monitoring, it is possible to visualize and display deteriorated or dirty parts of a structure, or to notify the appropriate party (management company). For example, it is possible to determine the amount of dirt accumulated in a structure such as an apartment building. By extracting images with only a specific angle of view from a fixed camera and performing language-image matching with the detected expression "dirt" from the image two years ago, one year ago, and the present, for example, if the dirt level two years ago was 0.1, the dirt level one year ago was 0.2, and the dirt level currently is 0.3, the difference can be used to determine the degree of dirt. Narrowing down the object to a specific location reduces the amount of information and increases the accuracy of the determination because the image does not contain unnecessary elements.

[0045] Furthermore, it is possible to visualize and display fires, accidents, and other abnormalities in structures and urban areas, natural disasters such as river flooding and cliff collapses, as well as abandoned or neglected objects, and notify the appropriate authorities (fire departments, police stations, etc.).

[0046] As described above, according to this embodiment, behavior recognition is detected from the degree of matching between the detected / tracked image and natural language, so pre-learning to detect each and every behavior is not required, improving the versatility of the behavior recognition system for application to various systems and also improving user convenience.

[0047] In the above embodiment, the language-image matching unit 150 calculates the matching between language and image, but the language-image matching unit 150 may be a language-video matching unit that calculates the matching between language and video. In this case, the language-video matching unit calculates, for example, the similarity (Cos similarity) between vector data of natural language and vector data of video.

[0048] Next, a second embodiment of the present invention will be described. In the behavior recognition system 100 according to the first embodiment shown in Fig. 1, moving images detected / tracked by the recognition target image detection / tracking unit 120 are provided in real time to the language-image matching unit 150. However, in the behavior recognition system 200 according to the second embodiment, as shown in Fig. 10, moving images (or a plurality of still images or frame images) detected / tracked by the recognition target image detection / tracking unit 120 are stored in a moving image storage unit 210. The moving images stored in the moving image storage unit 210 are then provided to a language-image / video matching unit 150A, and the matching results are provided to a determination unit 170.

[0049] The video storage unit 210 stores video for a certain period of time or with a certain storage capacity. Furthermore, the video is not limited to video captured by a single camera, but also includes video captured by multiple cameras installed at multiple locations. In the latter case, the video is associated with the capture location. The video storage unit 210 provides the selected video (e.g., the period or capture location) to the language-image / video matching unit 150A in response to, for example, an instruction from a controller (not shown) or the application unit 180.

[0050] According to this embodiment, since the detected / tracked video images are saved, it becomes possible to easily detect behaviors for a required period or in a required location. In addition, in the first embodiment, the text of the detection expression is determined in advance to detect behavior recognition from images detected / tracked in real time, but in this embodiment, the detection expression can be input later and behavior recognition can be detected.

[0051] Next, a third embodiment of the present invention will be described. Fig. 11 is a diagram showing the configuration of a behavior recognition system according to the third embodiment. In the second embodiment, the behavior recognition system 200 stores detected / tracked video in the video storage unit 210 and performs language matching with the stored video. However, in the third embodiment, the behavior recognition system 300 includes a person / object vector storage unit 310 that stores vector data of video of people / objects, and a language-video vector matching unit 320 that calculates the similarity between the vector data of the video and the vector of the detected expression (text).

[0052] FIG. 12 is a diagram showing an example of vector data of a moving image of a person / object. The vector data of the moving image is not particularly limited, but may be, for example, a vector that analyzes each frame of the moving image and represents its content or characteristics (color, brightness, edge, etc.), a vector that represents the movement or speed of an object in the moving image, a vector that represents the facial posture or expression of a person in the moving image, or a vector that represents an audio waveform in the moving image. The vector data is not limited to one vector data, and may include multiple vector data. It may also be tensor data, which has more dimensions than a vector.

[0053] The language-video vector matching unit 320 calculates the similarity between the vector data of the natural language input from the detection expression input unit 130 and the vector data of the video. The vector data of the natural language is data in which text or sentences are expressed as numerical vectors, and is not particularly limited to, for example, Word Embeddings, which map words to a fixed-dimensional vector space using an algorithm such as Word2Vec or GloVe and express the semantic relationship or similarity between words as vector distance, or Text Embeddings or Document Embeddings, which express sentences as vectors using an algorithm such as Doc2Vec or BERT.

[0054] For example, a vector representing the frequency of appearance of words used in searches on the Internet or on social networking sites is input from the detection expression input unit 130, and the similarity between this vector and the vector data of the video is calculated. This makes it possible to extract from the video actions that match words that are frequently used on the Internet.

[0055] According to this embodiment, by storing vector data of the moving images instead of storing the moving images, it is possible to speed up the search for behavior recognition. Furthermore, by analyzing the vectors of the moving images as in this embodiment, if an unusual vector is detected, it is also possible to detect abnormal behavior.

[0056] Next, a fourth embodiment of the present invention will be described. Fig. 13 is a diagram showing the configuration of a behavior recognition system according to the fourth embodiment, which is a higher-level representation of the behavior recognition systems of the first to third embodiments. The behavior recognition system 400 of the fourth embodiment is configured to include an image acquisition unit 110, a detection / tracking unit 410, a matching unit 420, and a time-series processing unit 430, similar to those of the first to third embodiments.

[0057] The detection / tracking unit 410 detects and tracks images of people, objects, and other objects to be recognized from the images / videos acquired from the image acquisition unit 110. The images to be recognized are not limited to people and objects, and can be any images that can be analyzed and extracted as long as they are features.

[0058] The matching unit 420 calculates the degree of match or similarity between the recognition target image detected / tracked by the detection / tracking unit 410 and specific information for identifying a search target (e.g., an action or an object). The matching unit 420 may include, for example, language-image matching, language-video matching, matching using a multimodal model, and image-image matching.

[0059] As described above, language-image matching involves matching images with linguistic information such as sentences, character strings, and keywords expressed in natural language. The linguistic information may be, for example, vector data in natural language that indicates semantic associations or frequency of appearance. In this case, the recognition target image is vector data of a moving image, and the degree of matching between these vector data is calculated. Finally, the search target identified by the linguistic information is identified or extracted from the recognition target image.

[0060] In image-image matching, the specific information does not necessarily have to be linguistic information. For example, the vector of an image of a person wearing the clothing to be searched for is used as the specific information, and the degree of match between the specific information and the detected / tracked recognition target image is calculated. Finally, the person wearing the clothing specified by the specific information is identified or extracted from the recognition target image.

[0061] The time series processing unit 430 determines the search target searched for by the specific information based on the matching result of the matching unit 420. In one aspect, the time series processing unit 430 can use processing generally referred to as time series processing, such as time series averaging processing of the matching result or time series rule-based processing (for example, a correct behavior is recognized when a threshold value of 0.7 or more continues for 3 seconds). In another aspect, it is also possible to use AI processing in the same way as handling normal time series data.

[0062] When the specific information for identifying the search target includes natural language, it is possible to identify the corresponding search target from the recognition target image using linguistic information such as, for example, "a person carrying a 'backpack' must have a threshold value of 0.8 or above" or "a 5-second period in which a person carrying a 'handback' satisfies a threshold value of 0.4 or below."

[0063] As described above, the behavior recognition system of this embodiment can identify or extract behavior recognition corresponding to identification information from detected / tracked images without requiring prior learning.

[0064] Next, a fifth embodiment of the present invention will be described. The fifth embodiment relates to a modification of the image acquisition unit 110 in the first to fourth embodiments. FIG. 14 is a diagram showing the operational flow of the image acquisition unit of the fifth embodiment. The image acquisition unit 110 receives a series of consecutive frame images (S500) and determines whether or not there is a difference between the frames (S510). Only if there is a difference, the image acquisition unit 110 outputs the frame images to the recognition-target image detection / tracking unit 120, causing the recognition-target image detection / tracking unit 120 to detect and track the recognition-target image (S520). If there is no difference, the image acquisition unit 110 does not output the frame images to the recognition-target image detection / tracking unit 120, preventing the recognition-target image detection / tracking unit 120 from detecting and tracking the recognition-target image (S630).

[0065] In this way, by filtering the image data by the image acquisition unit 110, the calculation cost by the recognition target image detection / tracking unit 120 can be reduced, and the substantial cost of the behavior recognition system can be reduced.

[0066] Next, a sixth embodiment of the present invention will be described. FIG. 15 is a diagram showing the configuration of a behavior recognition system according to the sixth embodiment. The behavior recognition system 600 according to the sixth embodiment uses an LLM (Large Language Model) or a generative AI unit 610 instead of the language-image matching unit 150 of the behavior recognition system 100 according to the first embodiment. The LLM or generative AI unit 610 is a natural language processing model trained using a large amount of text data, such as an AI learning model such as ChatGPT.

[0067] In this embodiment, the LLM or generation AI unit 610 generates text or numbers for the detection expression input from the detection expression input unit 130, or the prompt and the image matching it, or the image matching the output of the LLM, and provides the generated text or numbers to the data storage unit 160. Here, an example using the data storage unit 160 is shown, but in the case of real-time processing, the generated text or numbers may be provided to the determination unit 170 without going through the data storage unit 160.

[0068] For example, when a detection expression such as "Is there anything unusual in the room?" is input from the detection expression input unit 130, the LLM or generation AI unit 610 generates text from the image that matches the detection expression, such as "Looking at the image, it appears that the room is filled with smoke. This could be a sign of a fire. However, this is a guess based on image analysis, and it cannot be confirmed whether this is the actual situation. If the footage is actually from a surveillance camera, appropriate safety measures should be taken immediately. For example, you may want to check if the fire alarm is working and call the fire department."

[0069] The determination unit 170 performs natural language processing of the text generated by the LLM or generation AI unit 610. For example, when a detection expression such as "Is there anything unusual in the room? Is there a fire?" is input from the detection expression input unit 130, the determination unit 170 performs natural language processing determination of "There is a fire" from the generated text and provides this determination result to the application unit 180. The natural language processing by the determination unit 170 is not essential, and the text generated by the LLM or generation AI unit 610 may be provided to the application unit 180 as is.

[0070] The application unit 180 displays the image and the generated text on a display. In one aspect, the application unit 180 extracts keywords from the generated text and notifies the text itself to an external party (e.g., an administrator).

[0071] In this way, according to this embodiment, it is possible to answer questions about detected expressions in natural language not only with images but also with text. Note that the sixth embodiment can also be applied to the second to fifth embodiments.

[0072] Furthermore, as a variation of the sixth embodiment, for example, a specific person or a specific object may be indicated from the image acquired by the image acquisition unit 110, and the LLM or generation AI unit 610 may determine what behavior the specific person or specific object will exhibit.

[0073] 15A, the user designates a rectangular mark surrounding a specific person P, and in response to this designation, the recognition target image detection / tracking unit 120 detects / tracks the specific person P. The LLM or generation AI unit 610 generates a text response regarding the predicted behavior of the detected / tracked specific person P.

[0074] Next, a seventh embodiment of the present invention will be described. FIG. 16 is a diagram illustrating the configuration of a behavior recognition system according to the seventh embodiment. In the behavior recognition system 700 according to the seventh embodiment, the recognition target image detection / tracking unit 120 further includes a function for cropping the detected / tracked image. The cropped portion can be a part of a person or a part of an object. The cropped portion may be specified in advance, or may be determined based on instructions from a user. In the example of FIG. 16, the recognition target image detection / tracking unit 120 crops out parts of a person, such as "hands," "face," and "bag," and tracks the cropped "hands," "face," and "bag." By cropping, unnecessary image data for matching can be deleted, reducing the amount of computation required for matching and ultimately improving matching accuracy.

[0075] For example, if the detected expression is "a person wearing sunglasses," only the "face" can be detected by the recognition target image detection / tracking unit 120 and matched by the language-image matching unit 150. Alternatively, the upper one-third of the "person" detected by the recognition target image detection / tracking unit 120 can be cut out, and the language-image matching unit 150 can match it with an image showing only the upper body.

[0076] By cutting out a part of a person or object in this way, it is possible to reduce unnecessary data in matching, improve the accuracy of calculating the degree of matching, and reduce the calculation time or calculation cost. Note that in the above example, a face image or a finger image is cut out from a human body image, but this is just one example, and it is also possible to cut out, for example, an image of a foot or an image of a hand holding a bag.

[0077] Next, an eighth embodiment of the present invention will be described. FIG. 17 is a diagram showing the configuration of a behavior recognition system according to the eighth embodiment. While the seventh embodiment involved the segmentation of a portion of a person or object, the eighth embodiment enables the segmentation of a physical area from an image. The recognition target image detection / tracking unit 120 segments an image of a predetermined specific area and tracks the specific area. The specific area can be determined arbitrarily, for example, coordinates representing an apartment floor captured by a fixed camera or specific coordinates of an intersection, and the recognition target image detection / tracking unit 120 segments an image of the specific area based on those coordinates. FIG. 17 shows an example in which locations A and B are segmented from an acquired image. By performing such segmentation, unnecessary image data for matching can be deleted, reducing the amount of computation required for matching and ultimately improving matching accuracy.

[0078] For example, if the detected expression is "person wearing sneakers," "woman wearing red sandals," "woman with a ring on her foot," "airplane flying in the sky," or "abandoned object on the road," the language-matching unit 150 calculates the degree of matching between the detected expression and, for example, a cut-out image of the lower half area or the upper half area cut out from the image.

[0079] Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments, and various modifications and changes are possible within the scope of the gist of the invention described in the claims.

[0080] 100, 200, 300, 600, 700, 800: Behavior recognition system 110: Image acquisition unit 120: Recognition target image detection / tracking unit 130: Detection expression input unit 140: List storage unit 150: Language-image matching unit 160: Data storage unit 170: Determination unit 180: Application unit 210: Video image storage unit

Claims

1. A behavior recognition system having an acquisition means for acquiring an image, a detection / tracking means for detecting a recognition target image from the acquired image and tracking the recognition target image, and a calculation means for calculating a degree of matching between specific information for identifying a search target and the recognition target image detected / tracked by the detection / tracking means.

2. The behavior recognition system according to claim 1, further comprising: a discrimination means for discriminating the search target from the recognition target image based on the degree of matching.

3. The behavior recognition system of claim 1, wherein the specific information includes linguistic information of a sentence, character string, or keyword expressed in a natural language for identifying a search target, and the calculation means calculates the degree of matching between the linguistic information and the recognition target image.

4. The behavior recognition system of claim 3, wherein the language information includes vector data of natural language, and the calculation means calculates the degree of matching between the vector data of the recognition target image detected / tracked from a video image and the vector data of the natural language.

5. The behavior recognition system according to claim 1, wherein the specific information includes image information for identifying a search target, and the calculation means calculates a degree of matching between the image information and the recognition target image.

6. The behavior recognition system according to claim 1, wherein the specific information includes voice information for identifying a search target, and the calculation means calculates a degree of matching between the voice information and the recognition target image.

7. The behavior recognition system according to claim 1, further comprising a storage means for storing a recognition target image from the detection / tracking means, and the calculation means calculates a matching degree of the recognition target image stored in the storage means.

8. The behavior recognition system according to claim 1, further comprising a vector data storage means for storing vector data of a recognition target image detected / tracked from a video image, and the calculation means calculates a degree of matching between vector data of a natural language for identifying a search target and the vector data of the recognition target image.

9. The behavior recognition system according to claim 2, further comprising a calculation result storage means for storing the calculation result of the calculation means, and the discrimination means discriminates the search target based on the calculation result stored in the calculation result storage means.

10. The behavior recognition system according to claim 1, further comprising an input means for inputting the specific information.

11. The behavior recognition system according to claim 2, wherein the recognition target image includes a person, and the discrimination means discriminates the behavior and / or behavior attributes of the person.

12. The behavior recognition system according to claim 2, wherein the recognition target image includes a person and an object, and the discrimination means discriminates the behavior and / or behavior attributes of the person including the object.

13. The behavior recognition system according to claim 1, wherein the acquisition means has a function of filtering images and provides the images to the detection / tracking means when there is a difference between images between frames.

14. The behavior recognition system of claim 1, further comprising a cut-out means for cutting out the recognition target image based on the specific information, and the calculation means calculates the degree of matching between the specific information and the image cut out by the cut-out means.

Images