Change captioning system

Abstract

To provide a change captioning system that effectively searches for a state change in a data sequence and presents a search result thereof.SOLUTION: A method, which causes a system to search for a state change in a data sequence, comprises the steps of: acquiring data in the data sequence in order of generation; determining state transition and a change caption between each acquired data and past data of the acquired data; updating change caption information according to the state transition and the change caption; updating state transition information according to the state transition; receiving a search query including a natural language; selecting a change caption corresponding to the natural language from change caption information; and presenting state change information associated with selected change caption in the state transition information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a transformation captioning system. [Background technology]

[0002] There are many situations where it is necessary to take differences between past and current information, such as visually detecting abnormalities or checking the progress of construction projects. Automatically generating explanations of the differences in a large number of images obtained from surveillance cameras and patrol robots and processing them later into searches and reports can be expected to contribute to improving work efficiency.

[0003] Until now, in order to make difference information searchable in natural language, humans had to manually provide explanatory text (captions) and register them in a database. However, in recent years, a technology field called Change Captioning has emerged that "captures changes while automatically generating explanatory text for the changes," making it possible to automate the registration process.

[0004] In addition, JP 2020-134955 A discloses that "the inspection report support system 1 has a data processing server 3. The data processing server 3 combines a large number of images obtained by photographing the object to create multiple images representing the entire inspection object and different parts, and displays on the user terminal 7 an inspection work screen displaying the images requested by the user terminal 7 and a comment input tool. Comments entered by the user on the work screen are saved in association with the displayed images. The data processing server 3 automatically creates a report in accordance with the report form using the saved comments and the images associated with them" (summary). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2020-134955 A Summary of the Invention [Problem to be solved by the invention]

[0006] To make difference information a search target in natural language, the method of generating sentences in the natural language to be searched becomes problematic. It is unrealistic in terms of processing costs to apply Change Captioning to all pairs of stored data in advance to generate sentences to be searched. For this reason, it is conceivable to generate sentences using Change Captioning only for pairs with the closest dates and times to each data, prepare them as search targets, and list the sentences found from the prepared search targets. However, this method may overlook state changes that should be detected, and may present information on state changes that is not organized.

[0007] Therefore, a technique that can efficiently search for state changes in a data sequence and present the results of that search is desirable. [Means for solving the problem]

[0008] One aspect of the present invention is a change captioning system that includes a calculation device and a memory device, wherein the memory device stores state transition information regarding state transitions and change caption information, the state transition information includes information on state changes for each state transition, and the change caption information associates and stores the state transitions in the state transition information with the change captions, the calculation device acquires data in a data sequence in order of generation, determines state transitions and change captions between each acquired data and past data of the acquired data, updates the change caption information using the state transitions and the change captions, updates the state transition information using the state transitions, accepts search queries including a natural language, selects a change caption corresponding to the natural language from the change caption information, and presents information on state changes associated with the selected change caption in the state transition information. Effect of the Invention

[0009] According to one aspect of the present invention, it is possible to effectively search for state changes in a data sequence and present the search results. Further features related to the present invention will become apparent from the description of the present specification and the accompanying drawings. Furthermore, problems, configurations and advantages other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0010] [Figure 1] 1 illustrates a schematic diagram of an example of a logical configuration of a change captioning system according to an embodiment of the present specification. [Diagram 2] 1 shows an example of a hardware configuration that can be used for a change captioning system. [Diagram 3] 13 shows an example of the configuration of caption log information contained in a caption log DB. [Figure 4] An example of a state transition modeled by a finite automaton is shown below. [Diagram 5] Here is an example of state-change correspondence information: [Figure 6] An example of a change ID list is shown below. [Figure 7] 1 shows a flowchart of an overview of processing performed on a captured image by the change captioning system. [Figure 8] An example of a configuration for executing a change detection process for identifying changed areas from two image inputs is shown. [Figure 9] An example of a configuration for executing a process for generating a change caption sentence from a detected region will be described. [Figure 10] An example of a change caption template is shown. [Figure 11] The following shows examples of rules for determining changes based on the text expression and the change mask conditions. [Figure 12] An example of caption log information is shown below: [Figure 13] An example of a state model is shown below. [Figure 14] An example of state-change correspondence information is shown below. [Figure 15] An example of a mutating ID list is shown below. [Figure 16]16 shows a changing ID list obtained by updating the changing ID list shown in FIG. [Figure 17] A state transition model in which state S2 has been added is shown. [Figure 18] State S2 indicates the added state-change correspondence information. [Figure 19] Indicates updated caption log information. [Figure 20] The state transition model is shown with the information of state S3 added. [Figure 21] The state-change correspondence information is shown with the information of state S3 added. [Figure 22] 1 shows a flowchart of an example of processing by the output generating unit 110. [Diagram 23] The following shows an example of information on whether or not a change ID occurred within a specified period, its appearance time, and its end time. [Figure 24] An example of a UI for change caption search is shown. [Diagram 25] An example of a UI for displaying change caption search results is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings. When necessary for convenience, the description will be divided into a number of sections or embodiments, but unless otherwise specified, they are not unrelated to each other, and one is related to the other as a partial or complete modification, detail, supplementary explanation, etc. Furthermore, in the following, when the number of elements (including the number, numerical value, amount, range, etc.) is mentioned, it is not limited to the specific number, and may be more than or less than the specific number, unless otherwise specified or when it is clearly limited to a specific number in principle.

[0012] The system of an embodiment of this specification may be a physical computer system (one or more physical computers) or a system built on a group of computing resources (multiple computing resources) such as a cloud infrastructure. The computer system or the group of computing resources may include one or more interface devices (including, for example, a communication device and an input / output device), one or more storage devices (including, for example, a memory (main memory) and an auxiliary storage device), and one or more arithmetic devices.

[0013] When a function is realized by a computing device executing a program including instruction codes, the defined processing is performed using a storage device and / or an interface device, etc., as appropriate, and therefore the function may be considered to be at least a part of the computing device. Processing described with the function as the subject may be processing performed by the computing device or a system having the computing device. The program may be installed from a program source.

[0014] The program source may be, for example, a program distribution computer or a computer-readable storage medium (for example, a computer-readable non-transitory storage medium). The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0015] A change captioning system according to an embodiment of the present specification will be described below. The change captioning system has a function of searching for previously calculated change caption content in natural language. The change captioning system records a series of previously calculated changes as state transitions. The change captioning system executes a search process according to a search query and generates information to be displayed as a search result.

[0016] 1 is a schematic diagram showing an example of a logical configuration of a change captioning system according to an embodiment of the present specification. The change captioning system includes an acquired data control unit 104, a change captioning calculation unit 106, a change state management unit 107, an output generation unit 110, and a search information input / output unit 111. The change captioning system stores an acquired data storage database (DB) 105, a caption log DB 108, and a state transition information DB 109.

[0017] 2 shows an example of a hardware configuration that can be used for the change captioning system. The change captioning system has a general computer configuration, and includes a processor 321, a main memory 322, an auxiliary memory 323, an input device 324, an output device 325, and a network interface 327. Each part of the change captioning system is communicatively connected to each other via a communication means such as a bus 326. Note that the change captioning system may be one in which all or part of the configuration is realized by a virtual resource such as a cloud server.

[0018] The processor 321 is configured using a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), etc. The processor 321 reads and executes a program stored in the main memory 322, thereby realizing the functions of the change captioning system.

[0019] The main memory 322 is a device for storing programs and data, and may be a Read Only Memory (ROM), a Random Access Memory (RAM), a Non-Volatile RAM (NVRAM), or the like.

[0020] The auxiliary memory 323 is, for example, an NVRAM such as an SSD (Solid State Drive), an SD memory card, an optical storage device such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), an HDD (Hard Disc Drive), a storage area of ​​a cloud server, etc. The auxiliary memory 323 includes a non-transitory storage medium for storing programs and data. The programs and data stored in the auxiliary memory 323 are loaded into the main memory 322 as needed.

[0021] For example, the auxiliary memory 323 may store programs corresponding to the acquired data control unit 104, the change captioning calculation unit 106, the change state management unit 107, the output generation unit 110, and the search information input / output unit 111. In addition, the auxiliary memory 323 may store an acquired data storage database (DB) 105, a caption log DB 108, and a state transition information DB 109.

[0022] The input device 324 is an interface that accepts input of information, and is, for example, a keyboard, a mouse, a touch panel, a card reader, a microphone, etc. Alternatively, the change captioning system may be configured to accept input of information between another device and the change captioning system via some communication means.

[0023] The output device 325 is an interface that outputs various types of information, and is, for example, a screen display device such as a liquid crystal monitor, LCD (Liquid Crystal Display), or a graphics card, a printer, or an audio output device such as a speaker. Alternatively, the change captioning system may be configured to output information between itself and other devices via some communication means. The network interface 327 is a device that allows the change captioning system to communicate with other devices. Some of the components shown in FIG. 1 may be omitted, and other components may be added.

[0024] The change captioning system can be configured with one or more computers. Thus, the change captioning system can include one or more processors and one or more storage devices. The one or more processors operate as a predetermined functional unit by executing a program stored in the one or more storage devices.

[0025] Returning to Fig. 1, acquired data position and time information 101 input from outside together with acquired data includes information indicating where and when acquired data 102 was acquired. Examples of position information include coordinate information or GPS information assigned on a site map, or an ID assigned to a surveillance camera or the like.

[0026] In one embodiment of the present specification, change captioning is not performed on all previously accumulated data by exhaustively searching for change captioning on the acquired data 102, but data accumulated for each scene or device corresponding to the acquired data 102 is subject to change captioning. The acquired data position and time information 101 is utilized to select the scene or device corresponding to the acquired data 102. Depending on the design, the acquired data position and time information 101 may be omitted as long as the order of the acquired data is maintained. The order according to time of the acquired data is the order in which the data was generated.

[0027] The acquired data 102 is data that is one of the inputs to the change captioning calculation unit 106. This includes various data acquired from outside the system by sensors such as a camera and a microphone. Change captioning can target various types of data such as still images (including still images extracted from videos), 3D point clouds, and sound data. In the following description, the data type is described as still images. Before calculation by the change captioning calculation unit 106, a storage process is performed by the acquired data control unit 104.

[0028] The comparison data 103 is data that is input to the change captioning calculation unit 106, similar to the acquired data 102. Since the change captioning technique finds changes by comparing two pieces of data of the same type, the acquired data 102 and the comparison data 103 are data of the same type (still images, sounds, etc.).

[0029] For example, based on acquired data position and time information 101, the most recent data accumulated in the past corresponding to the same position (image acquisition position) is extracted as comparison data 103. If permissible from the viewpoint of change detection, comparison data 103 does not have to be the immediately previous data. Acquired data control unit 104 is responsible for interpreting the position information and retrieving data. Acquired data storage DB 105 is the database from which the data is retrieved.

[0030] The acquired data control unit 104 retrieves and writes data to the acquired data storage DB 105. One of the processing cases is retrieval of comparison data 103. Another processing case is writing the acquired data 102 together with acquired data position time information 101 to the acquired data storage DB 105 as the latest data.

[0031] Furthermore, when the search function is executed, the acquired data control unit 104 processes access to retrieve data from the output generation unit 110 in order to retrieve stored data corresponding to the search results. The acquired data control unit 104 assigns a unique data ID to each piece of data when it is saved. This information is saved in the acquired data saving DB 105, and the acquired data 102 and comparison data 103 are also held as their associated information.

[0032] The acquired data storage DB 105 is a database that stores acquired data 102 in a format that allows selection from the viewpoint of location based on acquired data location and time information 101. The write and retrieve functions are performed by an acquired data control unit 104.

[0033] The change captioning calculation unit 106 receives the acquired data 102 and the comparison data 103 as input, detects the change position and estimates the change content described in natural language, and then transmits the data to the change state management unit 107. The details of the process of the change captioning calculation unit 106 will be described later.

[0034] The change state management unit 107 receives as input the output from the change captioning calculation unit 106 (including the change detection position and description related to the change), the acquired data position time information 101, and data ID information attached to each data of the acquired data 102 and the comparison data 103. Then, it performs an update process of the state change at the data acquisition position, and stores various information in the caption log DB 108 and the state transition information DB 109. The details of the process of the change state management unit 107 will be described later.

[0035] The caption log DB 108 is one of the databases that saves the data output by the change state management unit 107, and stores information on the change caption. The information managed by the caption log DB 108 is position and time information, state transition information, and change caption output. In this way, the state transition and the change caption are stored in association with each other.

[0036] Fig. 3 shows an example of the configuration of caption log information 181 included in the caption log DB 108. In the example configuration shown in Fig. 3, the caption log DB 108 stores tabular data for each data acquisition position. The caption log information 181 in Fig. 3 shows a caption log whose data acquisition position ID is "1".

[0037] The caption log DB 108 stores the acquisition position ID, acquisition time, state transition information, data ID, and changed caption output for each pair of acquired data and comparison data. The acquisition time indicates the time when the acquired data was acquired. The state transition information indicates the state transition (pair of transition states) between the comparison data and acquired data. The data ID indicates the IDs of the acquired data and comparison data.

[0038] The state transition information DB109 is one of the databases that stores data output by the change state management unit 107. The data to be stored includes a state transition model (e.g., finite automaton) defined for each data acquisition position, state-change correspondence information that represents state changes in the state transition model, and a change ID list that manages the changes. The state transition model may be represented in any format and may include time information. A unique ID (state ID) is assigned to a state S represented by the state transition model. The ID corresponds to the description of the state transition information in the caption log DB108.

[0039] A state transition is information focusing on a changing state, and can be represented by a state pair of a start state and an end state. A state transition can also be defined between the same states. A state change is information focusing on the content of a change in a state transition between different states, and can be distinguished in this specification by a change caption, or a change caption and a change position in an image. Information on a state transition or a state change can indicate a state transition and a state change.

[0040] FIG. 4 shows an example of a state transition 190 modeled by a finite automaton. The symbols in the circles indicate state IDs. FIG. 5 shows an example of state-change correspondence information 191 of the state transition model 190. The state-change correspondence information 191 indicates the change IDs of each state ID. The change IDs of each state ID are a list of IDs of changes that have not been completed between the initial state and the state. FIG. 6 shows an example of a change ID list 192. The change ID list 192 manages the change direction, caption and its embedding vector, and change position of each change ID. The change position is represented by the relationship between change mask images described later. Different change IDs are given to different pairs of embedding vectors and change positions. In this way, the change ID list 192 stores association information between state changes and change captions.

[0041] The state transition information DB 109 is state transition information, and includes a state transition model 190, its state-change correspondence information 191, and a change ID list 192. The contents and usage examples of the state transition model and the change ID list will be described in the explanation of the specific processing of the change state management unit 107.

[0042] The output generation unit 110 receives a search query (a time range to be searched and a natural language expressing a change to be searched) from the search information input / output unit 111, searches for corresponding data, and returns the output to the search information input / output unit 111. Details of the process will be described later.

[0043] The search information input / output unit 111 has a function of displaying a search user interface (UI) to the user and presenting search results in the same way. An example of the search UI will be described later. In the search UI, it is possible to specify a period and a natural language for change search. In addition, the search results can include a time series chart that allows the presence or absence of changes during the specified period and the completion or non-completion of changes to be confirmed.

[0044] The change captioning system according to an embodiment of the present specification performs two processes as a rough processing flow: (I) a process when an image is acquired from outside, and (II) a process when a search function is executed. First, the processing flow when an image is acquired from outside (I) will be described.

[0045] (I) Processing when images are acquired from outside 7 shows a flowchart of an outline of processing performed on an acquired image by the change captioning system. The change captioning system acquires acquired data position and time information 101 and acquired data 102 from the outside (S11).

[0046] The acquired data control unit 104 receives the acquired data position and time information 101 and the acquired data 102 (S12). The acquired data control unit 104 stores the acquired data in the acquired data storage DB 105. At this time, a unique data ID is issued and assigned to the acquired data 102 (S13). The acquired data control unit 104 refers to the acquired data position and time information 101, and loads the comparison data 103 having a time closest to that of the acquired data 102 from the acquired data storage DB 105 (S14).

[0047] The change captioning calculation unit 106 receives the acquired data 102 and the comparison data 103, and outputs data such as a change caption to the change state management unit 107 (S15).

[0048] The change state management unit 107 receives the acquired data position and time information 101 together with the output of the change captioning calculation unit 106, and loads corresponding information from the caption log DB 108 and the state transition information DB 109 (S16). The change state management unit 107 executes a state update process, and stores the results in the caption log DB 108 and the state transition information DB 109 (S17).

[0049] The following describes the details of the process S15 by the change caption calculation unit 106. As an example of the process, a case will be described in which the acquired data 102 and the comparison data 103 are still images. Note that any processing method other than the change captioning method described here can be applied as long as it is a method that can receive two still images as input and generate an explanation of the change location and the change content between the data as output.

[0050] In this embodiment, a method for implementing change captioning using a recognition model called CLIP (Contrastive Language-Image Pre-Training) is described. CLIP is a recognition model trained with hundreds of millions of paired data of images and their descriptions (captions), and has an image encoder and a text encoder that can embed features in the same vector space. This description is based on the assumption that VisionTransformer is used as the image encoder.

[0051] The process of the change captioning method using CLIP is divided into two parts: (i) a change detection process to identify each changed area from two image inputs, and (ii) a process to generate change caption text from the detected areas.

[0052] FIG. 8 shows a block diagram for executing process (i). Input data I A 401, I B 402 is a still image. For example, the input data I A 401 corresponds to the comparison image 103, and the input data I B 402 corresponds to the acquired image 102. These are passed through the image encoder 411 of CLIP to obtain a vector V A , V B 8, one image encoder 411 is represented by two blocks for ease of illustration. Note that two image encoders may be provided. Next, the COSINE dissimilarity calculation unit 422 calculates the cosine dissimilarity as a similarity index between the vectors by the following formula.

[0053]

number

[0054] Next, the changed region estimation unit 421 calculates the derivative of the calculated cosine dissimilarity. The derivative is performed according to the following formula. For ease of illustration, one changed region estimation unit 421 is represented by two blocks in Fig. 8. Note that two changed region estimation units may be provided.

[0055]

number

[0056] Here, A n and B n is the vector extracted before the Attention calculation (immediately after the first LayerNorm) in the final AttentionLayer (AL) 415 of the VisionTransformer in each CLIP 411. The purpose of this calculation is to identify areas in the image that increase the dissimilarity, that is, areas that are changed.

[0057] GradientA and GradientB are normalized to the range of 0-1 and converted to the input data I A 401, I B Then, the vector is interpolated and shaped to match the size of the image I402. Then, the changed region estimation unit 421 performs binarization using a threshold value. Finally, the image I A Change mask image D where the change position of 401 is value 1 and the other positions are value 0 A 431 is generated. A similar change mask image D is generated from GradientB. B 432 is generated. This is the end of the change detection process.

[0058] Next, we will explain the process of (ii) generating change caption text from the detected areas. In process (ii), we obtain text that best expresses each change area obtained in process (i) and text that expresses the presence or absence of a change and the type of change, and then apply these to a predefined change caption template to generate a change caption.

[0059] FIG. 9 shows an example of a configuration for executing the process (ii). The image cutout unit 451 cuts out the change mask image D A Based on 431, the input image I AOnly the change area in 401 is extracted. This is input to the image encoder 411 of CLIP, and the output vector V cropA Further, the image cutout unit 451 cuts out the change mask image D B Based on 432, the input image I B Only the change area in 402 is extracted. This is input to the image encoder 411 of CLIP, and the output vector V cropB For ease of illustration, one image cutout unit 451 and one image encoder 411 are each represented by two blocks. Alternatively, two image cutout units and two image encoders may be provided.

[0060] In parallel, the text representation data 454, which is a set of representation texts representing the change regions, is input to the CLIP text encoder 463, and the CLIP output vector M text One type of text expression data is an object description consisting of nouns alone or a combination of modifiers and nouns, such as "a person wearing a black hat," "brown liquid," or "blue object." Expressions that you want to appear in the change caption can be freely included in any number.

[0061] In addition, background descriptions may be included to classify changes (appearance, disappearance, movement, attribute change, no change) as described later. For example, sentences expressing areas such as walls, ground, and sky. In the appearance and disappearance change patterns, the output vector V cropA , V cropB Since one of the above information refers to the background, by clarifying which was the background, it is possible to distinguish whether the change is an appearance or disappearance change.

[0062] A single text representation T i Let V be the output vector of the CLIP text encoder for text_i Then, the set of representation texts St∋T i The output M from the CLIP text encoder for (i=1…n) text is defined as follows:

[0063]

number

[0064] M text is the CLIP output vector V of the cut image. cropA , V cropB Together with each other, they are input to the image-text similarity calculation unit 471. The image-text similarity calculation unit 471 outputs the output vector V cropA , V cropB The closest expression to text Select from.

[0065] Output vector V crop The specific calculation for max(V crop , M T text ) by M text V text_i and V crop Then, calculate the maximum inner product value with V text_i The original text representation T i is obtained from the text expression data set St454. cropA , V cropB The best text representation obtained for each A , Text B It is written as follows.

[0066] Finally, a method for obtaining the final output, the change caption CC481, is described. The change caption CC481 is obtained as the output of the change caption generator 475. As an input to the change caption generator 475, the best text representation Text A , Text B and the change mask image D A , D B , and five of the change caption template data are given.

[0067] Fig. 10 shows an example of a change caption template. The change caption template is a collection of sentences with holes corresponding to five basic changes, such as appearance, disappearance, movement, attribute change, and no change, as shown in Fig. 10. After determining which of these basic changes the change corresponds to, the change caption generating unit 475 fills the holed part of the sentence with the best text representation Text A , Text B The filled output is output.

[0068] FIG. 10 shows a simple example. There may be multiple templates for one type. For example, verbs such as "appeared" and "disappeared" can be replaced with other expressions, or a change mask image D A , D B The position information of the change obtained from the correspondence relationship is added to the template sentence of the movement change. In this case, the final output sentence may be determined through a process such as outputting the sentence evaluated as the most natural using an existing method for estimating the naturalness of the sentence.

[0069] A simple way to determine the basic change is to use the best text representation. A , Text B , and the change mask image D A , D B The decision can be made by case-by-case analysis based on the information obtained from the text representation Text A , Text B , and the change mask image D A , D B Here is an example of a rule for determining changes based on the above conditions. In order to improve the accuracy of this determination process, it is considered a natural extension to use a discrimination model, such as a neural network, to learn the patterns of determination, rather than simply dividing the cases.

[0070] The output sent from the change caption calculation unit 106 to the change state management unit 107 is a change caption CC and a change mask image D indicating the change detection position. A , D B, and the data IDs associated with the input acquired data 102 and comparison data 103.

[0071] The explanation so far has been based on the assumption that one data set contains only one change. If one data set contains multiple changes, the change mask image D is generated after performing spatiotemporal alignment of the data 102 and 103 (in the case of images, image correspondence processing, etc.) as preprocessing. A , D B According to the multiple change positions detected in and the image contents of the change positions, pairing of the change positions is performed, and the change caption calculation unit 106 performs processing for each change position pair. In this case, each output of the change caption calculation unit 106 increases by the number of pairs of change positions.

[0072] The process S17 by the change state management unit 107 is described below in detail. The main processes of the change state management unit 107 are (i) a process of updating the change state, and (ii) a process of saving various information in the caption log DB 108 and the state transition information DB 109. The process (ii) simply overwrites the information updated in the process (i) to the original save location. The process (i) is described below.

[0073] In this process, first, based on the acquired data position time information 101 received as input, caption log information 181 for the corresponding position is extracted from the caption log DB 108, and a state transition model 190, state-change correspondence information 191, and change ID list 192 for the corresponding position are extracted from the state transition information DB 109.

[0074] Fig. 12 shows an example of extracted caption log information 181, and Figs. 13, 14 and 15 show examples of extracted state transition model 190, its state-change correspondence information 191 and its change ID list 192. From reading the information shown in Figs. 12 to 15, it can be seen that the "state in which a man in black appears" defined in state S1 is the latest state of the corresponding position.

[0075] Here, suppose that the newly input change caption is "A blue flying object appears." First, the change state management unit 107 refers to the change ID list 192 to check whether this change is known or unknown. In this case, the embedding vector V c It is.

[0076] Let the embedding vector of the input change caption be V c_input When this is done, the change state management unit 107 checks each V c The cosine similarity is calculated for each. If the cosine similarity is greater than or equal to a certain threshold, the input change is determined to be known, otherwise it is determined to be unknown. This determination makes use of the property that the similarity between embedding vectors of sentences with similar meanings is high. By referring to the similarity between embedding vectors without directly comparing sentences, it is possible to absorb some of the variation in sentence expression.

[0077] In this case, each V c and V c_input The similarity is low, and the input change is determined to be unknown. If multiple change IDs with high similarity are found, it is possible to determine one change ID based solely on the cosine similarity, or it is also possible to determine one change ID based on the degree of agreement (e.g. distance) of the change position among multiple change IDs that satisfy a certain level of cosine similarity. This selection can be determined for each monitoring target (for example, buildings such as factories).

[0078] Here, since it is determined that the change is unknown, the change state management unit 107 first registers the input change as a new change in the change ID list 192. At that time, the estimated caption (C*-inverse) and its embedding vector (V c*_i) is registered. Reverse change captions are created by semantically reversing the forward change captions. For example, appearances and disappearances are swapped, and the order of movement positions and attribute changes is reversed. This information can be used to distinguish whether a future change will complete a known change or whether it is an entirely new change.

[0079] Fig. 16 shows a change ID list 192 obtained by updating the change ID list 192 shown in Fig. 15. As described above, a record with a change ID of "C2-forward" and a record with a change ID of "C2-inverse" have been added.

[0080] Next, the change state management unit 107 proceeds to a new state registration process. A new state S2 is defined at a position connected to state S1, with the transition condition from state S1 set to C2-forward. FIG. 17 shows a state transition model 190 to which the new state S2 has been added. The transition condition from state S2 to state S1 is C2-inverse. The change state management unit 107 further adds state S2 to state-change correspondence information 191, and sets the retained change ID list to [C1-forward, C2-forward].

[0081] 18 shows state-change correspondence information 191 to which state S2 has been added. The held change ID list shows a change ID "C1-forward" from state S0 to state S1, and a change ID "C2-forward" from state S1 to state S2.

[0082] Finally, the caption log information 181 (state transition list) is updated. Fig. 19 shows the updated caption log information 181. A third record indicating the state transition from state S1 to state S2 has been added.

[0083] Next, the processing that is performed when a known change (including a change corresponding to change ID list 192) occurs will be described. When a known change occurs, there are two cases: (i) a transition to a known state, and (ii) a transition to an unknown state. In the latter case, a new state must be defined, as in the addition case of state S2 above. Cases (i) and (ii) are distinguished by whether or not there is a match between the retained change ID list of the result of reflecting a known change in the current state and the retained change ID list of each state held by state-change correspondence information 191.

[0084] For example, suppose that a change corresponding to C2-inverse occurs following the addition of state S2 above. When a known inverse change occurs like this, the corresponding forward change C2-forward is deleted from the retained change ID list of the current state. Then, the updated retained change ID list becomes [C1-forward], which matches the existing state S1, so there is no need to add a new state, and the state transition corresponding to C2-inverse becomes from state S2 to state S1.

[0085] As another case, suppose that a change corresponding to C1-inverse occurs in state S2. In this case, the change itself is known, but since a state corresponding to the updated retained change ID list [C2-forward] does not exist in state-change correspondence information 191, a new state S3 is added together with the retained change ID list [C2-forward]. Figure 20 shows state transition model 190 to which information on state S3 has been added, and Figure 21 shows state-change correspondence information 191 to which information on state S3 has been added.

[0086] Basically, the above process is repeated every time an update is made, but over time, erroneous states due to detection or caption errors may accumulate. To deal with this, it is possible to provide a manual reset function, such as by preparing a new state transition model in a format separate from past state transition models and associated data, and registering the current state as the initial state.

[0087] Furthermore, when one data item contains multiple changes, the output change captions from the change captioning calculation unit 106 will be the same number as the number of changes. In this case, the process of the change state management unit 107 described above is performed for each change, and the point in time when the process for all changes is completed is regarded as the final state. In the above example, state changes and change captions are registered in response to image acquisition. In another example, the above process may be performed for each image of an image sequence acquired within a specific period and registered in a database.

[0088] At the end of (I), we will explain the special settings for when the change type is "movement change". When the object that is changing is a human or an animal, it may move frequently within the screen, and movement changes may be observed frequently. The inverse change of a movement change is defined as a coordinate change in the opposite direction to the forward change, but depending on the object that is changing, the conditions for the inverse change may be difficult to meet, and a large number of new states may be defined every time the object moves, which may result in an increase in the amount of data required for state management. As a special setting to prevent this, when the change type is determined to be "movement", it is possible to assume that a self-transition has occurred from the state Sn that belongs to that time to the same state Sn, without determining whether the change type is known or unknown, or whether the state is new or not. In this case, the state transition model, state-change correspondence information, and change ID list are not updated, and only the change caption log information is updated. Under this setting, it is possible to maintain the search function for the change caption output recorded in the change caption log while preventing the proliferation of new states.

[0089] (II) Processing when performing a search function The process when using the search function is described below. The process when using the search function is substantially the same as the process of the output generation unit 110. The output generation unit 110 receives a search query as an input from the change caption search UI. Furthermore, it executes a process of collecting relevant data from the acquired data storage DB 105, the caption log DB 108, and the state transition information DB 109, and generating an output including a time series chart and detailed information of each change. This information is displayed, for example, in a change caption search result display UI.

[0090] 22 shows a flowchart of an example of processing by the output generation unit 110. The output generation unit 110 receives a search query input by a user from the search information input / output unit 111 (S31). Next, the output generation unit 110 extracts information from the caption log DB 108 and state transition information DB 109 that corresponds to the specified period, data type, and specified location of the search query (S32). Specifically, caption log information, state transition model, state-change correspondence information, and change ID list are extracted. Note that if the specified period, data type, and specified location are not specified, the entire period, all data, and all locations are targeted.

[0091] Next, the output generation unit 110 reproduces the state transition from the start to the end in the state transition model for each specified location, with the oldest state in the specified period as the start and the most recent state as the end. In the process, information on the presence or absence, appearance time, and end time of each change ID within the specified period is collected (S33).

[0092] FIG. 23 shows an example of information obtained as a result of reproducing state transitions, and indicates the presence or absence, appearance time, and end time of each change ID within the specified period. If the oldest state S within the specified period is the initial state S 0 If it is different, the change corresponding to the change ID held by state S is recorded as having already occurred since the start of the specified period (as mentioned above). Conversely, if the latest state S' in the specified period is S 0 If the change ID is different from the change ID, the change corresponding to the change ID held by state S' is recorded as continuing even at the end of the specified period (continuation). In this way, the output generation unit 110 obtains time-series information such as already appeared, appeared, ended, and continued within the specified period for each change ID, including changes that appear or end outside the specified period.

[0093] Next, the output generating unit 110 deletes, from among the pieces of changing ID information collected in the preprocessing, those that do not match the designated keyword of the search query (S34). Note that if the designated keyword does not exist, no particular processing is performed.

[0094] Next, the output generation unit 110 generates a time series chart to be displayed at the top of the UI for displaying the change caption search results shown in Fig. 25 from each piece of change ID information remaining in the preprocessing (S35). Furthermore, the output generation unit 110 collects necessary information from the caption log and the acquired data storage DB based on each piece of change ID information remaining in the preprocessing, and generates detailed information to be displayed at the bottom of the UI for displaying the search results shown in Fig. 25 (S36). Finally, the output generation unit 110 outputs the results to the search information input / output unit 111 (S37).

[0095] Figure 24 shows an example of a change caption search UI. In the change caption search UI, the user requests a search by specifying the period to be searched, words in the text, the type of data to be searched, and the location of the data. To specify the location, for example, the location of each data registered in the database is displayed on a map, and an area that can be specified by clicking is displayed.

[0096] FIG. 25 shows an example of a UI for displaying change caption search results. The upper part shows a time series chart of the relevant changes included in the target period. The chart is generated in step S35 of FIG. 23. The horizontal axis of the chart shows date and time. Each bar represents the duration of one change. For example, change numbered "1" occurs within the target period and ends within the target period. Change numbered "2" occurs within the target period and does not end within the target period. Change "3" occurs before the target period and ends within the target period. Although not explicitly shown in FIG. 25, the relevant change caption may be superimposed on each bar.

[0097] The lower part presents detailed information about each change. The detailed information is generated in step S36 of Fig. 23. For example, the detailed information may indicate the location where the target image was acquired, the occurrence and end times (if any) of the change (event) within the target period, the change caption, and images (if any) before and after the occurrence and end of the change within the target period. Some of the detailed information may be omitted, or other information may be presented.

[0098] The UI for displaying the change caption search results allows the user to obtain information about the period of an event that occurred at a target location beyond the target period, and to know the details of the event.

[0099] As described above, according to an embodiment of the present specification, it is possible to suppress the amount of pre-processing for a search and generate information to be output in response to a search query using lightweight processing that does not use images. In addition, it is possible to detect status changes that occur outside the specified search period and continue within the specified period, and present them to the user. In addition, even if the specified period is long, different status changes are distinguished from one another and information on their start and end is presented. The user can easily recognize information on status changes.

[0100] The present invention is not limited to the above-mentioned embodiment, and various modifications are included. For example, the above-mentioned embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0101] In addition, the above-mentioned configurations, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a computing device interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.

[0102] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0103] 101 Acquired data location and time information 102 Acquired Data 103 Comparison Data 104 Acquired data control section 105 Acquired data storage DB 106 Change Captioning Calculation Unit 107 Change Status Management Unit 108 Caption Log DB 109 State transition information DB 110 Output Generation Unit 111 Search information input / output unit

Claims

1. 1. A change captioning system, comprising: A computing device; a storage device; The storage device stores state transition information relating to state transitions and change caption information; The state transition information includes information on state changes of each state transition, The change caption information stores a state transition in the state transition information and a change caption in association with each other; The computing device includes: Obtaining data in the data sequence in order of generation; determining state transitions and change captions between each captured data and past data of the captured data; updating the change caption information according to the state transition and the change caption; updating the state transition information according to the state transition; Accepts search queries that include natural language, selecting a variation caption corresponding to the natural language from the variation caption information; A change captioning system that presents state change information associated with the selected change caption in the state transition information.

2. 2. The change captioning system of claim 1, comprising: the search query includes a specified time period; the state transitions are associated with time information; The computing device includes: generating a series of state transitions for the specified period from the state transition information; A change captioning system that selects, from the change caption information, a change caption that corresponds to the state transition and the natural language for the specified period.

3. 2. The change captioning system of claim 1, comprising: The computing device presents time information of the state change associated with the selected change caption.

4. 3. The change captioning system of claim 2, comprising: The computing device includes: determining a state change that continues in the oldest state during the specified period by referring to the state transition information; A change captioning system that presents information about the ongoing state change.

5. 2. The change captioning system of claim 1, comprising: The state transition information includes association information between a state change and a change caption, The computing device includes: In updating the state transition information, by referring to the association information and based on a similarity of a change caption, it is determined whether the state change of the acquired data is a new state change; A change captioning system, which adds information on a change caption corresponding to a new state change and an opposite state change to the new state change, in updating the association information.

6. 6. The change captioning system of claim 5, A change captioning system, wherein the state change information in the association information includes a position of the change within the data.

7. 6. The change captioning system of claim 5, A change captioning system in which the calculation device determines that the forward state change has ended when the state change of the acquired data matches the reverse state change in the association information and a forward state change corresponding to the reverse state change is continuing.

8. 1. A method for a system to search for state changes in a data sequence, comprising: The system stores state transition information relating to a state transition and change caption information; The state transition information includes information on state changes of each state transition, The change caption information stores a state transition in the state transition information and a change caption in association with each other; The method further comprises the steps of: Obtaining data in the data sequence in order of generation; determining state transitions and change captions between each captured data and past data of the captured data; updating the change caption information according to the state transition and the change caption; updating the state transition information according to the state transition; Accepts search queries that include natural language, selecting a variation caption corresponding to the natural language from the variation caption information; presenting state change information associated with the selected change caption in the state transition information.

Images