Change Captioning System

Abstract

To provide a change captioning system that effectively searches for a state change in a data sequence and presents a search result thereof.SOLUTION: A method, which causes a system to search for a state change in a data sequence, comprises the steps of: acquiring data in the data sequence in order of generation; determining state transition and a change caption between each acquired data and past data of the acquired data; updating change caption information according to the state transition and the change caption; updating state transition information according to the state transition; receiving a search query including a natural language; selecting a change caption corresponding to the natural language from change caption information; and presenting state change information associated with selected change caption in the state transition information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a change captioning system.

Background Art

[0002] There are many scenes where the difference between past information and current information is taken, such as detecting abnormalities by visual inspection and checking the progress of a building. If a large number of difference descriptions of images obtained from surveillance cameras and patrol robots can be automatically generated and then processed for later search and reports, it is expected to contribute to improving work efficiency.

[0003] So far, in order to make differential information searchable in natural language, it has been necessary for humans to manually provide descriptions (captions) and register them in a database. However, in recent years, a technical field called "Change Captioning", which "automatically generates descriptions of changes while capturing changes", has emerged, creating the possibility of automating the registration work.

[0004] In addition, Japanese Patent Application Laid-Open No. 2020-134955 discloses that "The inspection report support system 1 has a data processing server 3. The data processing server 3 combines a large number of images obtained by shooting a target, creates a plurality of images representing parts different from the whole of the inspection target, and displays an inspection work screen on the user terminal 7 that displays the requested image from the user terminal 7 and a comment input tool. Comments input by the user on the work screen are saved in association with the displayed images. The data processing server 3 automatically creates a report according to the report form using the saved comments and the images associated with them." (Summary)

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] To make differential information searchable in natural language, the problem lies in how to generate the natural language sentences to be searched. Applying Change Captioning to all pairs of stored data in advance to generate searchable sentences is impractical due to the processing cost. Therefore, one approach is to generate sentences using Change Captioning only for the pair with the closest date and time for each data point, prepare them as search targets, and then list the sentences found from the prepared search targets. However, this method may overlook state changes that should be detected, and may also present unorganized information about state changes.

[0007] Therefore, there is a need for a technology that can effectively search for state changes in data sequences and present the search results. [Means for solving the problem]

[0008] One aspect of the present invention is a change captioning system comprising a computing device and a storage device, wherein the storage device stores state transition information relating to state transitions and change caption information, the state transition information includes information on the state change for each state transition, the change caption information stores the state transitions and change captions in the state transition information in association, the computing device acquires data in a data sequence in the order of generation, determines the state transitions and change captions between each acquired data and past data of the acquired data, updates the change caption information based on the state transitions and change captions, updates the state transition information based on the state transitions, accepts a search query including natural language, selects a change caption corresponding to the natural language from the change caption information, and presents information on the state change associated with the selected change caption in the state transition information. [Effects of the Invention]

[0009] According to one aspect of the present invention, it is possible to effectively search for state changes in a data sequence and present the search results. Further features related to the present invention will become apparent from the description herein and the accompanying drawings. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawing]

[0010] [Figure 1] A schematic diagram illustrates the logical configuration of a change captioning system according to one embodiment of this specification. [Figure 2] This shows an example of a hardware configuration that can be used for a change captioning system. [Figure 3] This shows an example of the structure of caption log information included in the caption log database. [Figure 4] An example of a state transition modeled using a finite automaton is shown. [Figure 5] An example of state-change correspondence information is shown. [Figure 6] An example of a change ID list is shown below. [Figure 7] This shows a flowchart of the summary processing performed on acquired images by the change captioning system. [Figure 8] This example shows a configuration that performs change detection processing to identify each change region from two image inputs. [Figure 9] This example shows a configuration that performs a process to generate change caption text from the detected region. [Figure 10] An example of a change caption template is shown below. [Figure 11] Examples of rules for determining changes based on text representation and change mask conditions are shown below. [Figure 12] Here is an example of caption log information: [Figure 13] An example of a state model is shown. [Figure 14] An example of state-change correspondence information is shown. [Figure 15] An example of a change ID list is shown below. [Figure 16]The updated change ID list shown in Figure 15 is presented below. [Figure 17] This shows a state transition model with state S2 added. [Figure 18] State S2 indicates the state-change correspondence information that has been added. [Figure 19] This displays updated caption log information. [Figure 20] This shows a state transition model with added information for state S3. [Figure 21] This shows the state-change correspondence information to which the information for state S3 has been added. [Figure 22] A flowchart illustrating an example of the output generation unit 110's processing is shown. [Figure 23] This shows an example of information collected regarding the presence or absence, appearance time, and end time of change IDs within a specified period. [Figure 24] An example of a UI for searching for changes in captions is shown. [Figure 25] An example of a UI for displaying search results with changed captions is shown. [Modes for carrying out the invention]

[0011] The embodiments of the present invention will be described in detail below with reference to the drawings. Where necessary for convenience, the description will be divided into multiple sections or embodiments. Unless otherwise specified, these are not unrelated, and one may be a modification, detail, or supplementary explanation of part or all of the other. Furthermore, when referring to the number of elements, etc. (including number, numerical value, quantity, range, etc.), unless otherwise specified or clearly limited to a specific number in principle, the number is not limited to that specific number and may be greater than or less than that number.

[0012] The system in one embodiment of this specification may be a physical computer system (one or more physical computers) or a system built on a cloud infrastructure or other computing resource group (multiple computing resources). The computer system or computing resource group may include one or more interface devices (e.g., including communication devices and input / output devices), one or more storage devices (e.g., including memory (main memory) and auxiliary storage devices), and one or more arithmetic units.

[0013] When a function is realized by the execution of a program containing instruction codes by an arithmetic unit, the defined processing is carried out using memory and / or interface devices as appropriate, so the function may be at least a part of the arithmetic unit. The processing described with the function as the subject may be processing performed by the arithmetic unit or a system having that arithmetic unit. The program may be installed from the program source.

[0014] The program source may be, for example, a program distribution computer or a computer-readable storage medium (e.g., a computer-readable non-transient storage medium). The descriptions of each function are examples, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0015] The following describes a change captioning system according to one embodiment of this specification. The change captioning system has a function to search for previously calculated change captions in natural language. The change captioning system records a series of previously calculated changes as state transitions. The change captioning system executes a search process according to the search query and generates information to be displayed as search results.

[0016] Figure 1 schematically shows an example of the logical configuration of a change captioning system according to one embodiment of this specification. The change captioning system includes an acquired data control unit 104, a change captioning calculation unit 106, a change state management unit 107, an output generation unit 110, and a search information input / output unit 111. The change captioning system stores an acquired data storage database (DB) 105, a caption log DB 108, and a state transition information DB 109.

[0017] Figure 2 shows an example of a hardware configuration available for a variable captioning system. The variable captioning system has a general computer configuration and includes a processor 321, main memory 322, auxiliary memory 323, input device 324, output device 325, and network interface 327. Each part of the variable captioning system is connected to each other so as to be able to communicate via communication means such as a bus 326. Note that the variable captioning system may be implemented in whole or in part by virtual resources such as a cloud server.

[0018] The processor 321 is composed of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), and the like. The processor 321 reads and executes the program stored in the main memory 322, thereby realizing the functions of the change captioning system.

[0019] The main memory 322 is a device for storing programs and data, and includes ROM (Read Only Memory), RAM (Random Access Memory), NVRAM (Non-Volatile RAM), etc.

[0020] Auxiliary memory 323 includes, for example, an SSD (Solid State Drive), NVRAM such as an SD memory card, optical storage devices such as a CD (Compact Disc) or DVD (Digital Versatile Disc), an HDD (Hard Disc Drive), or the storage area of ​​a cloud server. Auxiliary memory 323 includes a non-transient storage medium for storing programs and data. Programs and data stored in auxiliary memory 323 are read into main memory 322 as needed.

[0021] For example, the auxiliary memory 323 can store programs corresponding to the acquired data control unit 104, the change captioning calculation unit 106, the change state management unit 107, the output generation unit 110, and the search information input / output unit 111. In addition, it can store the acquired data storage database (DB) 105, the caption log DB 108, and the state transition information DB 109.

[0022] The input device 324 is an interface that accepts information input, such as a keyboard, mouse, touch panel, card reader, or microphone. Alternatively, the change captioning system may be configured to accept information input from other devices via some means of communication.

[0023] The output device 325 is an interface that outputs various types of information, such as a screen display device like a liquid crystal monitor, LCD (Liquid Crystal Display), or graphics card, a printing device, or an audio output device like a speaker. Alternatively, the change captioning system may be configured to output information to other devices via some means of communication. The network interface 327 is a device for the change captioning system to communicate with other devices. Some of the components shown in Figure 1 may be omitted, and other components may be added.

[0024] A change-captioning system can be comprised of one or more computers. Thus, the change-captioning system may include one or more processors and one or more storage devices. One or more processors operate as a predetermined functional unit by executing programs stored in one or more storage devices.

[0025] Returning to Figure 1, the acquired data location and time information 101, which is input from an external source along with the acquired data, includes information indicating where and when the acquired data 102 was acquired. Examples of location information include coordinate information or GPS information assigned to a field map, or an ID assigned to a surveillance camera, etc.

[0026] One embodiment of this specification does not perform change capting on acquired data 102 by brute-force through all previously accumulated data, but rather targets data accumulated for each scene or device corresponding to acquired data 102 for change capting. Acquired data location and time information 101 is used to select scenes or devices corresponding to acquired data 102. Depending on the design, the acquired data location and time information 101 may be omitted if the order of the acquired data is maintained. The order of acquired data according to time is the data generation order.

[0027] The acquired data 102 is one of the inputs to the change captioning calculation unit 106. This includes various types of data acquired from outside the system by sensors such as cameras and microphones. Change captioning can target various types of data, including still images (including still images extracted from videos), 3D point clouds, and sound data. In the following explanation, the data type will be described as still images. Before calculations are performed by the change captioning calculation unit 106, the acquired data control unit 104 performs a saving process.

[0028] Comparison data 103, like acquired data 102, is data that is input to the change captioning calculation unit 106. Since the change captioning technique finds changes by comparing two identical data, acquired data 102 and comparison data 103 are identical data (still images, sound, etc.).

[0029] As comparison data 103, for example, the most recent data corresponding to the same location (image acquisition location) accumulated in the past is retrieved based on the acquired data location and time information 101. If acceptable from the perspective of change detection, comparison data 103 does not have to be the most recent data. The acquired data control unit 104 is responsible for interpreting location information and retrieving data. The acquired data storage DB 105 is the database from which the data is retrieved.

[0030] The data acquisition control unit 104 retrieves and writes data to and from the data acquisition storage DB 105. One processing case is the retrieval of comparison data 103. Another processing case writes the acquired data 102, along with the acquired data location and time information 101, to the data acquisition storage DB 105 as the latest data.

[0031] Furthermore, when the search function is executed, the acquired data control unit 104 processes access to retrieve data from the output generation unit 110 in order to retrieve the stored data corresponding to the search results. The acquired data control unit 104 assigns a unique data ID to each piece of data when it is saved. This information is stored in the acquired data storage DB 105, and the acquired data 102 and comparison data 103 are also retained as supplementary information.

[0032] The acquired data storage DB 105 is a database that stores acquired data 102 in a format that allows selection from a location perspective, based on the acquired data location and time information 101. The acquired data control unit 104 is responsible for writing and retrieving data.

[0033] The change captioning calculation unit 106 receives the acquired data 102 and comparison data 103 as input, detects the change location and estimates the change content described in natural language, and then transmits the data to the change state management unit 107. Details of the processing of the change captioning calculation unit 106 will be described later.

[0034] The change state management unit 107 receives as input the output from the change captioning calculation unit 106 (including the change detection location and description of the change), the acquired data location and time information 101, and the data ID information associated with each of the acquired data 102 and comparison data 103. It then performs a state change update process at the data acquisition location and saves various information in the caption log DB 108 and the state transition information DB 109. Details of the processing of the change state management unit 107 will be described later.

[0035] The caption log DB 108 is one of the databases that stores data output by the change state management unit 107, and it stores change caption information. The information managed by the caption log DB 108 is location and time information, state transition information, and change caption output. In this way, state transitions and change captions are stored in association with each other.

[0036] Figure 3 shows an example of the configuration of caption log information 181 included in caption log DB 108. In the example configuration shown in Figure 3, caption log DB 108 stores tabular data for each data acquisition location. The caption log information 181 in Figure 3 shows the caption log where the data acquisition location ID is "1".

[0037] The caption log DB108 stores the acquisition location ID, acquisition time, state transition information, data ID, and change caption output for each pair of acquired data and comparison data. The acquisition time indicates the time when the acquired data was acquired. The state transition information indicates the state transition (pair of transition states) between the comparison data and the acquired data. The data ID indicates the ID of the acquired data and the comparison data.

[0038] The state transition information DB109 is one of the databases that stores data output by the change state management unit 107. The data stored includes state transition models (e.g., finite automata) defined for each data acquisition location, state-change correspondence information representing the state changes in the state transition model, and a change ID list that manages those changes. The state transition model may be represented in any format and may include time information. Each state S represented by the state transition model is assigned a unique ID (state ID). This ID corresponds to the description of the state transition information in the caption log DB108.

[0039] A state transition is information that focuses on a changing state and can be represented by a state pair consisting of a start state and an end state. State transitions can also be defined between identical states. A state change is information that focuses on the content of the change in a state transition between different states and can be distinguished in this specification by a change caption, or by a change caption and the location of the change in the image. Information regarding state transitions or state changes can indicate both state transitions and state changes.

[0040] Figure 4 shows an example of state transitions 190 modeled with a finite automaton. The signs within the circles indicate state IDs. Figure 5 shows an example of state-change correspondence information 191 for the state transition model 190. State-change correspondence information 191 shows the change IDs associated with each state ID. The change IDs associated with each state ID are a list of IDs of changes that have not yet been completed between the initial state and that state. Figure 6 shows an example of change ID list 192. Change ID list 192 manages the direction of change, the caption and its embedding vector, and the change location for each change ID. The change location is represented by the relationship between change mask images, which will be described later. Different change IDs are assigned to different pairs of embedding vectors and change locations. In this way, change ID list 192 stores association information between state changes and change captions.

[0041] The state transition information DB109 contains state transition information, including a state transition model 190, its state-change correspondence information 191, and a change ID list 192. The contents and usage examples of the state transition model and change ID list will be described in the explanation of the specific processing of the change state management unit 107.

[0042] The output generation unit 110 receives a search query (a time range to search, natural language expressing the change to search for) from the search information input / output unit 111, searches for the corresponding data, and returns the output back to the search information input / output unit 111. Details of the process will be described later.

[0043] The search information input / output unit 111 has the function of displaying a search user interface (UI) to the user and similarly presenting search results. An example of the search UI will be described later. In the search UI, it is possible to specify the period and natural language for change searches. Furthermore, the search results can include a time-series chart that allows users to check whether there have been any changes or whether they have been completed or not during the specified period.

[0044] The change captioning system according to one embodiment of this specification performs two processes as a general processing flow: (I) processing when an image is acquired from an external source, and (II) processing when a search function is executed. First, the processing flow when an image is acquired from an external source (I) will be explained.

[0045] (I) Processing when an image is acquired from an external source. Figure 7 shows a flowchart of the summary processing of acquired images by the change captioning system. The change captioning system obtains acquired data location and time information 101 and acquired data 102 from an external source (S11).

[0046] The data acquisition control unit 104 receives the data acquisition location and time information 101 and the data acquisition data 102 (S12). The data acquisition control unit 104 saves the data acquisition data to the data acquisition storage DB 105. At this time, it issues a unique data ID and assigns it to the data acquisition data 102 (S13). The data acquisition control unit 104 refers to the data acquisition location and time information 101 and loads the comparison data 103 that is closest in time to the data acquisition data 102 from the data acquisition storage DB 105 (S14).

[0047] The change captioning calculation unit 106 receives the acquired data 102 and the comparison data 103, and outputs data such as change captions to the change state management unit 107 (S15).

[0048] The change state management unit 107 receives the acquired data location and time information 101 along with the output of the change captioning calculation unit 106, and loads the corresponding information from the caption log DB 108 and the state transition information DB 109 (S16). The change state management unit 107 executes the state update process and saves the results to the caption log DB 108 and the state transition information DB 109 (S17).

[0049] The details of the processing S15 by the change captioning calculation unit 106 are described below. As an example of processing, the case in which the acquired data 102 and the comparison data 103 are still images will be described. Note that any processing method other than the change captioning method described here can be applied as long as it can receive two still images as input and generate an output that describes the changes between the data and the content of those changes.

[0050] This example describes how to implement change captioning using a recognition model called CLIP (Contrastive Language-Image Pre-Training). CLIP is a recognition model trained on hundreds of millions of pairs of images and their corresponding captions, and it has an image encoder and a text encoder that can embed features into the same vector space. This explanation assumes that VisionTransformer is used as the image encoder.

[0051] The process of change captioning using CLIP is divided into two parts: (i) change detection processing, which identifies each change region from two image inputs, and (ii) change caption text, which is generated from the detected regions.

[0052] Figure 8 shows a configuration diagram for executing process (i). Input data I A 401, I B 402 is a still image. For example, input data I A 401 corresponds to comparison image 103, and input data I B 402 corresponds to acquired image 102. These are then processed through CLIP's image encoder 411 into vector V A , V B These are converted accordingly. In Figure 8, for ease of illustration, one image encoder 411 is represented by two blocks. Note that two image encoders may be provided. Next, the cosine difference calculation unit 422 calculates the cosine difference as a similarity index between vectors using the following formula.

[0053]

number

[0054] Next, the change region estimation unit 421 calculates the derivative with respect to the calculated cosine difference. The derivative is performed according to the following formula. For ease of illustration, Figure 8 shows one change region estimation unit 421 represented by two blocks. Note that two change region estimation units may be provided.

[0055]

Number

[0056] Here, A n and B n are the vectors extracted before the calculation of Attention (immediately after the first LayerNorm) in the final Attention Layer (AL) 415 of the Vision Transformer within each CLIP411. The meaning of this calculation is to identify the regions in the image that enhance the difference, that is, the changing regions.

[0057] GradientA and GradientB are respectively normalized to the value range of 0 - 1 in the change region estimation unit 421, and interpolation and shaping processing of the vectors for matching the sizes with the input data I A 401 and I B 402 are performed. Then, the change region estimation unit 421 performs binarization using a threshold value. Finally, from GradientA, a change mask image D A 431 with the same position as the change position of the image I A 401 having a value of 1 at the same position and 0 for the others is generated. From GradientB, a similar change mask image D B 432 is generated. This is the process of change detection.

[0058] Next, the process of generating a change caption text from the detected regions in (ii) will be described. In process (ii), after obtaining the text that most appropriately represents each change region obtained in process (i) and the text representing the presence or absence and type of change respectively, they are applied to a predefined change caption template to generate a change caption.

[0059] Figure 9 shows a configuration example for executing process (ii). The image extraction unit 451, based on the change mask image D A 431, extracts the input image I AOnly the changing region within 401 is extracted. This is input to CLIP's image encoder 411, and the output vector V for the extracted region is generated. cropA Furthermore, the image extraction unit 451 obtains the change mask image D. B Based on 432, input image I B Only the changing region within 402 is extracted. This is input to CLIP's image encoder 411, and the output vector V for the extracted region is generated. cropB This is obtained. For ease of illustration, one image cropping unit 451 and one image encoder 411 are each represented by two blocks. Alternatively, two image cropping units and two image encoders may be provided.

[0060] In parallel, text representation data 454, which is a set of representation texts representing the change region, is input to CLIP's text encoder 463, and the CLIP output vector M related to the representation text is input. text This is obtained. One of the text representation data is an object description consisting only of nouns or combinations of modifiers and nouns, such as "a person wearing a black hat," "a brown liquid," or "a blue object," and any number of expressions that you want to appear in the change caption can be freely included.

[0061] Furthermore, background descriptions may be included in order to classify the changes described later (appearance, disappearance, movement, attribute change, no change). For example, text describing areas such as walls, ground, and sky. In the appearance / disappearance change pattern, the output vector V cropA , V cropB Since either of these points refers to the background, clarifying which one was the background allows us to distinguish between a change in appearance and a change in disappearance.

[0062] One text representation T i The output vector of the CLIP text encoder for V text_i In this case, the set of expressive texts St∋T i Output M from CLIP text encoder for (i=1…n) text It is defined by the following equation.

[0063]

number

[0064] M text This is the CLIP output vector V of the extracted image. cropA , V cropB Each of these is input to the image-text similarity calculation unit 471. The image-text similarity calculation unit 471 outputs the output vector V cropA , V cropB The closest expression is M text Select from them.

[0065] Output vector V crop For a specific calculation, first, max(V crop M T text ) by M text V constitutes text_i and V crop Find the maximum dot product value between the two. Then, find the value of V given the maximum dot product value. text_i The original text expression T i This is obtained from the text representation data set St454. This process gives V cropA , V cropB For each, the best text representation obtained is, Text A text B This is how it is written.

[0066] Finally, we will explain how to obtain the final output, the change caption CC481. The change caption CC481 is obtained as the output of the change caption generation unit 475. The input to the change caption generation unit 475 is the best text representation Text A text B And, change mask image D A , D B And five change caption template data are given.

[0067] Figure 10 shows an example of a change caption template. A change caption template is a collection of sentences with blanks corresponding to five basic changes: appearance, disappearance, movement, attribute change, and no change, as shown in Figure 10. The change caption generation unit 475 determines which of these basic changes the sentence corresponds to, and then fills in the blanks with the best possible text representation. A text B Output the result filled with [the specified characters].

[0068] Note that Figure 10 shows a simple example. There may be multiple templates for a single type. For example, verbs such as "appeared" and "disappeared" can be replaced with other expressions, or the change mask image D A , D B This involves adding the location information of the changes obtained from the correspondence to the template text for movement and change. In that case, the final output text may be determined through processing such as outputting the one that is evaluated as the most natural using existing methods for estimating the naturalness of the text.

[0069] Regarding identifying fundamental changes, a simple method is to use the best text representation. A text B , and the change mask image D A , D B This can be determined by differentiating cases based on the information obtained. Figure 11 shows the text representation. A text B , and the change mask image D A , D B The following is an example of a rule for determining change based on the given conditions. To improve the accuracy of this discrimination process, it is a natural extension to use a discrimination model, such as a neural network, to learn and utilize discrimination patterns, rather than relying on simple case distinctions.

[0070] The output sent by the change captioning calculation unit 106 to the change state management unit 107 includes the change caption CC and the change mask image D indicating the change detection position. A , D BAnd these are the data IDs associated with the input acquired data 102 and comparison data 103.

[0071] The explanation so far has been based on the assumption that each data point contains only one change. If a single data point contains multiple changes, then preprocessing should be performed to spatiotemporal alignment of data points 102 and 103 (for images, this may involve image mapping), followed by the change mask image D. A , D B Based on the multiple change locations detected and the image content of those change locations, pairing of the change locations can be performed, and the change captioning calculation unit 106 can be processed for each change location pair. In this case, each output of the change captioning calculation unit 106 will increase by the number of change location pairs.

[0072] The details of processing S17 by the change state management unit 107 are described below. The main processing of the change state management unit 107 is (i) updating the change state and (ii) saving various information to the caption log DB 108 and the state transition information DB 109. For processing (ii), the updated information from processing (i) should be overwritten in the original storage location. Processing (i) will be described below.

[0073] In this process, first, based on the acquired data location and time information 101 received as input, the caption log information 181 for the corresponding location is extracted from the caption log DB 108, and the state transition model 190, state-change correspondence information 191, and change ID list 192 for the corresponding location are extracted from the state transition information DB 109.

[0074] Figure 12 shows an example of extracted caption log information 181, and Figures 13, 14, and 15 show examples of extracted state transition model 190, its state-change correspondence information 191, and its change ID list 192. By interpreting the information shown in Figures 12 to 15, it can be seen that the state defined in state S1, "the state in which a man in black clothes appears," is the latest state at the corresponding location.

[0075] Let's assume that the newly received change caption is "A blue flying object appears." First, the change state management unit 107 refers to the change ID list 192 to determine whether this change is known or unknown. For this, it uses the embedded vector V generated by inputting the text of the change caption into CLIP's text encoder, etc. c That is the case.

[0076] The embedding vector of the input change caption is V c_input In this case, the change state management unit 107 uses each V held in the change ID list 192 c The cosine similarity is calculated for each input. If the cosine similarity is above a predetermined threshold, the input change is determined to be known; otherwise, it is determined to be unknown. This determination utilizes the property that embedding vectors of sentences with similar meanings have a high degree of similarity. By referring to the similarity between embedding vectors rather than directly comparing sentences, it is possible to absorb some degree of variation in sentence expression.

[0077] In this case, each V held in change ID list 192 c and V c_input The similarity is low, and the input change is determined to be unknown. If multiple change IDs with high similarity are found, it is possible to determine a single change ID based solely on the cosine similarity, or to determine a single change ID based on the degree of matching (e.g., distance) of the change locations among multiple change IDs that satisfy a certain level of cosine similarity. This choice can be determined for each monitored object (e.g., a building such as a factory).

[0078] Since this was determined to be an unknown change, the change state management unit 107 first registers the input change as a new change in the change ID list 192. At that time, for changes in the opposite direction to the input change caption, the estimated caption (C*-inverse) and its embedding vector (V) are also registered. c*_i) is registered. The caption for the reverse change is created by semantically reversing the caption for the forward change. For example, appearance and disappearance are swapped, and the order of movement positions and attribute changes is reversed. This information can be used to distinguish whether a change that occurs in the future completes a known change or is an entirely new change.

[0079] Figure 16 shows the updated Change ID List 192 shown in Figure 15. As mentioned above, records with Change IDs "C2-forward" and "C2-inverse" have been added.

[0080] Next, the change state management unit 107 moves on to the new state registration process. A new state S2 is defined at the position connected to state S1, with the transition condition from state S1 being C2-forward. Figure 17 shows the state transition model 190 with the new state S2 added. The transition condition from state S2 to state S1 is C2-inverse. The change state management unit 107 further adds state S2 to the state-change correspondence information 191 and sets the retained change ID list to [C1-forward, C2-forward].

[0081] Figure 18 shows the state-change correspondence information 191 to which state S2 has been added. The retained change ID list shows the change ID "C1-forward" from state S0 to state S1 and the change ID "C2-forward" from state S1 to state S2.

[0082] Finally, update the caption log information 181 (state transition list). Figure 19 shows the updated caption log information 181. A third record has been added, indicating the state transition from state S1 to state S2.

[0083] Next, we will explain the processing when a known change occurs (which includes a change corresponding to the change ID list 192). When a known change occurs, there are two cases: (i) transitioning to a known state, and (ii) transitioning to an unknown state. In the latter case, it is necessary to define a new state, similar to the additional state S2 case described above. The distinction between case (i) and case (ii) is made by checking whether there is a match between the retained change ID list held by each state in the retained change ID list held by the state-change correspondence information 191 for the retained change ID list held by the result of reflecting the known change in the current state.

[0084] For example, suppose that following the addition of state S2 described above, a change corresponding to C2-inverse occurs. When a known inverse change occurs in this way, the corresponding forward change C2-forward is removed from the current state's retained change ID list. The updated retained change ID list then becomes [C1-forward], which matches the existing state S1, so there is no need to add a new state, and the state transition corresponding to C2-inverse is from state S2 to state S1.

[0085] In another case, suppose a change corresponding to C1-inverse occurs in state S2. In this case, although the change itself is known, the state corresponding to the updated retained change ID list [C2-forward] does not exist in the state-change correspondence information 191, so a new state S3 is added along with the retained change ID list [C2-forward]. Figure 20 shows the state transition model 190 with the information of state S3 added, and Figure 21 shows the state-change correspondence information 191 with the information of state S3 added.

[0086] Basically, the aforementioned process is repeated with each update, but over time, incorrect states may accumulate due to detection errors or caption errors. To address this, it is possible to implement a manual reset function by preparing a new state transition model, etc., in a format separate from past state transition models and associated data, and registering the current state as the initial state.

[0087] Furthermore, if a single data point contains multiple changes, the number of change captions output from the change captioning calculation unit 106 will be equal to the number of changes. In this case, the processing of the change state management unit 107 described above is performed for each change, and the final state is defined as the point at which processing for all changes is completed. In the above example, state changes and change captions are registered in accordance with image acquisition. In other examples, the above processing may be performed for each image in an image sequence acquired within a specific period and registered in the database.

[0088] Finally, in (I), we will explain the special settings for when the change type is "movement change". When the object of change is a human or animal, movement changes can be observed frequently, such as by moving frequently within the screen. Inverse movement change is defined as a coordinate change in the opposite direction to forward change, but depending on the object of change, the conditions for inverse change may be difficult to satisfy, and a large number of new states may be defined each time it moves, potentially increasing the amount of data required for state management. As a special setting to prevent this, when the change type is determined to be "movement", it is possible to skip the determination of whether the change type is known or unknown and whether it is a new state or not, and instead assume that a self-transition from the state Sn to which it belongs at that time has occurred. In this case, the state transition model, state-change correspondence information, and change ID list are not updated, and only the change caption log information is updated. Under this setting, it is possible to prevent the proliferation of new states while maintaining the search function for change caption output recorded in the change caption log.

[0089] (II) Processing when executing the search function The following describes the process when using the search function. The process when using the search function is substantially the same as the process of the output generation unit 110. The output generation unit 110 receives a search query as input from the change caption search UI. Furthermore, it collects the relevant data from the acquired data storage DB 105, the caption log DB 108, and the state transition information DB 109, and executes a process to generate output that includes a time-series chart and detailed information for each change. This information is displayed, for example, in the change caption search result display UI.

[0090] Figure 22 shows a flowchart of an example of the processing of the output generation unit 110. The output generation unit 110 receives a search query entered by the user from the search information input / output unit 111 (S31). Next, the output generation unit 110 extracts information from the caption log DB 108 and the state transition information DB 109 that corresponds to the specified period, data type, and specified location of the search query (S32). Specifically, caption log information, state transition model, state-change correspondence information, and change ID list are extracted. If no specified period, data type, or location is specified, the entire period, all data, and all locations will be targeted.

[0091] Next, the output generation unit 110 reproduces the state transitions from start to end in the state transition model for each specified location, using the oldest state in the specified period as the start and the latest state as the end. In doing so, it collects information on the presence or absence, appearance time, and end time of each change ID within the specified period (S33).

[0092] Figure 23 shows an example of information obtained as a result of reproducing state transitions, indicating the presence or absence, appearance time, and end time of each change ID within the specified period. If the oldest state S within the specified period is different from the initial state S0, the change corresponding to the change ID held by state S is recorded as having already occurred from the start of the specified period (already occurred). Conversely, if the latest state S' within the specified period is different from S0, the change corresponding to the change ID held by state S' is recorded as continuing at the end of the specified period (continued). In this way, the output generation unit 110 obtains time-series information such as previously occurred, appearing, ending, and continuing within the specified period for each change ID, including changes that appear and end outside the specified period.

[0093] Next, the output generation unit 110 deletes any change ID information collected in the preprocessing step that does not match the specified keyword of the search query (S34). If the specified keyword does not exist, no processing is performed.

[0094] Next, the output generation unit 110 generates a time-series chart to be displayed at the top of the UI for displaying change caption search results, as shown in Figure 25, from the change ID information remaining from the preprocessing (S35). Furthermore, the output generation unit 110 collects necessary information from the caption log and the acquired data storage DB based on the change ID information remaining from the preprocessing, and generates detailed information to be displayed at the bottom of the UI for displaying search results, as shown in Figure 25 (S36). Finally, the output generation unit 110 outputs the results to the search information input / output unit 111 (S37).

[0095] Figure 24 shows an example of a UI for searching for change captions. In the change caption search UI, the user requests a search by specifying the search period, words or phrases within the text, the type of data to be searched, and the location of the data. For specifying a location, for example, the acquisition locations of each data registered in the database are displayed on a map, and an area is displayed that allows the user to specify a location by clicking.

[0096] Figure 25 shows an example of a UI for displaying change caption search results. The top section shows a time-series chart of the relevant changes included in the target period. The chart is generated in step S35 of Figure 23. The horizontal axis of the chart shows the date and time. Each bar represents the duration of a single change. For example, change number "1" occurred and ended within the target period. Change number "2" occurred within the target period but did not end within the target period. Change "3" occurred before the target period and ended within the target period. Although not explicitly shown in Figure 25, the corresponding change caption may be superimposed above each bar.

[0097] The lower section presents detailed information about each change. This detailed information is generated in step S36 of Figure 23. For example, the detailed information may include the location where the target image was acquired, the time of occurrence and end of the change (event) within the target period (if any), the caption for the change, and images before and after the occurrence and end of the change within the target period (if any). Some of the detailed information may be omitted, and other information may be presented.

[0098] The UI for displaying change caption search results allows users to obtain information about events that occurred at a given location, extending beyond the target time period. It also allows them to learn more about the details of those events.

[0099] As described above, according to one embodiment of this specification, the amount of pre-processing required for searching can be reduced, and information to be output can be generated using lightweight processing that does not use images in response to search queries. Furthermore, state changes that occur outside the specified search period and continue within the specified period can be detected and presented to the user. In addition, even if the specified period is long, information on the start and end of each different state change is presented separately. The user can easily recognize the state change information.

[0100] The present invention is not limited to the embodiments described above, and includes various modifications. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0101] Furthermore, each of the above-mentioned configurations, functions, and processing units may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. Alternatively, each of the above-mentioned configurations and functions may be implemented in software by having the arithmetic unit interpret and execute programs that implement each function. Information such as programs, tables, and files that implement each function can be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.

[0102] Furthermore, the control lines and information lines shown are those deemed necessary for explanatory purposes, and not all control lines and information lines are necessarily shown in the actual product. In practice, it is reasonable to assume that almost all components are interconnected. [Explanation of symbols]

[0103] 101 Acquired data: location and time information 102 Acquired Data 103 Comparative Data 104 Data Acquisition Control Unit 105 Data Acquisition Storage Database 106 Change Captioning Calculation Unit 107 Change Status Management Department 108 Caption Log DB 109 State Transition Information Database 110 Output generation unit 111 Search Information Input / Output Unit

Claims

1. A change captioning system, The computing unit and Includes a storage device, The aforementioned storage device stores state transition information and change caption information related to state transitions. The aforementioned state transition information includes information on the state change for each state transition, The aforementioned change caption information stores the state transition, change caption, and time information in relation to the state transition information. The aforementioned computing device is The data in the data sequence is retrieved in the order it was generated. Determine the state transitions and change captions between each acquired data and its past data. The state transition and the change caption are used to update the change caption information. The state transition information is updated by the state transition, It accepts search queries written in natural language and including a specified time period. From the aforementioned change caption information, select the state transitions for the specified period and the change captions corresponding to the natural language. By referring to the state transition information, if the oldest state during the specified period is different from the initial state, the change held by the oldest state is determined to be a state change that has been continuing since before the start of the specified period. A change captioning system that, in displaying search results, presents information about the state change associated with the selected change caption, and, if the state change associated with the selected change caption has been ongoing since before the start time, presents it in a way that indicates it has already occurred since the start time.

2. The change captioning system according to Claim 1, The calculation device is a change captioning system that presents time information of state changes associated with the selected change caption.

3. A change captioning system according to Claim 1, The aforementioned state transition information includes information relating state changes to change captions. The aforementioned computing device is In updating the state transition information, the association information is referenced, and based on the similarity of the change captions, it is determined whether the state change of the acquired data is a new state change. A change captioning system that, in updating the aforementioned association information, adds information about a change caption corresponding to a new state change, as well as information about a change caption corresponding to the reverse state change of the new state change.

4. The change captioning system according to claim 3, The state change information in the aforementioned association information includes the change location within the data, and is a change captioning system.

5. The change captioning system according to claim 3, A change captioning system in which the calculation device determines that the forward state change has ended if the state change of the acquired data matches the reverse state change in the association information, and the forward state change corresponding to the reverse state change is continuing.

6. A method for a system to search for a state change in a data sequence, The system stores state transition information and change caption information related to state transitions. The aforementioned state transition information includes information on the state change for each state transition, The aforementioned change caption information stores the state transition, change caption, and time information in relation to the state transition information. The above method is performed by the system, The data in the data sequence is retrieved in the order it was generated. Determine the state transitions and change captions between each acquired data and its past data. The state transition and the change caption are used to update the change caption information. The state transition information is updated by the state transition, It accepts search queries written in natural language and including a specified time period. From the aforementioned change caption information, select the state transitions for the specified period and the change captions corresponding to the natural language. By referring to the state transition information, if the oldest state during the specified period is different from the initial state, the change held by the oldest state is determined to be a state change that has been continuing since before the start of the specified period. A method for displaying search results, which presents information about the state change associated with the selected change caption, and, if the state change of the selected change caption has been ongoing since before the start time, presents it in a way that makes it clear that it has already occurred since the start time.

Images