Natural language image searching method and image searching system

Abstract

A natural language image search method is applied to an image search system comprising an image analysis unit, a data management unit, and an instruction input unit. The method includes: an image feature vector encoder in the image analysis unit receiving a detection image and generating an image feature vector; the image analysis unit sending the detection image, the image feature vector, and a timestamp associated with the detection image to the data management unit; a text encoder in the instruction input unit generating a text feature vector according to a query statement and sending it to the data management unit; and the data management unit determining, based on a comparison result, whether the query statement generated in natural language by the instruction input unit matches the detection image.

Description

Technical Field

[0001] This invention provides an image search method and system, particularly a natural language image search method and system. Prior Technology

[0002] Traditional image search technology requires users to first define known search targets and attributes, such as pedestrians or vehicles. Subsequent image searches can only identify images containing pedestrians or vehicles from a sequence of images (video data) based on these targets. Images without pedestrians or vehicles (not known search targets) will not appear in the search results. This means that users must precisely define their search criteria. If the user is unfamiliar with the technology and fails to set appropriate criteria, or even if familiar but subjectively biased, the traditional image search technology cannot identify the relevant images from the vast amount of video data, potentially missing crucial images. Conversely, if users cannot know the image content beforehand, i.e., cannot define known search targets and attributes, the image search results will be limited, potentially missing images that meet the user's needs. Therefore, designing an image search method that does not require precise setting of search criteria and can use natural language for feature search, allowing users to easily and conveniently set image search conditions in a conversational manner, while also being able to search for the correct target images broadly and quickly, is one of the development goals of the relevant surveillance industry. Summary of the Invention

[0003] This invention provides a natural language image search method and image search system to solve the above-mentioned problems.

[0004] This invention discloses a natural language image search method applied to an image search system comprising an image analysis unit, a data management unit, and an instruction input unit. The data management unit connects the image analysis unit and the instruction input unit. The natural language image search method includes an image feature vector encoder in the image analysis unit receiving an image sequence and generating image feature vectors using the detected images in the image sequence; the image analysis unit sending the detected images, image feature vectors, and timestamps and / or related information associated with the detected images to the data management unit; a text encoder in the instruction input unit generating text feature vectors according to the query statement and sending the text feature vectors to the data management unit; and the data management unit determining whether the query statement generated in the instruction input unit using natural language matches the detected images based on the comparison result between the text feature vectors and the image feature vectors.

[0005] The scope of the patent application of this invention further discloses that the natural language image search method includes performing machine learning training on multiple images and their respective corresponding descriptive statements to generate learning results, and the text encoder generates text feature vectors based on the query statement and the learning results.

[0006] The claims of this invention also disclose an image search system having a data management unit, an image analysis unit, and an instruction input unit connected together. The image search system includes the execution of the aforementioned natural language image search method.

[0007] The natural language image search method and system of this invention can perform fast image searches using queries written in natural language. The natural language image search method searches the database of the image search system for the detected image with the closest image feature vector within a given time period, along with its timestamp and / or geographical location, and other related information, as well as previous and subsequent images, based on received text feature vectors and computer-formatted information. This information is then transmitted to a client device such as a display screen. In other words, the natural language image search method and system of this invention can parse text feature vectors and computer-formatted information from queries written in natural language, comparing them with the abstract features (image feature vectors) parsed from the detected image. This eliminates the need for users to use specific formats and standards for image searches, providing a better user experience. It also expands the scope of image search results beyond the limitations of traditional, predetermined queries. Furthermore, by training natural language descriptions using machine learning, this invention can adjust search conditions based on the learning outcomes, further significantly improving the ease of use, accuracy, and speed of image retrieval. Simple Explanation of the Diagram

[0008] Figure 1 is a functional block diagram of the image search system according to an embodiment of the present invention. Figure 2 is a flowchart of the natural language image search method according to an embodiment of the present invention. Implementation

[0009] Please refer to Figure 1, which is a functional block diagram of the image search system 10 according to an embodiment of the present invention. The image search system 10 may have a data management unit 12, an image analysis unit 14, and a command input unit 16 connected to each other. The image analysis unit 14 may be defined as a device end, used to capture image sequences (e.g., video data) on its own, or to receive image sequences (e.g., video data) captured by external devices via wired or wireless means. The command input unit 16 is defined as an operation end, where users can input control commands according to their needs to find the required data from the image sequences of the image analysis unit 14. The data management unit 12 is located between the image analysis unit 14 and the command input unit 16, used to cooperate with other units to execute the natural language image search method of the present invention. It is worth mentioning that the data management unit 12, the image analysis unit 14, and the command input unit 16 may be integrated into the same device within the image search system 10, or they may be independent devices, depending on the design requirements.

[0010] The natural language image search method of this invention means that users are not limited to a specific command format; they can find the required image data from the image sequence of the image analysis unit 14 simply by inputting control commands in natural language at the command input unit 16. The image analysis unit 14 may include an image feature vector encoder 18 and an image analyzer 20. The image feature vector encoder 18 receives the image sequence and analyzes each detected image ID in the image sequence to generate an image feature vector Vif. The image analyzer 20 determines whether the detected image ID meets predetermined conditions; the predetermined conditions refer to the presence of a specific type of object within the detected image ID, such as pedestrians, vehicles, or any moving object, but practical applications are not limited to these. Image analysis (determining whether the detected image ID meets predetermined conditions) can be applied to all detected image IDs in the image sequence or only to a portion of the detected image IDs, depending on design requirements.

[0011] The image analyzer 20 is selectively configured to determine if a detected image ID meets predetermined conditions. When the detected image ID meets these conditions, it sends the detected image ID to the image feature vector encoder 18 for encoding to obtain the image feature vector Vif. This aims to effectively save computational resources. After the detected image ID sent to the image feature vector encoder 18 is successfully encoded to obtain the image feature vector Vif, it can be searched and found using the query statement Qs provided by the instruction input unit 16. Detected image IDs that have not been encoded can be discarded or retained, but because they do not possess an image feature vector Vif in subsequent processes, they will not be searched and found by the query statement Qs.

[0012] The instruction input unit 16 may include a text encoder 22, a time segment decoder 24, and an input interface 26. Users can input a query statement Qs in natural language using the input interface 26. The text encoder 22 then generates a text feature vector Vt based on the query statement Qs. In this invention, the query statement Qs is a sentence used to describe the detected image ID. The text encoder 22 can convert specific words within the query statement Qs, such as the type, color, or behavior of an object, or a description of a time range, into computer format information Cf that the data management unit 12 can parse. The processing unit 28 may be part of the data management unit 12, the image analysis unit 14, and / or the instruction input unit 16, or may be independent of the data management unit 12, the image analysis unit 14, and the instruction input unit 16. The data management unit 12 may include a memory 30. The memory 30 stores data from the image analysis unit 14 and the instruction input unit 16. The processing unit 28 executes the natural language image search method of the present invention based on the data in the data management unit 12, the image analysis unit 14, and the instruction input unit 16, for continuously performing data storage and encoding / decoding operations during image analysis. In another embodiment, the instruction input unit 16 may further include other information encoders (not shown), such as, but not limited to, geographic information encoders. Any information that can be used for image content analysis can be a type of other information encoder of the present invention and can be applied by the instruction input unit 16 of the present invention.

[0013] It is worth mentioning that the comparison method between the image feature vector Vif and the text feature vector Vt can be the K-Nearest Neighbor (KNN) algorithm or other algorithms with similar functions. The application of this algorithm is not the main technical content of this invention, and therefore will not be described in detail here.

[0014] For example, if the query Qs is "a pedestrian in red appearing every Monday morning between January 1, 2019 and February 5, 2019", the text encoder 22 will convert the pedestrian in red into the corresponding text feature vector Vt. The computer format message Cf of the start time is rewritten by the time period decoder 24 into the computer-readable format "20190101000000", and the computer format message Cf of the end time is also rewritten into the computer-readable format "201902050000". These numbers represent the year, month, date, hour, minute, and second in sequence. The computer format message Cf of the schedule is rewritten as "0 0 6-12 * * 1"; these numbers represent the seconds, minutes, hours, date, month, and week in sequence, meaning every Monday morning from 6:00 to 12:00, regardless of month or date. The query statement Qs can be generated using the syntax parsing described above. The variations depend on the design requirements and are not limited to the above examples.

[0015] In other possible variations, the text feature vector Vt and the corresponding image feature vector Vif can be obtained using neural network training. The data management unit 12 can further train multiple images (not labeled in the attached figures) within the image sequence with their corresponding descriptive statements using machine learning, and then establish a relevant training model based on the learning results. In this way, once the training model reaches a predetermined level of completion, the text encoder 22 of the instruction input unit 16 can generate the text feature vector Vt according to the query statement Qs and the training model based on the learning results.

[0016] Please refer to Figure 2, which is a flowchart of the natural language image search method according to an embodiment of the present invention. The natural language image search method described in Figure 2 is applicable to the image search system 10 shown in Figure 1. First, step S100 is selectively executed, whereby the image analyzer 20 determines whether the detected image ID of the image sequence (e.g., video data) meets predetermined conditions. If the detected image ID does not meet the predetermined conditions, it means that this detected image ID does not contain a specific type of object, and step S102 can be executed, whereby the image analyzer 20 sends this detected image ID to the data management unit 12 to be stored in the storage 30, or discards it directly. If the detected image ID meets the predetermined conditions, it means that the detected image ID has a specific type of object. Steps S104 and S106 can be executed. The image analyzer 20 sends the detected image ID that meets the predetermined conditions to the image feature vector encoder 18 so that the image feature vector encoder 18 generates an image feature vector Vif based on the detected image ID, and sends the detected image ID, the image feature vector Vif, and the timestamp Ts associated with the detected image ID and / or related information (e.g., geographical location) to the data management unit 12.

[0017] Next, steps S108 and S110 are executed. The text encoder 22 of the instruction input unit 16 generates a text feature vector Vt according to the query statement Qs provided by the input interface 26 and sends it to the data management unit 12. The time segment decoder 24 of the instruction input unit 16 parses the query statement Qs to obtain the computer format message Cf and sends it to the data management unit 12. Next, step S112 is executed, where the data management unit 12 compares the text feature vector Vt with the image feature vector Vif. If the text feature vector Vt does not match the image feature vector Vif, it indicates that the detected image ID is not the query object of the query statement Qs, and step S114 can be executed to exclude this detected image ID. If the text feature vector Vt matches the image feature vector Vif, then step S116 is executed. The data management unit 12 determines that the query statement Qs using natural language corresponds to the detected image Id, and outputs the detected image Id and its timestamp Ts and / or related information (i.e., the timestamp Ts and / or related information associated with the detected image Id) to external devices such as display screens for users to view.

[0018] Step S116 can send the detected image ID matching the query statement Qs, its timestamp Ts, and / or related information to the display screen (not marked in the accompanying drawings). As described in the previous embodiment, the user can see the red-clad pedestrian and the specific time and location of his appearance (i.e., timestamp Ts and / or geographical location, etc.) on the display screen. However, generally speaking, specific types of objects will not suddenly appear within the field of view of an image sequence. An image sequence can be a series of continuous video images, which will have previous images earlier than the detected image ID (not marked in the accompanying drawings) and subsequent images later than the detected image ID (not marked in the accompanying drawings). Therefore, the natural language image search method of the present invention can also selectively send the previous images and subsequent images related to the detected image ID to the display screen together when executing step S116, so that the display screen can play a short video covering specific types of objects.

[0019] In a preferred embodiment of the present invention, the data management unit 12 may optionally include a data decoder 32. In step S100, the data management unit 12 can store the metadata of the detection image IDs that meet predetermined conditions in the storage 30; the data management unit 12 can further use the data decoder 32 to parse the query statement Qs to obtain keywords. For example, the natural language query statement Qs is "a pedestrian in red appearing every Monday morning between January 1, 2019 and February 5, 2019". The data decoder 32 can parse out the keywords "red" and "pedestrian", and then find the detection image IDs with the keywords from the metadata in the storage 30 for classification. Detection image IDs classified as having no keywords can be discarded without performing subsequent calculations, while detection image IDs classified as having keywords continue to perform other steps of the natural language image search method. This simplifies the total amount of calculation and effectively improves the calculation efficiency and accuracy.

[0020] In a preferred embodiment of the present invention, after the data management unit 12 obtains the detected image IDs according to the above-described natural language image search method, it can further automatically analyze the image content based on the metadata of these detected image IDs, perform additional analysis on the image content of these detected image IDs in addition to the search conditions set by the query statement Qs, and provide the results of the additional analysis to the user for reference, so that the user can find the image they want more quickly.

[0021] In summary, the natural language image search method and image search system of the present invention can perform fast image searches using queries written in natural language. The natural language image search method searches the database of the image search system for the detected image with the closest image feature vector within a given time period, along with its timestamp and / or geographical location, and other relevant information, as well as previous and subsequent images, based on the received text feature vector and computer format information. This information is then transmitted to a client device such as a display screen. In other words, the natural language image search method and image search system of the present invention can parse text feature vectors and computer format information from queries written in natural language, which are then used to compare the abstract features (image feature vectors) parsed from the detected image. This eliminates the need for users to use specific formats and specifications for image searches, thus providing a better user experience. The natural language image search method and system of this invention also enable image search results to no longer be limited by the scope of traditional pre-defined query statements (e.g., the search items and content options must be pre-set, and users can only select search conditions from those options, which not only limits the freedom of retrieval but also the breadth of search results), thus relatively improving the breadth of image search results. Furthermore, by using machine learning to train natural language description statements, this invention can adjust search conditions based on the learning results, further achieving a significant improvement in the accuracy and speed of image retrieval. The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made in accordance with the claims of the present invention shall be covered by the present invention.

[0022] 10: Image Search System 12: Data Management Unit 14: Image Analysis Unit 16: Instruction Input Unit 18: Image Feature Vector Encoder 20: Image Analyzer 22: Text Encoder 24: Time Period Decoder 26: Input Interface 28: Processing Unit 30: Storage 32: Data Decoder ID: Detection Image Ts: Timestamp and / or related information Qs: Query statement Vif: Image Feature Vector Vt: Text feature vector Cf: Computer Format Message S100, S102, S104, S106, S108, S110, S112, S114, S116: Steps

Claims

1. A natural language image search method, applied to an image search system comprising an image analysis unit, a data management unit, and an instruction input unit, wherein the data management unit is connected to the image analysis unit and the instruction input unit, the natural language image search method comprising: an image feature vector encoder of the image analysis unit receiving an image sequence and generating an image feature vector using a detection image of the image sequence; the image analysis unit sending the detection image, the image feature vector, and a timestamp and / or related information associated with the detection image to the data management unit; a text encoder of the instruction input unit generating a text feature vector according to a query statement provided by an input interface and sending the text feature vector to the data management unit; and the data management unit determining whether the query statement generated by the instruction input unit using natural language matches the detection image based on a comparison result between the text feature vector and the image feature vector.

2. The natural language image search method as described in claim 1, wherein the image sequence includes a previous image earlier than the detected image and a subsequent image later than the detected image, the natural language image search method further includes: the image analysis unit sending the previous image and the subsequent image to the data management unit.

3. The natural language image search method as described in claim 2 further includes: when it is determined that the query statement matches the detected image, the data management unit outputs the detected image, the previous image and the subsequent image associated with the timestamp and / or the related information of the detected image.

4. The natural language image search method as described in claim 1 further includes: an image analyzer of the image analysis unit determining whether the detected image meets a predetermined condition; the image analyzer sending the detected image that meets the predetermined condition to the image feature vector encoder; and the image feature vector encoder generating the image feature vector based on the detected image.

5. The natural language image search method as described in claim 4, wherein the predetermined condition refers to the presence of a specific type of object in the detected image.

6. The natural language image search method as described in claim 1 further includes: a time period decoder of the instruction input unit parses the query statement to obtain a computer-formatted message, and sends the computer-formatted message to the data management unit.

7. The natural language image search method as described in claim 1 further includes: a data decoder of the data management unit parses the query statement to obtain a keyword; and a processing unit of the data management unit uses the keyword to classify the detected images that meet the predetermined conditions.

8. The natural language image search method as described in claim 7, wherein the processing unit stores the metadata of the detected image that meets the predetermined conditions in a storage unit of the data management unit.

9. The natural language image search method as described in claim 7, wherein the processing unit further comprises performing machine learning training on a plurality of images and their respective corresponding descriptive statements to generate a learning result, and the text encoder generates the text feature vector according to the query statement and the learning result.

10. A natural language image search method, applied to an image search system comprising an image analysis unit, a data management unit, and an instruction input unit, wherein the data management unit is connected to the image analysis unit and the instruction input unit, the natural language image search method comprising: an image feature vector encoder of the image analysis unit receiving an image sequence and generating an image feature vector using a detection image of the image sequence; the image analysis unit sending the detection image, the image feature vector, and a timestamp and / or related information associated with the detection image to the data management unit; a text encoder and a time period decoder of the instruction input unit generating a text feature vector and a computer format message respectively according to role information and time information of a query statement provided by an input interface, and sending the text feature vector and the computer format message to the data management unit; and the data management unit determining whether the query statement generated by the instruction input unit using natural language matches the detection image based on a comparison result of the text feature vector and the image feature vector, and transmitting the detection image that matches the computer format message to an external device.

11. An image search system having a data management unit, an image analysis unit, and an instruction input unit connected together, the image search system comprising: the data management unit, the image analysis unit, and the instruction input unit performing a natural language image search method as described in any one of claims 1 to 10.

Images