Information retrieval system

Abstract

To solve the problem that it is difficult to retrieve various information, such as moving images, images, sound, general purpose data, and text data because a conventional retrieval system has information processing depending on indexes, meta data and learning.SOLUTION: An information retrieval system is implemented by a computer. The information retrieval system includes a plurality of semiconductor information retrieval accelerators provided in each different information kind to each perform independent retrieval, a plurality of device drivers installed in the information retrieval system to drive each semiconductor information retrieval accelerator, and a retrieval engine for making the respective semiconductor information retrieval accelerators execute retrieval in parallel through the device drivers according to a retrieval condition of each kind of information given from the outside, and individually outputting their outputs or aggregating and outputting individual results.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to information retrieval systems and methods, and in particular to media and multimedia information retrieval, and multimodal retrieval. [Background technology]

[0002] (Overview of current search technology) In a big data society, there is a wide variety of information (information types include images, audio, and general-purpose data, and information forms include media and multimedia). Without the ability to search for information, this information cannot be put to good use. Furthermore, the act of searching for information is the optimal means for satisfying human intellectual desires. Therefore, information retrieval processing is closely related to the daily lives and work of people all over the world, and is the information processing method that is most in demand and most frequently performed among all the various information processing methods. Given the above background, information retrieval is the most important and fundamental information processing that supports modern society.

[0003] However, CPUs, the main processors in current computers, are not configured for information searching, so the process of finding the desired information from among the information stored in memory or storage, as well as information provided from external sources, is inefficient and complex, and the various types of information available in our big data society are not being fully utilized.

[0004] Specific examples are given below.

[0005] (Examples of current search technology A: Stream information, temporary information) In order to detect (search) specific information (things) from among information such as videos, images, and audio, it is necessary to create metadata in advance prior to the search. Typical examples include the "Haar" algorithm for detecting (searching) human faces, the "HOG" algorithm for detecting (searching) human bodies, the "OCR" algorithm for detecting (searching) characters, and the "Siri" algorithm for voice metadata.

[0006] These require the creation of metadata and complex algorithms for each desired search target.

[0007] In recent years, image and voice search using machine learning has become common, but preprocessing called learning is essential.

[0008] (Example of current search technology B.....Storage type information) Generic data and text data need to be pre-indexed before they can be searched.

[0009] Creating these indexes requires a certain amount of processing (waiting) time and automatic or manual index tuning (optimization). Without proper tuning, high-speed searches will not be possible.

[0010] (A common problem with current search technology) The speed of searches themselves can be increased through the creation of metadata, learning, and indexes, but real-time searches remain difficult.

[0011] In addition to metadata and indexes, there are methods to assign search tags and annotations to target items, but the burden of assigning tags using human labor is large and it is difficult to generalize.

[0012] Information retrieval processing is highly specialized, and since there is no comprehensive solution to these problems, information retrieval technology is evolving slowly.

[0013] (An example of current search technology based on the above background) It is expected that the realization of quantum computers will lead to major advances in search technology, but it is expected to take a considerable amount of time before they can be put to practical use, and even if they are realized, it will be difficult to miniaturize and generalize them unless room-temperature superconductivity is achieved.

[0014] Aiming to eliminate data bus bottlenecks, there has been active research aimed at improving information processing efficiency through process in memory technology, which integrates memory and arithmetic units (search calculation elements), and process near memory technology, which places memory and search calculation elements adjacent to each other. However, there are no examples of research that attempt to comprehensively solve the wide-ranging and diverse problems of information search. Furthermore, although many years have passed since research began on ontology, a form of semantic language processing that understands meaning, and research into convenient search methods such as the semantic web (web search that understands meaning) began in the early 2000s, there is still no way to obtain satisfactory information with the least amount of effort and in the shortest amount of time.

[0015] In addition, as explained above, the evolution of search technology has reached saturation, and in recent years, attention has been focused on the artificial intelligence (AI) technologies listed below.

[0016] (An example of current artificial intelligence (AI) technology) Although various research and development efforts have been made on artificial intelligence (AI) since the birth of computers, its performance and range of applications have been limited, and it has not become widespread.

[0017] In recent years, AI technology has accelerated dramatically due to the evolution of GPU technology, which can be called the second information processing processor, and deep learning technology, and has now evolved to the stage where many people around the world use generative AI on a daily basis.

[0018] However, the biggest challenge with current AI systems is the huge hardware required and the enormous burden of software development, making multimodal AI that integrates multiple media a high hurdle.

[0019] A particular problem is that current machine learning AI using CPUs and GPUs requires learning, making it difficult to achieve AI processing that includes the latest information; this is a common technical issue in AI, with natural language being one example. Therefore, at present, we are still far from achieving what could be called ideal standalone multimodal AI. (Relationship between search technology and artificial intelligence (AI) technology) We first demonstrated that without a search function the data cannot be utilized. The same is true in the field of artificial intelligence.

[0020] The deep learning-based AI currently in use requires the collection and selection of large amounts of information (data), and the current reliance on indexes and metadata for data collection and selection not only has a significant impact on development time, but also poses various challenges, such as the burden on developers.

[0021] Additionally, in the medical field, due to security concerns, there is a need for on-premise systems that can resolve data security and privacy issues that prevent information from being sent to the outside, such as through cloud processing.

[0022] Due to power consumption, it is difficult to apply GPU-based AI engines to fields such as IoT and autonomous driving.

[0023] Amdahl's Law, an important indicator of the parallel performance of computers (information processing), suggests that even if some information processing is sped up, the improvement in overall system performance will be limited if other information processing remains the same.

[0024] The inventors of this application believe that information retrieval is the most highly sought-after and frequently used process in information processing, and that search processing in a broad sense accounts for as much as 70% of information processing. They therefore believe that solving the problems of information retrieval processing on a broad scale will contribute to improving the efficiency of information processing as a whole, and will lead to the realization of "high-performance, inexpensive, highly user-friendly, and power-saving computers." They have therefore been researching and developing semiconductor information retrieval accelerators using the process-in-memory (PIM) and process-near-memory (PNM) methods for many years.

[0025] The recent trend in databases is to use large amounts of DRAM memory and in-memory databases that enable high-speed information searches. However, the complexity of index creation and the information processing related to indexes remains, and these systems tend to be highly specialized, expensive, and large-scale. This is because we believe that in order to actively utilize information, we need search processing that is essentially free from indexes and metadata.

[0026] In addition, taking image recognition as an example of a recent trend in AI, since it is difficult to define objects and events when recognizing them, deep learning, a learning method in which large amounts of image data of the target objects or events are collected and the system is taught to memorize them through machine learning, has become common. This is because there is no need to define objects or events, and they can be recognized with relatively high accuracy.

[0027] However, we thought that there was no better recognition method than this, because it is easy to define objects and events, conditions can be set with a small amount of image data, it consumes little power, is easy to use even for amateurs, and has highly accurate recognition technology.

[0028] Furthermore, it is believed that it will have a greater impact on solving the problems facing information processing as a whole than new computing technologies such as quantum computers.

[0029] Patent documents 10 to 15 described below are examples of such patents. The product name of these semiconductor information retrieval accelerators 101 is called Memorism Processors, and representative accelerators include five types of processors with product names of SOP (2D, 3D), DBP, XOP, and TSP. The contents shown in these patents have been proven by various methods.

[0030] Non-Patent Document 1, which will be described later, is an academic paper written by the inventor of this application, which shows that commonly used information searches can now be realized using a processor with integrated memory. In this paper, Process Near Memory (PNM) is referred to as Near Memory Process (NMP), but they are the same thing, and the paper introduces an overview of memristive processor technology as a technology that processes together with memory (Process with Memory: PWM) by integrating both Process Near Memory (PNM) and Process In Memory (PIM) technologies.

[0031] The process in-memory (PIM) method is characterized by the fact that although development costs are high, high performance products can be produced at low cost, while the process near memory (PNM) method is characterized by the fact that although it does not match the performance of PIM, it can produce inexpensive products by keeping development costs down.

[0032] In either case, since processors with integrated large-capacity memory such as DRAM and SSDs can be supplied to the market at low cost, it is proposed that the total cost of search systems can be significantly reduced, and this is shown to be the optimal solution for renewing current computers to be smaller, lighter, more energy-efficient, and more powerful.

[0033] Although this paper describes multimodal search, it does not mention the system configuration, nor does it provide any concrete examples of applications such as real-time processing that includes newly obtained information.

[0034] This invention clarifies the configuration of a multimodal search system, describes the configuration of a semiconductor search accelerator, and provides examples of its application to AI, disclosing its application to advanced multimodal search and AI technology that enables real-time processing (true real-time processing) including newly obtained new information (highly fresh information), which is difficult to achieve and for which expectations are high. [Prior art documents] [Patent documents]

[0035] [Patent Document 1] Patent application 2010-065597 Memory with information narrowing down detection function [Patent Document 2] Patent application 2014-508027 Memory with set operation function [Patent Document 3] Patent application 2016-148348 Memory with information search function [Patent Document 4] Patent application 2018-506366 Data comparison processor [Patent Document 5] Patent Publication 2023-76921 Information processing program [Patent Document 6] Patent Publication 2021-103557 Search method for multimedia content [Patent Document 7] Patent Publication 2011-103082 Multimedia search system [Patent Document 8] JP2002-7418 Multimedia information search device [Patent Document 9] JP2024-059598 Visual search decision for text-to-image replacement [Patent Document 10] Patent 5992073 Memory with information narrowing detection function (SOP related patent 1) [Patent Document 11] Patent No. 6014120 Memory with set operation function and set operation processing method using the same (SOP related patent 2) [Patent Document 12] Patent No. 6516775 Information matching device and method (SOP related patent 3) [Patent Document 13] Patent No. 5981666 Memory with information search function (DBP related patent) [Patent Document 14] Patent No. 6393852 Data comparison calculation processor and calculation method using the same (XOP related patent) [Patent Document 15] Patent No. 7475078 Full-text search processor (TSP related patent) [Non-Patent Document 1] IEICE ICD Research Paper: Information Retrieval Processor Using PWM Architecture Summary of the Invention [Problem to be solved by the invention]

[0036] As explained above, conventional search systems rely on indexes, metadata, and learning for information processing, making it difficult to search for a wide variety of information (data, media) such as videos, images, audio (human voices, music, and other sounds), general-purpose data (table data), and text data.

[0037] Furthermore, since there is no comprehensive solution to these problems, it is even more difficult to perform real-time or high-speed searches of multimedia information that combines various media.Specifically, real-time or high-speed searches of multimodal search conditions (simultaneous searches using multiple search conditions) of multimedia information that combines various media places a heavy burden on the CPU and GPU, resulting in large amounts of power consumption. [Means for solving the problem]

[0038] In order to solve the above problems, the present invention provides the following inventions.

[0039] (1) A computer-implemented information retrieval system, comprising: A plurality of semiconductor information search accelerators each of which performs an independent search for a different type of information, Each semiconductor information accelerator: A semiconductor memory is provided for storing search target information of a specific information type, the search target information stored in the semiconductor memory is searched for in accordance with given search conditions, and the search result is output. A semiconductor information search accelerator; A plurality of device drivers are installed in the information retrieval system and drive each semiconductor information retrieval accelerator; A search engine that is installed in the information search system and executes searches in parallel on each semiconductor information search accelerator through a device driver according to search conditions for each type of information given from the outside, and outputs the results individually or aggregates the individual results and outputs them; An information retrieval system characterized by:

[0040] (2) In the information retrieval system according to (1) above, The search engine provides a user interface that prompts the user to enter input conditions and outputs search results to the user. 1. An information retrieval system comprising:

[0041] (3) In the information retrieval system according to (1) above, The search engine provides the given search conditions to each semiconductor search accelerator according to the type of search target information stored in the semiconductor search accelerator. 1. An information retrieval system comprising:

[0042] (4) In the information retrieval system according to (3) above, The information retrieval system is characterized in that the search engine uses search results output by a semiconductor search accelerator for one information type as search conditions for a semiconductor search accelerator for another information type depending on the information type of the search result.

[0043] (5) In the information retrieval system according to (1) above, The plurality of semiconductor information retrieval accelerators include two types of semiconductor information retrieval accelerators, The first type of semiconductor information retrieval accelerator is a semiconductor information retrieval accelerator capable of retrieving features of two-dimensional array pattern information stored in the accelerator using a predetermined feature retrieval condition; The second type of semiconductor information retrieval accelerator is capable of storing in advance, as record table data, characteristics of the plurality of two-dimensional array pattern information retrieved by the first type of semiconductor information retrieval accelerator for each piece of information; Furthermore, the second type of semiconductor information retrieval accelerator is a semiconductor information retrieval accelerator capable of retrieving the record table data by using features of the two-dimensional array pattern information newly retrieved by the first type of semiconductor information retrieval accelerator as a retrieval condition of the plurality of pre-stored two-dimensional array pattern information. 1. An information retrieval system comprising:

[0044] (6) The information retrieval system described in (1) above, characterized in that the two-dimensional array pattern information is general-purpose database information such as an image, a frame image of a video, audio, stock price information, weather information, map information, and two-dimensionally arranged personal information, sales performance information, and customer information.

[0045] (7) The media information retrieval system according to (1) above, wherein the retrieval condition for the first type of semiconductor information retrieval accelerator is a pattern matching condition for two-dimensional array information.

[0046] (8) The media information retrieval system according to (1) above, wherein the retrieval conditions for the two types of semiconductor information retrieval accelerators are bitmap operation retrieval conditions and bit serial operation retrieval conditions.

[0047] (9) The media information retrieval system according to (1) above, characterized in that both the retrieval conditions of the first type of semiconductor information retrieval accelerator and the retrieval conditions of the second type of semiconductor information retrieval accelerator are automated.

[0048] (10) The media information retrieval system according to (1) above, wherein the record table data is increased by using the two types of semiconductor information retrieval accelerators in parallel.

[0049] (11) In the information retrieval system according to (1) above, The plurality of types of semiconductor information retrieval accelerators store multimedia information including at least text and images provided from an external information source for each type of media; The information search engine searches for information based on search conditions and search instructions for each type of media set by the user, The function to output the information search results and An information retrieval system comprising:

[0050] (12) The information retrieval system according to (11) above, wherein the plurality of types of semiconductor information retrieval accelerators are capable of retrieving commonly used information such as voice and data tables in addition to text and images.

[0051] (13) The information retrieval system according to (11) above, wherein the external information source is a data storage, a database system, information that forms the basis of a database, and information available via wired or wireless connections.

[0052] (14) The information retrieval system according to (11) above, in which the media information retrieval system according to (1) above is used as the external information source.

[0053] (15) In the information retrieval system according to (11) above, The information search engine, A function for the user to pre-register multiple pieces of information necessary for searching for the information he or she desires, and a function for the user to set search conditions and give search instructions. a function of synthesizing the plurality of pieces of pre-registered information and the search conditions based on the search instruction to generate first and second search conditions; A function of searching a search system outside the multimedia information search system using the generated first search condition and importing the result information into the multimedia information search system. A function of narrowing down and searching the retrieved result information using the generated second search condition and outputting the narrowed down search result to a user. An information retrieval system comprising: (16) Non-computer equipment including storage, image equipment, and audio equipment incorporating the information retrieval system described in (1) above. (17) The information retrieval system according to (1) above, wherein the plurality of types of semiconductor information retrieval accelerators are FPGAs or ASICs.

[0054] (18) The multiple types of semiconductor information retrieval accelerators 1st category: 2D array information including images and audio 2. Molecular structure and 3D sequence information including 3D maps 3. Structured data information including table data 4. Structured data information including two sets of table data 5. Character string information contained in the text 2. The information retrieval system according to claim 1, characterized in that it is configured with a semiconductor information retrieval accelerator capable of storing the above five types of information and executing searches on the stored information.

[0055] (19) The information retrieval performed by the information retrieval system described in (1) above is An information retrieval system characterized by functions including data searching, image and voice recognition, user authentication, data matching, feature and anomaly detection, data matching, data sorting, and data classification.

[0056] (20) The multimodal search system described in (1) above, wherein the multiple types of semiconductor information search accelerators are semiconductor information search accelerators that implement both the memory and search function of a process-in-memory (PIM) architecture semiconductor that stores the data to be searched within itself and can search the data stored within itself, in an FPGA or ASIC.

[0057] (21) The multimodal search system described in (1) above, wherein the multiple types of semiconductor information search accelerators are semiconductor information search accelerators in which the search function of a process near memory (PNM) architecture semiconductor capable of searching data stored in a memory adjacent to the data to be searched is implemented in an FPGA or ASIC. Effect of the Invention

[0058] 1. Understand the personality and intent of search users and provide the necessary information in the shortest possible time 2. Achieving standardization of information retrieval technology 3. Realizing advanced information search and artificial intelligence (AI) processing in a small-scale system 4. Realization of products and services using new information search technology 5. Contributing to a wide range of fields in the big data society, such as decision support systems and academic research support systems, through new information search technology 6. Realizing true real-time search systems and AI systems at low cost and with low power consumption 9. Achieve advanced information search and artificial intelligence (AI) processing in a standalone environment 10 AI system with easy-to-define recognition conditions [Brief description of the drawings]

[0059] [Figure 1] FIG. 1 illustrates the principle of SOP pattern matching in one embodiment of the present invention.

[0060] [Diagram 2] FIG. 2 also shows an example of pattern matching for specific object detection using SOP.

[0061] [Diagram 3] FIG. 3 also shows an example of a template for exhaustive pattern matching using SOP.

[0062] [Figure 4] FIG. 4 also shows an example of a synthetic pattern matching template using SOP.

[0063] [Diagram 5] FIG. 5 also shows an example of a voice pattern.

[0064] [Figure 6]FIG. 6 also shows an example of a personal information table data search using DBP.

[0065] [Figure 7] FIG. 7 also shows an example A of image information table data search using DBP.

[0066] [Figure 8] FIG. 8 also shows an example B of image information table data search using DBP.

[0067] [Figure 9] FIG. 9 also shows the information retrieval system-A, a media information retrieval system.

[0068] [Figure 10] FIG. 10 also shows the information retrieval system-A, a media information retrieval system.

[0069] [Figure 11] FIG. 11 also shows an example of searching two sets of personal information table data using XOP.

[0070] [Figure 12] FIG. 12 also shows an example of full-text search of text data using the TSP.

[0071] [Figure 13] Figure 13 is also an information search matrix of information types and search functions.

[0072] [Figure 14] FIG. 14 shows the same information retrieval system B, a multimedia information and multimodal retrieval system.

[0073] [Figure 15] FIG. 15 shows the same information retrieval system B, a multimedia information and multimodal retrieval system.

[0074] [Figure 16] FIG. 16 also shows Information Retrieval System-B, a multimedia information and multimodal retrieval system.

[0075] [Figure 17] FIG. 17 also shows an example of condition setting for an information retrieval system / multimodal retrieval system.

[0076] [Figure 18] FIG. 18 shows the information retrieval system-C, a multimedia information / multimodal information refinement retrieval system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0077] (Basic concept) Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0078] The process of finding desired information is usually referred to as information retrieval.

[0079] Here, "information" is a natural expression when considered from the perspective of information processing, but when considered from the standpoint of computing (hardware), it would be more appropriate to express "information" as "data" or "media."

[0080] Meanwhile, in modern society, "search" means a web search, but in the present invention, it is interpreted in a broad sense as the general process of searching for intended information or data. In other words, in the present invention, "search" means "the act of searching for the general process of searching for intended information or data," and includes not only so-called "search," but also "recognition," "authentication," "collation," "detection," "extraction," "match," "sorting," and "classification."

[0081] In the following explanation, wherever important, we have tried to list different types of searches side by side, explaining the meaning of each search.

[0082] Here, Amdahl's Law, which is an important indicator of computer (information processing) performance, suggests that even if some information processing is sped up, if other information processing remains the same, the improvement in the performance of the entire system will be limited.

[0083] Therefore, the inventors of the present application have been researching and developing a process-in-memory type semiconductor information retrieval accelerator (indicated by reference numeral 101 in Figs. 9 and 10) for many years, believing that solving the problem of information retrieval on a wide scale will contribute to improving the efficiency of information processing as a whole, since information retrieval is the most frequently used process in information processing. This semiconductor information retrieval accelerator is a device configured to hold data in a memory and to perform logical operations on the data in the memory using a given comparison value.

[0084] The product name of these accelerators is Memorism Processors, and the four most representative accelerators are SOP, DBP, XOP, and TSP. SOP has two-dimensional and three-dimensional versions, so if these are included, there are five in total.

[0085] Here, SOP (Set Operating Processor) is a semiconductor information retrieval accelerator 101 that performs pattern matching using set operations on images, audio, stock price information, weather information, map information, and even two-dimensional array information (data) arranged in two dimensions, with the purpose of detecting objects (=search, recognition, matching) using images as an example, and extracting features contained in two-dimensional array information (data) using images as an example.

[0086] DBP (Data Base Processor) is a semiconductor information retrieval accelerator 101 for the purpose of searching and collating table data. This DBP corresponds to the patent document 13 (DBP related patent), and is good at searching for select syntax such as SQL databases. XOP (Cross Operating Processor) is a semiconductor information retrieval accelerator 101 that aims to compare two sets of table data, X and Y. This XOP corresponds to Patent Document 14 (XOP related patent), and is good at searching for group syntax, for example, SQL databases. The TSP (Text Search Processor) is a semiconductor information retrieval accelerator 101 aimed at searching text data, that is, full-text searches. This TSP corresponds to Patent Document 15 (TSP-related patent), and excels at keyword searches of general character string information.

[0087] Each of the four semiconductor information retrieval accelerators above is an independent memory device, and data appropriate for each purpose is stored in its own memory. The above four (five) semiconductor information search accelerators 101 are capable of various searches, but only representative operations necessary for explaining the intent of the present invention will be described.

[0088] For more detailed information, please refer to the cited patent documents.

[0089] Since the purpose of this invention application is to bring about a fundamental evolution in information retrieval processing (including AI processing), we have described the application examples, reference examples, and future prospects of this technology as much as possible.

[0090] In particular, we have endeavored to present as many numerical values ​​as possible for the computational performance based on past demonstration results, so that the progressiveness of this technology can be concretely understood.

[0091] Please use this as a guideline as results may vary depending on search conditions and test environment.

[0092] (SOP) The first type of semiconductor information retrieval accelerator 101, SOP (Set Operating Processor), is a semiconductor information retrieval accelerator 101 that performs pattern matching by set operations among images, voice, stock price information, weather information, map information, and even two-dimensionally arranged two-dimensional array information (data), and is intended to detect objects (=search, recognize, collate) using images as an example, and extract features contained in two-dimensional array information (data) using images as an example. This SOP corresponds to Patent Documents 10 and 11 (SOP-related Patents 1 and 2), and performs pattern matching by set operations among images and voice, which are uncompressed media or data, such as BMP format, and even two-dimensionally arranged stock price information, weather information, map information, personal information, sales performance information, and customer information, and performs pattern matching by set operations among two-dimensional array information (data) such as images and voice, which are uncompressed media or data, such as BMP format, and even two-dimensionally arranged stock price information, weather information, map information, personal information, sales performance information, and customer information, and detect objects (=search, recognize, collate) using images as an example, and extract features contained in two-dimensional array information (data) using images as an example.

[0093] On the other hand, 3D SOP, which targets 3D information, corresponds to Patent Document 12 (SOP-related Patent 3), and is a processor that includes pattern matching (feature extraction) of one-dimensional arrays to N-dimensional arrays.

[0094] A notable feature of this SOP is that by making it a semiconductor, it does not require the creation of complex metadata, any pattern can be set and pattern matched, and pattern matching and feature extraction are possible on the order of microseconds and with extremely low power consumption.

[0095] Although pattern matching has traditionally been the most basic process for object recognition and matching, its range of application has been limited due to issues with the time required for matching processing when processed by a CPU using software programs.

[0096] However, the situation would change dramatically if ultra-high-speed pattern matching processing became possible.

[0097] SOPs can detect a wide variety of information, but there is a high demand for searching, so we will use the example of image searching, which is the key point of this invention, as an explanation. SOPs are also good at detecting (extracting) features, so by combining them with DBP, which will be described later, it is possible to realize processing equivalent to artificial intelligence (AI), such as image recognition and authentication.

[0098] As will be described later, the SOP is a semiconductor information retrieval accelerator 101 that expands the dimensions and enables recognition of 2-dimensional to N-dimensional array information, and can be applied to not only images but also audio and other 2-dimensional to N-dimensional array information.

[0099] The SOP is a semiconductor search accelerator that focuses on the fact that VGA and HD images are an array of data (color and brightness) stored in a large number of pixels, and is configured to detect pixel addresses that pattern-match the provided template search conditions in the images to be searched stored in the SOP and output them to the outside by providing (inputting) a pattern matching template (hereinafter simply referred to as template search conditions) to the SOP, which serves as search conditions consisting of the relative position of pixels at specified sampling points and a set of data (color and grayscale) of those pixels.

[0100] Figure 1 shows the principle of pattern matching in SOP.

[0101] The details of pattern matching when the information to be searched is an image will be shown with reference to the figure. The pattern matching template 20 indicates the sampling points 11 of pixels on the image, and in this example, five points (pixels) indicated as A, B, C, D, and E are specified.

[0102] The sampling point indicated by A is reference point 12.

[0103] The sampling points indicated by B, C, D, and E are relative points of the reference point 12, and any relative position (pixel position) can be set.

[0104] In this example, A is the reference point 12, but the reference point can be any of A, B, C, D, or E.

[0105] These sampling points can specify a point range 13, and the range can be arbitrary.

[0106] In this example, there are five sampling points 11, A, B, C, D, and E, but any number of sampling points 11 can be set.

[0107] The set operation formula 21 is an operation formula that specifies the template search conditions of the above-mentioned pattern matching template 20 in the SOP.

[0108] From left to right, the set operation conditions A, B, C, D, and E are shown.

[0109] A is the reference point 12 shown above. The set operation mark 14 is a mark that means a table for performing Boolean operations including space (two-dimensional space in this example, but expandable to N-dimensional space).

[0110] Below the set operation mark 14 are shown sampling points 11 of A, B, C, D, and E, where A is the pixel address of the reference point 12. B, C, D, and E are assigned a relative point address of 17 with respect to A.

[0111] At sampling points 11 B and E, the point range 13 of the specified sampling point 11 is indicated by relative pixel addresses.

[0112] On the other hand, above the set operation mark 14, point (pixel) data 15, which may be R, G, B data or luminance data in the case of color, is specified. Color information is diverse, including red, green, blue, yellow, purple, reddish, bluish, and whitish.

[0113] The SOP is designed to allow maximum and minimum ranges to be specified independently for the R, G, and B color data, so that any color from this seemingly infinite variety of colors can be set as conditions. Therefore, it becomes possible to set subtle color conditions such as reddish, bluish, whitish, etc.

[0114] Because conditions can be set as described above, the color information on the image can be classified into eight types of color information, such as red, green, blue, red-green composite color, red-blue composite color, green-blue composite color, red-green-blue composite color, or in other words, white and black, and conditions can be set by classifying the color information in the image into eight types, such as red, green, blue, red-green composite color, red-blue composite color, red-green-blue composite color, white, and black. It is capable of processing not only color images but also grayscale images.

[0115] Set operations can be easily understood by thinking of color or brightness data (information) being placed on the specified sampling points 11, and a Boolean operation being performed on both the data (information) and its position.

[0116] The operation condition 19 is an operator that performs a set operation including spaces, and in this example, the "AND" operator and the "=" end operator are shown. As with normal Boolean algebra, operators include "AND", "OR" and also "NOT".

[0117] In the case of the "NOT" operation, it is possible to negate the color or brightness data of the point (pixel) data 15, that is, to specify the complementary color, and to negate the point range 13, that is, to specify a value outside the range of the point range 13.

[0118] The above calculations make it possible to detect various features of an object in an image, such as edges that represent the contour of the object and segments that represent the area of ​​the object.

[0119] In this example, all are "AND" operators, so if the five color or brightness data specified as the five sampling points 11 all match the search conditions, and a pattern in which the relationship of each relative point address 17 matches the search conditions exists in the image to be searched stored in the SOP, the pixel address of the image that is the reference point 12 is output as a pixel address 18 that matches the search (detection) conditions.

[0120] There are cases where multiple reference points 12 are output, and cases where no reference points 12 are output.

[0121] A wide variety of pattern matching is possible by setting the conditions of set operation formula 21, so it is possible to perform a variety of pattern matching searches, including global and local color and brightness feature detection on an image, edge feature detection on an image, and segmentation on an image (a block of color or brightness), as well as combinations of these.

[0122] Since it is possible to set ambiguous conditions for the relative position of the pixel at the sampling point and the data of that pixel (color or grayscale), and it can also absorb displacements in the color and position of the image, it is suitable for ambiguous recognition of specific objects and ambiguous feature extraction from within an image.

[0123] Even if the image to be searched is compressed or enlarged, the correlation of pixel addresses is maintained, so by providing a pattern matching template 20 with a relative address similar to the reference address, it is possible to detect the enlarged or reduced image features.

[0124] By setting point range 13 appropriately and scaling 10 times at 10% increments, you can enlarge or reduce the image by approximately 2.3 times, and by scaling 10 times at 20% increments, you can enlarge or reduce the image by approximately 5.2 times.

[0125] Although this varies depending on the image size to be searched and the number of sampling points 11 and pixel positions of the template search conditions, it has been verified that for general template search conditions, a single pattern matching time can be achieved in a few microseconds to approximately 10 microseconds.

[0126] Even if scaling is performed 10 times, the time is about several tens of microseconds to 100 microseconds.

[0127] Taking advantage of the high speed described above allows for a variety of applications, but two representative search methods are introduced below. In the first search method, the search conditions given are template search conditions for specific object recognition that are prepared in advance.

[0128] FIG. 2 shows an example of pattern matching for specific object detection (recognition) using SOP.

[0129] The left side of Figure 2 shows an example of human face detection, and the right side shows an example of Japan map detection.

[0130] The pattern matching template 20 for specific object recognition in face detection has sampling points 11, point ranges 13, and point data 15, which is color and brightness information, set with attention to the image characteristics of the eyes and mouth.

[0131] The pattern matching template 20 for recognizing specific objects on a map of Japan is conditioned by plotting sampling points 11, point ranges 13, and point data 15, which is color and brightness information, with a focus on the shape of the map of Japan.

[0132] This is an example where land and sea are plotted using two different color information. As described above, by providing a pattern matching template 20 of a relative address similar to a reference address, that is, by scaling, it is possible to detect image features that have been enlarged or reduced.

[0133] As an example, by repeatedly performing detection 80 times using eight types of color information and ten types of size (scale) of the target map image as variables, it becomes possible to search with a high probability for maps of Japan that differ in color, brightness, and size.

[0134] Needless to say, if the case where some of the sampling points 11 do not match can be remedied, it becomes possible to detect an image including partial loss or occlusion. Such point designation is simple and can be completed in a short time since there is no need to be aware of the size of the target object.

[0135] A search is performed for the image to be searched using these template search conditions, and if a pattern match is found, it is determined that the image is the same or a similar image.

[0136] The greatest feature of this search method is that it is extremely fast, taking only a few microseconds to 10 microseconds per pattern matching template20.

[0137] By using multiple and various pattern matching templates 20 according to the purpose, advanced detection (recognition) becomes possible.

[0138] It has also been verified that it is possible to automatically generate optimal templates for specific object recognition.

[0139] We will now introduce the second search method.

[0140] Figure 3 shows an example of a template for exhaustive pattern matching using SOP.

[0141] The multiple template conditions shown in the upper part of Figure 3 are used to detect the characteristics of color changes in an image. The eight types of color conditions shown above are set for the left / right, diagonal, top / bottom sampling points, and the color change points that make up the image objects are detected as features.

[0142] The multiple template conditions shown in the middle section are used to detect the shape characteristics of objects in an image. Eight types of color conditions are set at sampling points that specify the shape of the object using various edges (contours), and the edge features that make up the image object are detected.

[0143] The multiple template conditions shown in the middle section are used to detect the shape characteristics of objects in an image. Eight types of color conditions are set at sampling points that specify various segments (areas) of the object's shape, and the segment features that make up the image object are detected.

[0144] For each template of the color change points, edge (contour) and segments (areas) of the exhaustive pattern matching described above, it is advisable to set exclusive sampling points 11 (not shown) to absorb the large number of matched pixel addresses 18 output (search noise).

[0145] A statistical method is used using a plurality of sample images to verify the feature detection capability, either alone or in combination, of the frequency of occurrence of matched pixel addresses 18 and the position of occurrence on the image, and to adopt the necessary template.

[0146] The sample images required in this case can be realized with a smaller number of sample images compared to sample images generated by deep learning.

[0147] Once a template for exhaustive pattern matching like the one described above is created, it can be commonly used for searching (detecting) any image thereafter, and can be freely added or deleted.

[0148] An example of how to use the above exhaustive pattern matching templates will now be described. It is assumed that images to be used as search samples are stored in the SOP.

[0149] In the first step, several types of color change points described above are applied as template conditions to the search target image stored in the SOP, and the resulting output (matched) pixel address 18 is stored in the system's work memory.

[0150] Similarly, in the second step, several types of edges (contours) are applied as template conditions to the search target image stored in the SOP, and the pixel addresses 18 output (matched) as the result are stored.

[0151] Similarly, in the third step, several types of segments (areas) are applied as template conditions to the search target image stored in the SOP, and the pixel addresses 18 output (matched) as a result are stored.

[0152] It is not necessary to execute all patterns in the first, second and third steps. If a predetermined pixel address 18 is detected, the search for that step is terminated and the process moves to the next step.

[0153] At least one pixel address is selected from the multiple addresses obtained in each step and stored in the system's work memory, and at least three selected pixel addresses are combined to create a single pattern matching template 20.

[0154] At this time, the exclusive sampling points 11 that were previously set to absorb search noise are excluded.

[0155] In other words, the composite pattern matching template 22, which is a relative combination of a number of local features, becomes a template that has the overall features of the image, thereby improving the accuracy of image search (detection).

[0156] Reference point 12 can be any address.

[0157] By creating and using not just one but a plurality of these composite pattern matching templates 22, reliable image retrieval becomes possible.

[0158] The following are some more effective ways to use it.

[0159] The image is divided into several areas (hereinafter referred to as divided areas), and one pixel address is selected for each divided area from the multiple matching pixel addresses 18 output in the first, second, and third steps, and this pixel address is set as the sampling point for that divided area.

[0160] Using any one of the three pixel addresses selected above as the reference point 12, a composite pattern matching template 22 is created. This composite pattern matching template 22 is a template that contains local image features of the image stored in the SOP.

[0161] FIG. 4 shows an example of a synthetic pattern matching template 22 according to SOP.

[0162] In the figure, an image is divided into nine divided areas, and a composite pattern matching template 22 is automatically created using the three types of features detected in steps 1, 2, and 3.

[0163] By using this composite pattern matching template 22 as a search criterion, it becomes possible to detect the same or similar images, and even images in which the target image is partially hidden or cropped, based on the principle that "the same image contains the same image characteristic pattern."

[0164] If the image is the same size as the sample image, when the composite pattern matching templates 22 of the nine divided areas are sequentially applied, the matching pixel addresses 18 corresponding to the reference points of each divided area are sequentially output. If the image is cropped or occluded, some of the divided areas will not have matching pixel addresses 18 output.

[0165] If the sample image and the image to be searched are enlarged or reduced, the enlarged or reduced image can be detected by providing a composite pattern matching template 22 in which the relative address is scale-converted.

[0166] When rotation is included in each image, coordinates are transformed by affine transformation, and a search (detection) can be performed using a composite pattern matching template 22 that combines rotation and scaling.

[0167] This is possible because the color and brightness data are range configurable, as are the pixel positions, and the small variations in pixel color, brightness and position can be accommodated.

[0168] Furthermore, since it is possible to set characteristic data that should not be present (undesired) in the search conditions, it is possible to set search conditions that reflect the user's intentions.

[0169] When searching for images that are only vaguely similar in appearance, by allowing freedom in the number of pixel addresses to be detected and their relative positions, it becomes possible to define the ambiguity of the search (recognition), in other words, the degree of similarity, by allowing freedom in the way the degrees of similarity are given.

[0170] In much of machine learning, such as deep learning, the relationship between input and output cannot be defined as intended by the developer.

[0171] This leads to the so-called black box problem.

[0172] In contrast, this method allows the developer to define the relationship between input and output as intended, meaning that the calculation conditions can be fine-tuned based on the results.

[0173] The following shows an estimate of the time required for automatically creating the composite pattern matching template 22. Since the time it takes for the SOP to search (detect) is at most about 10 microseconds per search, even if the composite pattern matching template 22 is tried 100 times each for steps 1, 2, and 3, the time it takes for the SOP to search (detect) is about 3 milliseconds.

[0174] If the software for creating the composite pattern matching template 22 is optimized, it is expected that the search process can be completed in about 10 milliseconds in total, including the CPU processing time.

[0175] Creating nine synthetic pattern matching templates 22 takes about 90 milliseconds.

[0176] By defining the image search rules described above, a pattern matching template 20 that can instantly search for a target image when a sample image is given is automatically generated, and the method is characterized in that it can be used to search for the target image immediately after generation.

[0177] Such automatic generation of the pattern matching template 20 is possible only because the pattern matching time is extremely fast.

[0178] Next, we will show an estimate of the time it takes to search for the same image or similar images, including scaling. When performing 10 scaling operations using nine types of locally synthesized pattern matching templates, the pattern matching time per operation is approximately 30 microseconds since three types of pattern matching templates are synthesized, so the total time is approximately 3 milliseconds.

[0179] The above processing places almost no burden on the CPU, so it is expected that 300 images can be searched and processed per second.

[0180] The software automatically creates appropriate composite pattern matching template search conditions and determines search results. If it can properly process the sending and receiving of image data, it will be possible to search and match up to 300 images per second.

[0181] Therefore, it is possible to detect the desired information from a single video frame in real time, and it can also be used to automatically annotate each video frame. The automatic annotation will be described later.

[0182] In just one minute, you can search through thousands of images to find the best one. Needless to say, if N SOPs are used, the search time is reduced to almost 1 / N, enabling faster and more advanced searches (detections).

[0183] The above examples are just a few.

[0184] It is also effective to combine the first and second methods to perform a search. By preparing a search algorithm suitable for the object to be found, it becomes possible to search for a variety of intended images.

[0185] Instead of processing by software, more efficient processing can be achieved by creating the template pattern to be given to the SOP by using an FPGA, or by processing the match pixel address 18 output from the SOP by using an FPGA.

[0186] Another major feature of this method is that it can significantly compress the data volume of image data.

[0187] Current image search systems and similar image recognition systems store the image to be searched for in the user's hand (on the edge side, such as a PC or smartphone), and send the image via a communication line to a web image search site. The web image search site then searches for images that match or are similar to the image sent from among the images on the web image search site, and sends the results back to the user.

[0188] By using this method, features are extracted from the target image in hand using SOP, and the extracted data (compressed data) is sent, making it possible to significantly reduce the amount of data transmitted and the time it takes to transmit.

[0189] Details of the web image search site system will be described later.

[0190] FIG. 5 shows an example of a voice pattern.

[0191] The X-axis of the diagram is time, and the Y-axis represents the loudness or timbre of a sound, or in the case of a human voice, phonemes, etc., and the diagram shows a variety of examples of sounds.

[0192] Since all of these sounds can be captured as patterns just like images, SOP can search for the sounds using the same search methods described above.

[0193] The time axis of a single audio signal can be any length, such as a signal divided into periods of 1 millisecond, 1 second, 1 minute, or 1 hour, a single phrase of conversation, or a single piece of music.

[0194] Therefore, SOP can be applied to voice recognition and matching.

[0195] Details of the voice recognition and matching system will be described later.

[0196] Without going into details, using a similar concept, it is possible to perform pattern matching using set operations on stock price information, weather information, map information, and two-dimensional array information (data) other than images and audio, and search (detect) various features contained in the two-dimensional array information (data).

[0197] SOP enables ultra-fast information retrieval, making it ideal for information (data, media) mining (repeated retrieval).

[0198] As explained above, the SOP is a semiconductor information search accelerator 101 that can commonly realize pattern matching of a wide variety of two-dimensional array information (data, media), but it can be freely configured by removing some functions or adding more advanced functions depending on the target information and the purpose of the search.

[0199] An SOP is usually a semiconductor configuration targeted at searching for a single image or a single sound, but it is also possible to store multiple images in the SOP in the case of small images such as thumbnail images, or multiple sounds in the case of short audio.

[0200] It is also possible to realize a semiconductor that uses multiple SOPs, has a buffer memory inside the SOP, stores a large number of images (including video frame images), audio, and other two-dimensional array information (data, media) in the buffer memory, and sequentially searches and processes the stored two-dimensional array information (data, media).

[0201] The present invention is described on the premise that the semiconductor information retrieval accelerator 101 has the above configuration.

[0202] (DBP) The second semiconductor information search accelerator, DBP (Data Base Processor), is a semiconductor information search accelerator 101 aimed at searching and collating table data. This DBP corresponds to the patent document Patent Document 13 (DBP related patent), and is good at searching for select syntax such as SQL databases.

[0203] FIG. 6 shows an example of a personal information table data search using DBP.

[0204] The figure shows table data of a personal database, such as age, height, weight, and blood type, for records 1 to N, as well as search conditions. The DBP loads such table data into its internal memory and outputs records that match the search criteria to the outside of the DBP. In this example, record 6 is detected and output to the outside.

[0205] DBP converts table data into a matrix and defines rows as DBP addresses and columns as DBP records. As an example, if the maximum "age" shown in the figure is 128 years old, then the "age data" is stored in 7 bits and 7 bits can be accessed with 7 DBP addresses, enabling a variety of searches through bitmap operations on rows (DBP addresses) using massively parallel operations in the row direction as well as bit serial operations. Typical search contents are data match (=), similarity (≒), greater than>, less than<, <range>, and bitmap search.

[0206] What is particularly noteworthy is that by implementing semiconductor technology, it is no longer necessary to create and tune indexes such as B-Tree or Binary Search, and table data can be searched at the same speed as or faster than the index method.

[0207] Even with the FPGA method, DBP can search and process several gigabytes of information, and if it is implemented as an ASIC, it can also search several terabytes of information.

[0208] The present invention is described on the premise that the semiconductor accelerator has the above configuration.

[0209] FIG. 7 shows an example A of image information table data search using DBP. For convenience of explanation, this method will be referred to as DBP search method 1.

[0210] The records shown in the figure correspond to image numbers, and in this example, N images and X types of templates are registered as a table database.

[0211] The registered data includes pattern matching templates 20, either a pattern matching template 20 in which the SOP described above searches for a specific object, a pattern matching template 20 using an exhaustive pattern matching template, or a composite pattern matching template 22 which is a composite of exhaustive pattern matching templates, which are registered as bitmap data (0 / 1 data) from T-1 to TX in the figure.

[0212] In other words, the pattern matching template 1, which is the search condition of the SOP, is linked to X and assigned as a feature ID to the DBP addresses T-1 to TX.

[0213] Furthermore, in the accompanying information 1 to X, data such as pixel addresses pattern-matched with each template and image division area numbers are registered. Examples of images found using the above search criteria are shown on the right side of the figure.

[0214] Image A is an image that is searched for among images in records 1 to N using search criteria T-3.

[0215] In other words, the image feature IDs T-1 to TX become the image recognition IDs.

[0216] Image B is an image that is searched for using the two search criteria T-4 and T-6. Image M is an image detected under the three search conditions of T-1, T-7 and accompanying information 1.

[0217] Hereinafter, data that combines the type of template that identifies a specific image with its associated information will be described as feature data that indicates the unique features of the image.

[0218] When a user gives a sample image to a system consisting of SOP and DBP, the sample image is immediately imported into the SOP and stored.

[0219] When one or more pattern matching templates 20 described above are given to the SOP as predetermined search conditions, the SOP immediately searches for features within the images stored therein and outputs pixel addresses 18 that match for each pattern matching template 20.

[0220] By using the above SOP output as the search conditions for DBP, DBP immediately performs table data calculations using the following content.

[0221] The feature data (feature ID) assigned from feature 1 to X has a pattern match or not, and in the case of image A shown above, the DBP performs calculations based on the bitmap calculation conditions of the DBP address assigned to T-3, and in the case of image B, the DBP performs calculations based on the two DBP addresses T-4 and T-6, to detect (search) images A and B. In the case of image M including accompanying information 1 to X indicating values ​​such as pixel addresses and divided area numbers, DBP performs calculations under bit serial calculation conditions in addition to the bitmap calculations described above to detect image M.

[0222] In this example, record 9 is found (searched).

[0223] The records found (searched) are nothing but the images provided by the user as samples, i.e., images that match the image intended by the user. It goes without saying that multiple records may be found, or no records may be found at all.

[0224] Furthermore, as explained above, the templates do not need to be matched with all templates and associated information from 1 to X as shown in the figure, but rather with templates related to SOP searches. Therefore, a major feature of this method is that it is efficient even when there are a large number of template patterns.

[0225] Therefore, this search process is extremely fast, as will be described later. The allocation of feature data to table data is not limited to the above configuration, and any other allocation is possible.

[0226] The DBP search method 1 explained above is the most basic table database configuration for realizing high-speed image searches, but with this method, since there is no limit to the type of template, the number of DBP addresses can become enormous.

[0227] FIG. 8 shows an example B of image information table data search using DBP. For convenience of explanation, this method will be referred to as DBP search method 2.

[0228] The figure shows, from the left, data and associated information such as the reference template, relative 1 template, vector 1 direction, and vector 1 length. This data is registered as seven types of data: a reference sampling point based on one of the three types of templates for the first step, second step, or third step shown above, the type of template for the two relative sampling points based on this, and the directions and lengths of the two vectors consisting of the reference sampling point and the two relative sampling points. Assuming that each piece of data is 16 bits (65,536 types), there are 7 types for a total of 112 DBP addresses.

[0229] If nine image areas are set, the total number of DBP addresses will be 1,008.

[0230] It is possible to increase or decrease each piece of data as necessary.

[0231] In this method, although it is necessary to calculate all DBP addresses, the table data can be compressed.

[0232] By treating the 16-bit data as four BCD data, and determining whether or not each BCD data is included in the search target, as well as by performing a range search, it becomes possible to detect similar images as intended.

[0233] In the case of DBP search method 1 described above, if a search for one piece of feature data with or without a feature is performed in the entire image in 10 nanoseconds, the search will be completed in 1 microsecond if the feature data at DBP address 100 is searched using bitmap and bit-serial calculation conditions.

[0234] If there are nine image division areas, this takes approximately 10 microseconds.

[0235] On the other hand, in the case of DBP search method 2, when there are nine image division areas and bit serial calculation of DBP address 1,000, the search is completed in 10 milliseconds.

[0236] Even with the inclusion of additional information, the calculation time is less than 20 milliseconds.

[0237] This makes it possible to automate search conditions for feature data and to apply the same to video.

[0238] If scaling is required, it is performed in the manner described above.

[0239] The two methods shown above are merely typical examples.

[0240] There are various application methods other than those explained above, so construct the most suitable table data configuration depending on the purpose.

[0241] DRAM-based DBPs can be put to practical use in modular form, similar to memory for PCs and servers.

[0242] The shape of the above DBP module is the same size as a general memory module, and the memory capacity can be the same as that of a general memory.

[0243] A case will now be described in which the image search database described above is constructed using a DBP module of 16 gigabytes.

[0244] The images to be searched are themselves stored in general memory or storage. Assuming there are 1,024 DBP addresses, the records in a 16 gigabyte DBP would be 128 megabytes, or approximately 102.8 million.

[0245] This means that a single DBP module can build a search database of over 100 million images.

[0246] It is possible to search over 100 million images in an extremely compact manner in just a few tens of milliseconds. Therefore, it is possible to build such a large-scale image search system using an ordinary PC.

[0247] The feature data to be registered in this database is determined according to the SOP.

[0248] If one SOP were to extract image feature data for 10 images per second, the number of images that could be registered per day would be 864,000, so it would be a good idea to use SOPs in parallel to register images into the database as needed.

[0249] By using a large number of these DBP modules, a very large image database system can be created.

[0250] The main features of this system are its small size, low power consumption, and constant search time even when the number of modules increases.

[0251] The feature of this database is that when a new image is given, the image features of each image are extracted using SOP, and those features are simply recorded in the DBP for each image, making it possible to perform real-time searches using a variety of search criteria at ultra-high speeds without the need to update indexes or perform other processes.

[0252] Even if a feature is simply present or absent on the image, it is also possible to make a database of the matched pixel addresses 18, i.e., the positions of the image features, and to check whether the positions of the features match.

[0253] If a given image has labels or tags that represent concepts such as an individual's face or a car model, the information in these labels or tags can be used for object recognition, authentication, and matching.

[0254] Once an image feature search database has been constructed using labeled and tagged images, features can be extracted from each video frame using SOP, and the extracted image features can be provided as search criteria to this image feature search database. This makes it possible to use the label and tag information associated with images detected from this image database in an equivalent manner to an AI recognition system.

[0255] An example of a video and audio database is shown below.

[0256] For a video at 30 frames per second, a two-hour video would have 21,600 frames, and if all of these frames were assigned as records, the database would have 216,000 records.

[0257] In the case of audio, if a database is created with 10 milliseconds as one record, one hour of audio would result in 360,000 records.

[0258] The above databases may be registered in their entirety in DBP, or only those parts having particularly important characteristics may be registered in the database.

[0259] For reference, if the DBP address of a single 16 gigabyte DBP module mentioned earlier is set to 1,024 as before, all frames of 1,185 hours of video can be stored in a database, and it is also possible to database only the necessary frames.

[0260] In the case of audio, approximately 355 hours of audio can be stored in a database, and it is also possible to store only the necessary audio portions in the database.

[0261] You are free to use video images alone, audio alone, or both.

[0262] Users are free to use this database however they like, but with the appropriate annotations added, the database can be used in advanced applications.

[0263] By comparing each frame of a newly created video with the above database, annotations linked to the video frames in the search results can be automatically added to the new video.

[0264] In particular, by detecting specific people or objects that appear in videos, the effort required for manual annotation can be significantly reduced.

[0265] It can be used as a reference for searching for all kinds of events and objects in systems such as human authentication systems for crime prevention, personal authentication, and anomaly recognition, surveillance camera systems, security systems, and anomaly detection systems for factory equipment.

[0266] This is possible because database systems are small and easy to build, and because database searches are fast.

[0267] This system is also ideal for searching surveillance camera footage for analysis after an incident has occurred, and for searching for illegal copies.

[0268] With text annotations for each frame of a video, viewers can watch only the parts of the video they want to watch, offering a new way to enjoy video. Additionally, proper annotation of film and TV archives can enhance and encourage the evolution of how archives are used.

[0269] This system, which combines SOP and DBP, is also characterized by its ability to significantly compress the volume of two-dimensional array information such as image data.

[0270] As an example, if this system is used as a web image search system, it becomes unnecessary to exchange raw image data as a search condition; it is sufficient to exchange image feature data (compressed image data).

[0271] The image communication time is significantly reduced, significantly reducing the burden on line traffic.

[0272] The above are application examples of a search system for images, videos, sounds, etc. that can be realized by combining SOP and DBP. As explained earlier, SOP is a semiconductor information search accelerator 101 that can perform high-speed pattern matching using set operations on stock price information, weather information, map information, and two-dimensionally arranged two-dimensional array information (data, media, hereafter simply referred to as two-dimensional array information).

[0273] The other DBP is a semiconductor information retrieval accelerator 101 that does not require the creation and tuning of indexes such as B-Tree or Binary Search, and can perform table data retrieval at a speed equal to or faster than the index method. Therefore, by combining SOP and DBP, it is possible to build an advanced and high-speed media retrieval system for a wide variety of information (media).

[0274] Example 1 FIG. 9 is a schematic system configuration diagram showing an information retrieval system-A (media information retrieval system), and FIG. 10 is a functional block diagram of the same.

[0275] 9 and 10 are diagrams showing a general-purpose media search system that can be configured using the two semiconductor information search accelerators 101, SOP and DBP, described above. Below, the inventive components of this system will be explained with reference to these diagrams.

[0276] This search system is configured by connecting the above-mentioned two types of semiconductor information search accelerators 101 to a bus that connects an arithmetic unit 108 such as a CPU or FPGA, a RAM, and an input / output interface. Also, a program storage unit and a data storage unit that are configured by a storage medium such as an SSD are connected to the bus.

[0277] The program storage unit stores, to name only those related to the operation of this search system, an operating system (OS), a search control engine 105, and a driver 102 for driving the two types of semiconductor information search accelerators 101. The information search engine 105 controls the two types of semiconductor information search accelerators 101 through the driver (drive software) 102.

[0278] The information search engine 105 also has a user input / output interface 103 for allowing the user to input search conditions and outputting search results, and an external information interface 103 for acquiring information from external information sources and a search database provided in this system.

[0279] The data storage unit also stores a search condition storage unit that stores search conditions input by a user or the like, a search result storage unit that stores search results, and the above-mentioned search DB.

[0280] Next, FIG. 10 is a schematic diagram showing the operation of this search system.

[0281] The first type of semiconductor information retrieval accelerator 101 in FIG. 10 is an SOP, and is a semiconductor information retrieval accelerator 101 capable of retrieving features of two-dimensional array pattern information stored in the semiconductor information retrieval accelerator using predetermined feature retrieval conditions.

[0282] The second type of semiconductor information retrieval accelerator 101 is a DBP, and is capable of storing in advance the characteristics of a plurality of pieces of two-dimensional array pattern information retrieved by the first type of semiconductor information retrieval accelerator 101 as record table data for each piece of information.

[0283] Furthermore, in this example, the second type of semiconductor information retrieval accelerator 101 is a semiconductor information retrieval accelerator 101 that can search the record table data by using the characteristics of the two-dimensional array pattern information newly retrieved by the first type of semiconductor information retrieval accelerator 101 as search conditions for the multiple pre-stored two-dimensional array pattern information.

[0284] As shown in Fig. 10, the information search engine 105 is expanded on the RAM by the CPU and executed to provide an external interface and a user interface as input / output means. Here, the user interface may be a search condition input / output screen displayed on the user's computer screen. The external interface may be a device that searches the Internet for information related to the data to be searched at a predetermined timing and uses the information for the search together with the search information stored in the information DB or inputted into the interface. The interface is also used to retrieve information stored in the information DB.

[0285] When a user inputs search information through a user interface, the information search engine has a function of distributing search conditions to a driving driver according to the type of information, or the driving driver may have such a function.

[0286] As a result, each semiconductor information search accelerator performs an information search independently and in parallel according to the type of information to be searched, and returns the result to the information search engine. The information search engine is configured to output the search results of each accelerator individually or in an integrated manner. In the integration, the two search results may be narrowed down by both search results and output.

[0287] In addition, in this example, the information search engine inputs the characteristics of the two-dimensional array pattern information newly searched for by the first type of semiconductor information search accelerator 101 (SOP) to the semiconductor information search accelerator 101 (DBP) as a second type of search condition.

[0288] Here, the two-dimensional array pattern information stored in the first type of semiconductor information retrieval accelerator may be general-purpose database information such as an image, a single frame image of a video, audio, stock price information, weather information, map information, two-dimensionally arranged personal information, sales performance information, customer information, etc.

[0289] The search condition for the first type of semiconductor information search accelerator may be a pattern matching condition for two-dimensional array information.

[0290] Furthermore, the search conditions for the two types of semiconductor information search accelerators may be bitmap operation search conditions and bit serial operation search conditions.

[0291] It is also possible to automate both the search conditions of the first type of semiconductor information search accelerator and the search conditions of the second type of semiconductor information search accelerator.

[0292] Furthermore, the record table data can be increased by using the two types of semiconductor information search accelerators in parallel.

[0293] According to this method, by using multiple DBPs in parallel, the amount and number of two-dimensional sequence information as well as the number of features can be made unlimited in principle.

[0294] This method does not require the creation of indexes or the use of complex algorithms, and search conditions are generated automatically, so even non-programmers can quickly build a database of two-dimensional array information, such as images and sounds, and even if a large amount of two-dimensional array information is added and table data is added, searches can always be performed in real time at a constant high speed, regardless of the number or amount of two-dimensional array information.

[0295] Table data can be handled simply and can be transferred in the same way as with a USB memory or SD card. Once transferred, table data can be searched immediately, making it possible for even non-information processing specialists to search for records of a wide variety of two-dimensional array information. Needless to say, the link data linked to the record also makes it possible to view actual images and audio.

[0296] (XOP) The third type of semiconductor information search processor, XOP (Cross Operating Processor), is a semiconductor information search accelerator 101 that aims to compare two sets of table data, X and Y. This XOP corresponds to Patent Document 14 (XOP related patent), and is good at searching for group syntax, for example, SQL database. Figure 10 shows an example of searching two sets of personal information table data by XOP.

[0297] The figure shows two sets of table data for personal databases, records 1 to N for men and women, including age, height, weight, and blood type, along with an example of searching (detecting) matching or similar records based on the search criteria. XOP imports (stores) such table data into its internal memory and outputs records that match the search criteria to the outside in pairs of male and female. In this example, a male record 9 and a female record 4 are detected as a pair.

[0298] The system calculates whether each row (address) matches or does not match in a bit-serial manner, making it possible to realize massively parallel comparison (search) of a fixed data width, such as 16-bit or 64-bit data, or combinations of these data, to detect matches, mismatches, or similarities. If either X or Y is known, rapid classification becomes possible. High-speed sorting is possible by setting either X or Y as ascending or descending information.

[0299] A notable feature of this technology is that by using semiconductor technology, searches such as comparison of data, grouping and sorting can be performed significantly faster than with conventional methods without using complex algorithms. Even using the FPGA method, XOP can search and process several gigabytes of information, and if it is converted into an ASIC, it can process several terabytes of information. The present invention is described on the premise that the semiconductor accelerator has the above configuration.

[0300] (TSP) The fourth semiconductor information retrieval accelerator 101, TSP (Text Search Processor), is a semiconductor information retrieval accelerator 101 that is intended for searching text data, that is, full-text searches. This TSP corresponds to Patent Document 15 (TSP-related patent), and excels at keyword searches of general character string information. FIG. 12 shows an example of full-text search of text data using TSP. The figure shows various sentences (in this example, Japanese) in text files 1 through N, along with the keyword "exclusive economic zone," which is the search criterion. The TSP takes in (stores) multiple pieces of such text data in its internal memory, and outputs to the outside the address of the keyword string that matches the search criteria, as well as the text file name.

[0301] In this example, text file 7 is found. TSP enables massively parallel detection (searching) of keyword-specified character strings from character data contained in linear text data.

[0302] A notable feature of this technology is that by being semiconductor-based, it does not require indexes such as morphological analysis or n-gram methods, and yet it is possible to achieve full-text searches that are faster than or equal to the index method.

[0303] Here are some examples of index problems:

[0304] We checked the index creation time using four PCs. The index creation time (wait time) for a full-text search in normal mode on 0.8 GB of text data was approximately 3.5 hours = 12,600 seconds for all four machines. Working backwards, that comes out to an indexing capacity of approximately 64 kilobytes per second. If 1 megabyte of text data is added, it will take approximately 15 seconds, and if 10 megabytes of text data is added, it will take approximately 3 minutes for the added text data to be searchable.

[0305] As an example, the above may affect when checking e-mail for the first time in a while, or when searching immediately after obtaining a large amount of text data.

[0306] In contrast, the search latency for a TSP of 0.8 gigabytes of text data is approximately 1 second.

[0307] The above examples are examples of indexing problems that are familiar to us, but they are problems that occur frequently and continuously in fields that handle big data.

[0308] There is a limit to how much time (waiting time) it takes to create an index, even if you make the system large-scale.

[0309] TSP realizes true real-time search without making the system larger, regardless of whether it is stream-type information or stored-type information.

[0310] One problem with full-text search is that if an index becomes corrupted, users will experience frustrating search times for a significant period of time until the index is repaired. These problems are especially prevalent in small systems such as PCs.

[0311] Full-text search using TSP does not have an index, so this problem does not occur.

[0312] The above is a common problem not only in full-text searches but also in any search process that relies on indexes.

[0313] Furthermore, TSP has many advantages that cannot be realized by indexing methods, such as high search accuracy, intermediate match, suffix match search, wildcard search, and the ability to be used in all languages ​​around the world.

[0314] Even with the FPGA method, the TSP can search and process several gigabytes of information, and if it is implemented as an ASIC, it can process several terabytes of information. The present invention is described on the premise that the semiconductor accelerator has the above configuration.

[0315] The first common feature of the four memristive processors, which are semiconductor information retrieval accelerators 101, is that there is no need for preprocessing related to searches, such as metadata or indexes, so once the information (data) to be searched is prepared, searches can be performed on the spot, making true real-time searches possible. The second common feature is that the accelerators have built-in memory for storing media information and a search engine, and each accelerator performs calculations independently (SIMD calculations), so that consistent, high-speed searches can be achieved regardless of the size of the information data. Moreover, they are small and energy-efficient.

[0316] It is possible to use a method in which all information is always stored in the memristive processor, or a method in which information required is loaded (stored) from outside into the memristive processor each time and searched (batch processing method).

[0317] The third common feature is that since there is no need for preprocessing related to searches, such as metadata and indexes, the restrictions on information retrieval accuracy and functionality that are imposed by metadata and indexes are eliminated, the level of specialization of information retrieval technology is reduced, and standardization of information retrieval technology becomes possible.

[0318] Information searches are difficult to systematize because the target information (data) is diverse, the search results desired when searching for the target information (data) are also diverse, and in the case of software, there are a wide variety of methods.

[0319] Figure 13 is an information search matrix of information types and search functions.

[0320] Figure 13 shows a systematization of information retrieval using a matrix of the types of retrieval data and retrieval functions of the four processors explained so far. The horizontal axis shows the types of information (data) that are commonly used, and the vertical axis shows the commonly used search functions.

[0321] Stream data, which is one of the types of information, refers to raw information (data) provided through wired or wireless broadcasting or communication, while storage + additional data refers to information (data) stored as a database and newly added information (data) to this.

[0322] The blank areas of the matrix represent meaningless or unused searches.

[0323] This matrix indicates that commonly used information retrieval can be realized by any one or combination of the four memrism processors described so far.

[0324] Here, commonly used searches refer to searches for specialized information that is not used in ordinary searches, such as searches for prime numbers or searches for combinatorial problems such as the traveling salesman problem.

[0325] Freeing the everyday information retrieval process from metadata and indexes is equivalent to solving many of the problems in information retrieval. Information retrieval processing is closely related to the daily lives and work of people all over the world, and is the most frequently and most needed of all the various types of information processing, so it can be used in a variety of applications in a variety of fields.

[0326] The most commonly used search in this matrix is ​​searching text databases using TSP, i.e. full-text search (keyword search). It is used widely for web searches, internal document databases, PC email searches, and even searching for lost files. Among the many types of searches, this is the most commonly used and most frequently used process.

[0327] The inventors of the present application have introduced optimal systems using each of the semiconductor information search accelerators 101 in their previous SOP, DBP, and XOP applications, but there have been no applications for specific application systems that combine and integrate multiple semiconductor information search accelerators 101. The present invention is based on a patent application for an accelerator for full-text search of TSP, and the completion of a prototype has made it possible to realize commonly used information searches using four semiconductor information search accelerators 101.

[0328] In the following, we will introduce a new search system that is realized by using the four semiconductor information search accelerators 101, namely, the memrism processor SOP, DBP, XOP, and TSP described above.

[0329] Example 2 14 to 16 show an information retrieval system-B, a multimedia information retrieval system.

[0330] This system is configured with four semiconductor information search accelerators by adding an XOP and a TSP to the information search system A described in Figures 9 and 10. Therefore, the same components as those described in Figures 9 and 10 are given the same reference numerals and their description will be omitted.

[0331] This search system, like information retrieval system A, is an information retrieval system that is entirely controlled by a CPU, FPGA, or other semiconductors, and the information search 105 of this search system is composed of four types of semiconductor information retrieval accelerators 101 (SOP, DBP, XOP, TSP) including the first and second types mentioned above that can store multiple types of multimedia information by media type and search for that information, and driver software 102 that drives these multiple types of semiconductor information retrieval accelerators 101.

[0332] In the memory of the multiple types of semiconductor information retrieval accelerators 101, multimedia information including at least text and images provided from an external information source 104 is stored for each type of media.

[0333] The information search engine 105 has a function of searching for information based on the search conditions and search instructions for each type of media set by the user, and outputting the information search results. Specifically, the information search engine 105 distributes the search conditions by information type, and passes them to the semiconductor information search accelerator that searches for each information type.

[0334] The multiple types of semiconductor information search accelerators 101 (SOP, DBP, XOP, TSP) are capable of searching commonly used information such as voice and data tables in addition to text and images.

[0335] External information sources 104 are data storage, database systems, database source information, and information available over wired or wireless connections.

[0336] The above configuration provides the following effects.

[0337] By making the database of this media information retrieval system 106 a database including label and tag information, a system having the ability to recognize, authenticate, and verify objects, in other words, a multimedia information retrieval system 107 having an intelligent search ability, can be realized.

[0338] By treating a wide variety of information such as images, audio, text, and general-purpose data as database data, it is possible to improve the ability to recognize and match information. In other words, the complex linking process of ontology is replaced by multimedia (known information) that has been given labels and tags.

[0339] The multiple types of semiconductor information retrieval accelerators 101 described above are all based on memristive processors that enable high-speed information retrieval without using metadata or indexes for information retrieval prior to the information retrieval, thereby realizing a multimedia information retrieval system 107 that does not have the search wait time associated with creating metadata, indexes, etc. for information retrieval.

[0340] This multiple types of semiconductor information retrieval accelerator 101 is composed of SOP, DBP, XOP, and TSP, and realizes a multimedia information retrieval system 107 capable of retrieving commonly used information such as text, image, sound, and table data.

[0341] The external information sources 104 of this multimedia information retrieval system 107 collectively refer to information that can be provided to this system, such as information from data storage (including USB memory, SD cards, etc.), information from database systems, information that forms the basis of the database, information obtained from communication networks such as the Internet, and information obtained via radio waves such as broadcasting.

[0342] For example, the following is an example of web information.

[0343] Typical web information is multimedia information (data) that combines text data with images and audio.

[0344] A normal search is a full-text search of text data (keyword search), like Google search.

[0345] Google Image Search can search for images that match or are similar to a sample image.

[0346] As described above, regular text search and image search are independent of each other.

[0347] According to the present invention, a user can obtain (find) desired information (multimedia information) from among these multimedia information (data) groups using a single search condition or a compound search condition.

[0348] For example, if you provide "exclusive economic zone" or "EEZ" as a full-text search criterion for text data and "map of Japan" as a search criterion for image data, it will be possible to find multimedia data that includes "exclusive economic zone" or "EEZ" as text data and "map of Japan" as image data. In other words, it is possible to search for multimedia data that covers news about the intrusion of flying objects or ships into the exclusive economic zone.

[0349] In this case, the "Map of Japan" given can be a web URL.

[0350] This system runs the SOP to mechanically detect images in the multimedia data that are similar to the "Map of Japan" at the specified URL, using the method described above.

[0351] At the same time, this system drives the TSP to mechanically detect multimedia data that contains the string "exclusive economic zone" or "EEZ" from the text data of the multimedia data.

[0352] Using the above SOP and TSP, it becomes possible to provide users with multimedia data (narrowed down information) that matches search criteria for both images and text.

[0353] In addition, by setting the frequency of occurrence of "exclusive economic zone" or "EEZ" as a full-text search condition for text data to, for example, five or more times, users can find the detailed news article they are looking for. The feature of this method is real-time (high-speed) multimedia search.

[0354] (Multimedia search condition setting problem) Generally, the things (including information) you want can be expressed relatively accurately (conditions can be set). However, due to problems with search conditions related to the description of the things, searchers may not be able to obtain the information they want, or may end up with a lot of unnecessary information, wasting the searcher's time and effort.

[0355] Four examples of the above will be explained.

[0356] The first example involving "such a thing" is the problem of a lot of unnecessary information being included if personal information is not taken into account.

[0357] For example, describing your body type in a way that corresponds to "like this" is a prerequisite when searching for the suit, dress, shoes, or gloves you want.

[0358] Personal information is very important for enabling users to accurately specify the information they want to obtain and for filtering out unnecessary information.

[0359] However, since setting it up every time is inefficient, it would be convenient if there was a way to register it in advance.

[0360] The second example, which relates to "such a phrase," is that even when searching text data, which is currently the most important means of information retrieval, it is extremely difficult to search for information by specifying search conditions equivalent to "such a phrase."

[0361] Here's an example of a relatively simple search:

[0362] For example, here is an example of "such as this" when the thing you want is literature related to your research.

[0363] The frequency with which terms such as "search," "index," and "narrow" or similar words appear in a document is an important factor in understanding the intent of the document. For example, if it occurs only a few times in a 10,000-character document, it is often a coincidental mention.

[0364] On the other hand, if it appears five times in a 1,000-character document, you can clearly understand what the document is trying to assert and what it intends to convey.

[0365] In addition to literature, if it were possible to perform frequency searches for news items such as "Japan," "peace," and "war," which frequently appear in pairs with terms such as "economy," "stock prices," and "employment," or tourist information, which frequently appears in pairs with terms such as "quiet," "cool," and "hot springs," or "close," "delicious," and "hot springs," this would be a very convenient search method. If it were possible to perform a proximity search to find pages or ranges where terms frequently appear or where pairs of terms occur within a large-scale manual, the burden on search users would be greatly reduced. As described above, if a full-text search system that allows a variety of search conditions to be combined and set is provided, it will be possible to eliminate unnecessary information and view only the information that users (search users) want to know, thereby reducing the effort and time wasted when searching. In particular, in everyday life, if e-mail searching had the above-mentioned functionality, it would be much more convenient to use e-mail.

[0366] The above example is an example of an advanced method that combines full-text search near searches and frequency searches, which are equivalent to "such" in text data.

[0367] TSP is ideally suited to handle such challenges in real time.

[0368] The third example involving "like this" is when the "like" of an object cannot be expressed in words or numbers.

[0369] Here are some third examples.

[0370] For example, if what you want is a wedding dress, "like this" might mean colors such as white, pink, red, yellow, blue, purple, etc., and shapes such as princess line, slender line, empire line, etc., which are introduced on the web; however, colors and shapes are very delicate and difficult for anyone other than an expert to describe accurately in words. In the above examples, sample image data that represents "such" color and shape is desired.

[0371] For example, if you want to eat ramen to your liking, not only the type of soup but also the type of toppings can be important factors. In the above examples, if there were a way to specify sample image data that represents "such as this," it would be possible to view only the information that one wants to know.

[0372] SOP is ideal for handling such challenges in real time.

[0373] For example, if you want to listen to music that will invigorate or motivate you, or that will soothe or comfort you, words alone are often not enough.

[0374] In the above example, if there were a means to specify sample audio data that represents "such as this," it would be possible to listen to only the music that one wants to hear. SOP also allows for fuzzy search (detection) of voice patterns, making it possible to apply it to voice recognition, etc.

[0375] The fourth example of "such" is extremely diverse, including the price, performance, size, location information such as where, and time information such as what time.

[0376] The above "such" refers to things that can be quantified or have conditions set, and if there is a way to set such conditions, it will be possible to view only the information you want to know. DBP and XOP are ideal for real-time processing of such tasks.

[0377] The above searches may involve various situations, such as when the information required is urgent down to the second, when a minute or so is not a problem, or in some cases when the results are important even if it takes an hour.

[0378] We will now introduce a method for setting conditions that overcomes the four "such" challenges mentioned above and allows users to set conditions for what they want.

[0379] (Example of setting conditions for an information search system) FIG. 17 shows an example of condition setting for an information retrieval system. The figure shows an example of a browser or GUI screen that allows you to pre-register desired information for yourself or someone else in order to efficiently obtain that information, set search criteria when the information is needed, issue a search command, and display the results.

[0380] The screen is designed so that you can register your "personal information" from the left, and by selecting the black circle button you can register your personal information.

[0381] "Personal Information" Text data: Your name, address, medical history, family structure, hobbies, interests Numerical data: Your age, height, weight, body type Image data Favorite color, favorite pattern, favorite scenery, favorite celebrity, favorite singer Audio data: Your favorite music, your favorite voice Note that the above is just one example.

[0382] "Custom Settings" allows you to set the time, location, budget and search time you want to get what you need.

[0383] It displays the "subject" which represents the person who wants the desired information, the "verb" which represents how they want to obtain the information, "such as this" which is an adjective to describe the thing, and finally the "thing" which is the desired information.

[0384] The adjective "such as" for an object and "thing" for information are used in many ways, so they are classified into categories, and detailed information is displayed when the black circle button for selecting a category is pressed.

[0385] As mentioned above, there are cases where "such" cannot be adequately expressed in words, so the system is configured to allow detailed condition settings for text data and the registration of sample image data (media) and audio data (media).

[0386] These sample data may be provided as links such as URLs of external data.

[0387] The search result message is displayed at the bottom of the screen.

[0388] Of course, there may be cases where there are no search results or the search time is insufficient.

[0389] These are configured to be displayed appropriately in search result messages.

[0390] Users can register these items in advance or on the spot, check the search condition buttons, and press the search button; the search results are then displayed in a search result message.

[0391] By performing the above operations, the user can obtain the information most suitable for the user within the search time specified.

[0392] Not all of these search functions are necessarily required.

[0393] The necessary functions can be selected and configured according to the use case.

[0394] While this example is a browser and GUI screen targeted at individual users, it can also be used as a decision-making support system for people who always need to make the best decisions within a limited time, such as company leaders and politicians.

[0395] In the event of a disaster, it can be used as a support system to consolidate various information and quickly disseminate the most appropriate information.

[0396] It can also be used as a support system for obtaining essential information for research and development and information from other companies in a timely manner.

[0397] The above browsers and GUI screens can be used as support systems for people who need to "control time and information" and for whom "information has great value."

[0398] Even if it were possible to set conditions using various adjectives that correspond to "such a thing," it would be meaningless if it would not be possible to perform an effective search, would take a significant amount of time, and would not provide the search results the user desires within the time frame desired.

[0399] It is difficult to comprehensively search not only text data but also a variety of data such as image data and audio data in real time using current information processing techniques alone.

[0400] We will introduce an information retrieval system that makes maximum use of existing search systems and communication lines, and provides users with the desired search results within the time frame they desire for multimedia information using the above-mentioned condition setting methods or equivalent search methods.

[0401] Example 3 FIG. 18 shows the information retrieval system-C, a multimedia information refinement retrieval system.

[0402] The constituent elements of the present invention will be described below with reference to the drawings.

[0403] This search system is based on the multimedia information search system 107 explained above. This search system has the following functions: a function for a user to pre-register a plurality of pieces of information necessary for searching the information he / she intends, and a function for a user to set search conditions and give a search instruction. a function of synthesizing the plurality of pieces of pre-registered information and the search conditions based on the search instruction to generate first and second search conditions; A function of searching a search system outside the multimedia information search system 107 using the generated first search condition and importing the result information into the multimedia information search system 107. The multimedia information retrieval system 107 has a function of performing a refined search of the retrieved result information using the generated secondary retrieval condition, and outputting the refined search results to the user.

[0404] The external search system of the multimedia information search system 107 is a multimedia information search system characterized in that it is a database system capable of information search, a cloud system capable of information search, a web search system, or a search system equivalent to these.

[0405] The primary search conditions for the above multimedia information retrieval system 107 are usually text search conditions, and the resulting search information is used as the secondary search conditions to narrow down the search, using the image search method introduced earlier as an example, thereby eliminating unnecessary information and providing the user with the information they truly need with the minimum of effort and time.

[0406] The secondary search criteria can be narrowing search criteria based on any media other than images, such as text, audio, or table data. When using text data such as papers or news as the secondary search criteria, the primary search criteria should be as broad as possible, which will reduce search omissions and provide the user with the information they truly need with the least amount of effort and in the least amount of time.

[0407] Needless to say, you can further narrow your search using third and fourth search criteria based on the second search results. Without going into details, information regarding price, delivery date, shape, etc. can be extracted from text data or image data and stored as table data in DBP and XOP, enabling real-time searches using a variety of search criteria.

[0408] The system described above has secondary and subsequent search functions on the user side, so it places less of a burden on search processing on web search systems, etc., and can be easily implemented by utilizing existing communication lines such as 5G or dedicated communication lines.

[0409] Furthermore, with this system configuration, the amount of information that the user needs to search for in the secondary search is limited, so there is no need for a large-scale system incorporating a high-capacity memrism processor; a small-capacity memrism processor is sufficient. It can also be realized as a system using a PC equipped with an FPGA-based memrism processor, and it is expected that this system will also be used in smartphones and other devices in the future.

[0410] Various application examples have been described above. In the case of voice recognition, special information processing is required to eliminate external noise and other disturbances, and to tolerate ambiguity in expression.

[0411] Therefore, if memorism processor technology is used as a technology to complement existing voice processing technology, it is expected that highly accurate voice recognition can be achieved with less development burden. A voice user interface using an ultra-fast voice recognition engine that combines SOP and DBP is ideal for independent terminal devices that do not rely on the cloud, and since it enables highly accurate voice recognition, it is expected to be used as a voice search interface that closely understands the characteristics of the user's speech.

[0412] Recent AI, as exemplified by ChatGPT, is not an actual artificial intelligence (AI) that can understand linguistic concepts such as ontology and provide answers, but rather an apparent artificial intelligence (AI) that is configured to provide answers that people intend. In order to provide information that appears to be what a person intends, it is necessary to prepare a large amount of data and have the system learn from that data so that it can provide the answer that the user intends, which requires extremely complex and labor-intensive information processing. Since performance is proportional to the size of the system, the system has to be huge.

[0413] Information processing such as ontology, which is a form of semantic language processing, requires a complex and numerous tag that links each word to the concept of the word, and in the case of images, it requires humans to recognize and label the objects contained in each image.

[0414] This method is based purely on the idea of ​​searching (detecting) the intended information, and is a method for mechanically finding the intended information.

[0415] The information retrieval process of this method, which efficiently finds information, is essential for various types of knowledge information processing.

[0416] Although developing software applications to utilize this technology requires some getting used to, the hardware search algorithm itself is very simple.

[0417] Naturally, by incorporating AI functions into systems 1, 2, and 3 described above, the accuracy and performance of the search will improve the more searches are performed, making it possible to realize a system that can provide the user with the information they are looking for. Since this AI function is an auxiliary means, it does not need to be a large-scale system; AI that can be realized using the CPU and GPU of a general-purpose PC will be sufficient.

[0418] We explain the memory technologies important to memristive processors.

[0419] The original semiconductor miniaturization technology known as Moore's Law has already reached its limits.

[0420] However, advances in manufacturing technologies such as the semiconductor stacking technology known as chiplets have made DRAM and FLASH memory extremely large-scale, and the cost per byte is decreasing year by year.

[0421] Memrism processors are advantageous because they can be developed and produced using these general-purpose memories.

[0422] In a typical system, software costs are overwhelmingly greater than the costs of hardware such as servers.

[0423] Search processing is essential for many information processing systems.

[0424] By providing inexpensive, large-capacity memrism processors to the market, it will be possible to reduce software costs and the overall system costs.

[0425] Furthermore, search servers using memristive processors require significantly less power and space than servers with the same performance, making them ideal for data centers and cloud providers that use a large number of servers.

[0426] This method not only reduces the level of expertise required for information search processing, but also places emphasis on security, making it ideal for on-premise (in-house) systems that place a high priority on security, such as personal information data and medical information data.

[0427] In the case of PCs, software (functions) for searching e-mails and full-text searches for missing files come standard, but the configuration and search accuracy are not necessarily sufficient to set sufficient search conditions.

[0428] In particular, if a full-text search index becomes corrupted, users have no choice but to endure long searches or give up on the search.

[0429] Using this system on a PC will eliminate these problems and provide a diverse and comfortable search environment on the PC as well.

[0430] This method does not require complex information processing, and furthermore, it significantly reduces waiting times such as preprocessing and learning processing, thereby shortening development effort and development time and reducing system prices.

[0431] This will enable faster, more powerful search technology, standardization of search technology, and application not only to search systems but also to knowledge processing systems, particularly small knowledge processing systems.

[0432] These can be used in new products such as non-computer devices, such as storage devices with built-in search functions and AV devices such as TVs with built-in intelligent search functions, as well as new types of web services.

[0433] In the future, by defining and standardizing the programming languages ​​and multimedia arrangements used to write web content, including the Internet, and by creating and posting web content in standardized forms, the information search efficiency of this system will be greatly improved, and this technology will evolve into a system that is even more convenient and meets the various expectations of users.

[0434] Sun Tzu's words, "He who controls the information controls the battle," have been passed down to the present day as "He who controls the information controls the world," and the "value of information" is universal. Furthermore, as symbolized by the saying "time is money," the "value of time" is also universal.

[0435] A decision support system that selects important and up-to-date information and provides it as quickly as possible will accelerate decision-making by leaders in a variety of fields, contributing to the development of companies, the economy, and society, as well as the reduction of all social risks. It is expected that this technology will contribute widely to the development of science and technology and improved operational efficiency at general companies, as well as to a wide range of fields in the big data society, such as eliminating the frustration people experience when searching and providing them with the highest level of intellectual satisfaction.

[0436] (Modification) The present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit and scope of the invention.

[0437] First, as mentioned above, SOP can be used not only for two-dimensional information but also for three-dimensional information. In other words, as will be described later, SOP can be applied to not only images but also voices and other two-dimensional array information to N-dimensional array information by using the semiconductor information retrieval accelerator 101 that can recognize two-dimensional to N-dimensional array information by expanding the dimension. SOP is a semiconductor information retrieval accelerator 101 that retrieves stream or temporary data. Since it is not data for viewing purposes, the range of applications is enormous even at QVGA size or even smaller.

[0438] SOPs come in a variety of data sizes and search functions, but in the case of a QVGA-sized ASIC, simulation results from the GDS development stage showed that power consumption during calculations was 1.2W, and power consumption during non-calculation periods was nearly 0W.

[0439] Even when the size is large for medical applications, etc., it is possible to perform calculations with a maximum of a few watts, making it ideal for AI systems that require low power consumption, especially stand-alone information processing.

[0440] In the case of FPGA, large data size is difficult, but practical SOP can be realized even with small capacity. Power consumption is about 25W, but it is characterized by the fact that it can be customized to the hardware configuration that is most suitable for customer needs.

[0441] Next, we will explain the configuration and power performance of DBP, XOP, and TSP. Unlike SOP, these three types of semiconductor information search accelerators 101 are targeted at accumulated data, and therefore are easier to use when the data capacity is large.

[0442] As an example, if a massively parallel search calculation element specialized for each search is incorporated into a DRAM type memory and integrated, it becomes a PIM type semiconductor information retrieval accelerator 101, and if the DRAM memory and each massively parallel search calculation element are configured adjacent to each other, it becomes a PNM type semiconductor information retrieval accelerator 101.

[0443] In terms of performance, the PIM method is ideal, but the drawback is that it requires embedding calculation elements inside the memory, making development very difficult.

[0444] On the other hand, the PNM method does not match the PIM method in terms of performance, but since it is only necessary to develop the calculation elements and connect them to general-purpose memory and general-purpose storage, the ASIC development is easy and it can be produced cheaply, and it can also be easily realized in FPGA.

[0445] In the PIM and PNM semiconductor information retrieval accelerators 101, the memory and the massively parallel search calculation elements are integrated or adjacent to each other, so the distance that electrons travel is short, making it possible to operate with low power. For example, the power of the semiconductor information retrieval accelerator 101 per 16 gigabytes is 10 W or less, enabling extremely energy-saving and highly efficient search processing.

[0446] The most noteworthy thing about using the three types of semiconductor accelerators, DBP, XOP, and TSP, in this invention is that searches are completed in the same time as or shorter than the index method, even though preprocessing such as indexing is not required.

[0447] Although a simple comparison is difficult due to the system configuration, it is expected that the power performance will be several tens to tens of thousands of times better than a CPU or GPU, and it is also inexpensive to use, so it can be used for a wide range of purposes, from big data to personal use.

[0448] In addition, in the above embodiment, it has been shown that image and voice recognition and AI-like information processing can be easily realized by combining 2D SOP and DBP. However, multimodal search systems and multimodal AI systems that combine information including spatial information such as 3D maps, molecular structures, and space structures, as well as XOP and TSP, which will be explained in more detail below, will be described later.

[0449] (Standardization of Information Processing) Here is an example of information retrieval standardization. So far, SOP has demonstrated its ability to quickly create various feature IDs from images and audio.

[0450] TSP can quickly create feature IDs from text data.

[0451] Furthermore, DBP can quickly search for these feature IDs, and XOP can quickly classify these feature IDs.

[0452] By taking advantage of the high speed characteristics of each of the above, it becomes possible to generate standardized feature IDs in real time when creating various multimedia content. The purposes of this standardized feature ID are diverse, including data retrieval across a wide range of multimedia content, image and voice recognition, user authentication, data matching, feature and anomaly detection, data matching, data sorting, and data classification.

[0453] This standardized feature ID can be used as a universal feature ID for various multimedia content, and will be extremely useful for searching the vast amount of big data. The above information is useful for searching any type of multimedia, including web information. In the above embodiment, we have introduced examples of application of the four semiconductor information search accelerators 101, SOP (2D, 3D), DBP, XOP, and TSP, individually, and examples of search systems and AI systems that can be realized by combining them, but they can also be applied to multimodal searches.

[0454] In other words, although the term "real-time processing" is widely used in the world, with current CPU and GPU methods of information processing, when the data in question becomes large, preprocessing such as indexes, metadata, and even machine learning becomes necessary, making true real-time processing difficult.

[0455] In contrast, the present invention makes it possible to realize a system capable of searching for information and performing AI, including recently obtained information (highly fresh information), which is currently the biggest challenge in multimodal search and AI, at low power consumption, high speed, and low cost.

[0456] In other words, by using a memorism processor consisting of four (five) semiconductor information retrieval accelerators 101, namely SOP (2D, 3D), DBP, XOP, and TSP, it is possible to realize a multimodal retrieval system or a multimodal AI system.

[0457] Hereinafter, a case where a multimodal search is performed using the information search system-B (FIGS. 14 to 16) will be described. The same components as those in FIG. 14 to FIG. 16 are designated by the same reference numerals and detailed description thereof will be omitted.

[0458] The external information sources 104 of this multimodal search system 107 are a collective term for anything that can provide information to this system through an external interface 103, such as information from data storage (including USB memory, SD cards, etc.), database system information such as feature IDs for the various images, audio, and videos that have been explained so far, information that forms the basis of the database, information obtained from cameras, microphones, and communication networks such as the Internet, and information obtained via radio waves such as broadcasting.

[0459] Of course, standalone operation without relying on wired or wireless connections is also possible. Similarly, input and output from the user is also performed through an interface 103 with the outside.

[0460] This search system 107 is entirely controlled by a CPU or other semiconductors, and is provided with an interface 103 for acquiring search target information from the outside, and an interface 103 for interacting with a user (which may be an object), The information search engine 105 of this search system is composed of multiple types of semiconductor information search accelerators 101 capable of performing real-time processing of broad searches of information including multiple types of information immediately after acquisition, Each of these semiconductor information search accelerators 101 is provided with driving software 102 for driving the respective semiconductor information search accelerators. This search system 107 is a multimodal search system 107 characterized in that the multiple types of semiconductor information search accelerators 101 independently search multiple types of information in parallel in accordance with instructions from the driving software 102 corresponding to search conditions for each type of information provided from the outside, and output the results individually or output an aggregate of the individual results.

[0461] The multiple types of semiconductor information search accelerator 101 is a multimodal search system 107 that does not require preprocessing including indexes, metadata, and learning, and is characterized by processing a broad search of information including the multiple types of information just obtained in real time.

[0462] The multiple types of semiconductor information search accelerators 101 1st category: 2D array information including images and audio 2. Molecular structure and 3D sequence information including 3D maps 3. Structured data information including table data 4. Structured data information including two sets of table data 5. Character string information contained in the text The multimodal search system 107 is composed of the above five types of semiconductor information search accelerators 101 and is characterized by enabling searches for commonly used information.

[0463] The broad definition of search is a multimodal search system 107 that includes data searching, image and voice recognition, user authentication, data collation, feature and anomaly detection, data matching, data sorting, and data classification.

[0464] The multimodal search system 107 is characterized by generating standard feature IDs related to the search of data related to various types of multimedia content, image and voice recognition, user authentication, data collation, feature and anomaly detection, data matching, data sorting, and data classification.

[0465] The multiple types of semiconductor information retrieval accelerators 101 are a multimodal search system 107 characterized in that they are semiconductor information retrieval accelerators 101 that implement both a memory for storing data to be searched and a process-in-memory (PIM) architecture semiconductor memory and search function capable of searching the data stored therein, in an FPGA or ASIC.

[0466] The multiple types of semiconductor information retrieval accelerators 101 are a multimodal search system 107 characterized in that they are semiconductor information retrieval accelerators 101 in which the search function of a process near memory (PNM) architecture semiconductor capable of searching data stored in a memory adjacent to the data to be searched is implemented in an FPGA or ASIC.

[0467] 14 to 16 show five types of semiconductor information search accelerators 101: SOP (2D, 3D), DBP, XOP, and TSP. However, multimodal searches can be made possible by using two or more types of semiconductor information search accelerators 101 depending on the purpose of use.

[0468] The number of accelerators (number of semiconductors) for each can be freely chosen, for example, two SOPs, three DBPs, etc.

[0469] The above configuration provides the following effects.

[0470] It frees search of extremely broad information from the need for pre-processing such as indexes, metadata, and machine learning.

[0471] Each of the power-saving semiconductor information retrieval accelerators 101 independently performs high-speed and advanced retrieval processing, enabling extremely efficient, flexible and convenient retrieval processing.

[0472] By making the database of the media information retrieval system 106 a database including label and tag information, a system having the ability to recognize, authenticate, and match objects, that is, a search capability including the concept of things, a so-called multimodal retrieval system 107 having intelligent search capabilities, in other words a multimodal AI system 108, can be realized.

[0473] By treating a wide variety of information such as images, audio, text, and general-purpose data as database data, it is possible to improve the ability to recognize and match information. In other words, the complex linking process of ontology is replaced by multimedia (known information) that has been given labels and tags.

[0474] The multiple types of semiconductor information retrieval accelerators 101 described above are all based on memorism processors that enable high-speed information retrieval without using machine learning metadata or indexes for information retrieval prior to information retrieval. This not only eliminates the search wait time associated with creating metadata or indexes for information retrieval, but also significantly reduces the level of expertise required for search processing. It therefore becomes possible to provide a highly convenient and flexible multimodal retrieval system 107 and multimodal AI system 108 to people other than software and hardware experts, thereby contributing to the elimination of the digital divide.

[0475] The information retrieval system-C shown in FIG. 18 can also be applied to a multi-modal refinement retrieval system.

[0476] The above example is just one example of aggregating multimodal search results. The main aim of multimodal search and AI is to take into account the user's search intent, aggregate search results, and structure them in a way that produces targeted results that are easy for people to understand.

[0477] A specific example of a highly convenient system is given below.

[0478] If there is no appropriate image or audio information to provide as a search condition for this multimodal search system 107, it is also possible to have a natural language generation AI generate the intended image or audio using a natural language prompt, and provide the generated image or audio as input to this multimodal search system 107.

[0479] Conversely, it is also possible to have generative AI process images and audio and convert them into natural language (text).

[0480] This is an example of a highly convenient system that can be used for any information, not just images, voice, and text, and that combines this multimodal search system 107 with AI technology. A typical usage of the multimodal search system 107 is shown below.

[0481] A method of using a multimodal search system 107 including a function for improving the accuracy of search results using machine learning technology.

[0482] A method of using the multimodal retrieval system 107 to select and normalize multimodal data.

[0483] The multimodal search system 107 includes a protocol and an external interface for standardizing the format of search results, and is a method of use that enables interoperability with different search systems.

[0484] The multimodal search system 107 is a method of use that standardizes the format, process, and external interface of the training dataset of an AI system, enabling exchange and reuse of the system for different AI systems.

[0485] A method of using the multimodal search system 107 for the purpose of eliminating the digital divide between the hardware and software technologies of users.

[0486] A method of using the multimodal search system 107 as a stand-alone system.

[0487] A method of using the multimodal search system 107 for an on-premise system from the viewpoint of security, including personal information.

[0488] A method of using the multimodal search system 107 for searching frames of a video that includes images and audio.

[0489] A method of use characterized by providing information generated by a generation AI to a search input of a multimodal search system 107.

[0490] Although not included in the claims due to the spirit of the invention, it is also possible to use generative AI, which is rapidly evolving, to develop driving software 102, develop new application methods, and predict the market.

[0491] Although not included in the claims due to the spirit of the invention, it is also possible to utilize the rapidly evolving chiplet technology, 3D semiconductor integration technology, and semiconductor interface technology to develop new types of semiconductor memory elements and semiconductor technologies, such as the PIM method, PNM method, and even the PWM method, which is a memory utilization technology in a broad sense.

[0492] It goes without saying that while the present invention is directed to multimodal search and AI, many of the application examples can be used in single-modal search and single-modal AI. [Industrial Applicability]

[0493] 1. Information retrieval technology in general, especially multimedia complex retrieval 2. Decision Support Systems 3. Academic research support system 4. Alternatives to Artificial Intelligence (AI) Processing Systems 5. Realization of advanced information search and artificial intelligence (AI) systems on PCs and small servers 6. Products and services that incorporate search functionality in new ways 7. Autonomous Driving Systems 8 Standalone multimodal search or AI systems 9 On-premise system that resolves data security and privacy issues 10 Systems linked to conventional search systems and AI systems 11 Promote global ID generation per unit time for video and audio 12 Effective use of dormant archives 13 Large-scale web search systems, SNS systems, and cloud systems that aim for low power consumption and high performance 14 Eliminating risks to financial systems, etc., with the aim of eliminating batch processing such as night batches 15 Contributing to promoting digital transformation of information processing and eliminating the digital divide 16 Expansion of the computing power of computers themselves and the range of applications [Explanation of symbols]

[0494] 11 Sampling points (pixels) 12 Reference points 13 Point Range 14 Set operation mark 15 points (pixels) of data 16 Reference Point Address 17 Relative Point Address 18 Matched pixel address 19 Calculation Conditions 20 Pattern Matching Templates 21 Set operation expressions (template search conditions) 22 Composite Pattern Matching Templates 101 Semiconductor Information Search Accelerator 102 Semiconductor Information Retrieval Accelerator Driven Software 103 External Interface 104 Source of information (external information) 105 Information Search Engine 106 Media Information Retrieval System 107 Multimedia Information Retrieval System

Claims

1. 1. A computer-implemented information retrieval system, comprising: A plurality of semiconductor information search accelerators each of which performs an independent search for a different type of information, Each semiconductor information accelerator: A semiconductor memory is provided for storing search target information of a specific information type, the search target information stored in the semiconductor memory is searched for in accordance with given search conditions, and the search result is output. A semiconductor information search accelerator; A plurality of device drivers are installed in the information retrieval system and drive each semiconductor information retrieval accelerator; A search engine that is installed in the information search system and executes searches in parallel on each semiconductor information search accelerator through a device driver according to search conditions for each type of information given from the outside, and outputs the results individually or aggregates the individual results and outputs them; An information retrieval system characterized by:

2. 2. The information retrieval system according to claim 1, The search engine provides a user interface that prompts the user to enter input conditions and outputs search results to the user.

1. An information retrieval system comprising:

3. 2. The information retrieval system according to claim 1, The search engine provides the given search conditions to each semiconductor search accelerator according to the type of search target information stored in the semiconductor search accelerator.

1. An information retrieval system comprising:

4. 4. The information retrieval system according to claim 3, The information retrieval system is characterized in that the search engine uses a search result output by a semiconductor search accelerator for one information type as a search condition for a semiconductor search accelerator for another information type depending on the information type of the search result.

5. 2. The information retrieval system according to claim 1, The plurality of semiconductor information retrieval accelerators include two types of semiconductor information retrieval accelerators, The first type of semiconductor information retrieval accelerator is a semiconductor information retrieval accelerator capable of retrieving features of two-dimensional array pattern information stored in the accelerator using a predetermined feature retrieval condition; The second type of semiconductor information retrieval accelerator is capable of storing in advance, as record table data, characteristics of the plurality of two-dimensional array pattern information retrieved by the first type of semiconductor information retrieval accelerator for each piece of information; Furthermore, the second type of semiconductor information retrieval accelerator is a semiconductor information retrieval accelerator capable of retrieving the record table data by using features of the two-dimensional array pattern information newly retrieved by the first type of semiconductor information retrieval accelerator as a retrieval condition of the plurality of pre-stored two-dimensional array pattern information.

1. An information retrieval system comprising:

6. 2. The information retrieval system according to claim 1, wherein the two-dimensional array pattern information is general-purpose database information such as an image, a frame image of a video, audio, stock price information, weather information, map information, two-dimensionally arranged personal information, sales performance information, and customer information.

7. 2. The media information retrieval system according to claim 1, wherein the retrieval condition for the first type of semiconductor information retrieval accelerator is a pattern matching condition for two-dimensional array information.

8. 2. The media information retrieval system according to claim 1, wherein the retrieval conditions for the second type of semiconductor information retrieval accelerator are a bitmap operation retrieval condition and a bit serial operation retrieval condition.

9. 2. The media information retrieval system according to claim 1, wherein both the retrieval conditions of said first type of semiconductor information retrieval accelerator and the retrieval conditions of said second type of semiconductor information retrieval accelerator are automated.

10. 2. The media information retrieval system according to claim 1, wherein the record table data is increased by using the two types of semiconductor information retrieval accelerators in parallel.

11. 2. The information retrieval system according to claim 1, The plurality of types of semiconductor information retrieval accelerators store multimedia information including at least text and images provided from an external information source for each type of media; The information search engine searches for information based on search conditions and search instructions for each type of media set by the user, The function to output the information search results and An information retrieval system comprising:

12. 12. The information retrieval system according to claim 11, wherein said plurality of types of semiconductor information retrieval accelerators are capable of retrieving commonly used information such as voice and data tables in addition to text and images.

13. 12. The information retrieval system according to claim 11, wherein the external information sources are data storage, database systems, information that is the source of a database, and information available via wired or wireless communication.

14. 12. The information retrieval system according to claim 11, wherein the external information source is the media information retrieval system according to claim 1.

15. 12. The information retrieval system according to claim 11, The information search engine, A function for the user to pre-register multiple pieces of information necessary for searching for the information he or she desires, and a function for the user to set search conditions and give search instructions. a function of synthesizing the plurality of pieces of preregistered information and the search conditions based on the search instruction to generate first and second search conditions; a function of searching a search system outside the multimedia information search system using the generated first search condition and importing the result information into the multimedia information search system; a function of narrowing down the retrieved result information using the generated second search condition and outputting the narrowed down search result to a user; An information retrieval system comprising:

16. A non-computer device including a storage device, an image device, and an audio device incorporating the information retrieval system according to claim 1.

17. 2. The information retrieval system according to claim 1, wherein said plurality of types of semiconductor information retrieval accelerators are FPGAs or ASICs.

18. The multiple types of semiconductor information search accelerators 1st category: 2D array information including images and audio 2nd category: Molecular structure and 3D sequence information including 3D maps Type 3: Structured data information including table data 4. Structured data information including two sets of table data 5. Character string information contained in the text 2. An information retrieval system according to claim 1, comprising a semiconductor information retrieval accelerator capable of storing up to five types of information and executing a search on the stored information.

19. The information retrieval executed by the information retrieval system according to claim 1 comprises: An information retrieval system characterized by functions including data searching, image and voice recognition, user authentication, data matching, feature and anomaly detection, data matching, data sorting, and data classification.

20. The multimodal search system of claim 1, wherein the multiple types of semiconductor information search accelerators are semiconductor information search accelerators that implement both a memory for storing data to be searched and a process-in-memory (PIM) architecture semiconductor memory and search function capable of searching the data stored therein, in an FPGA or ASIC.

21. The multimodal search system according to claim 1, wherein the multiple types of semiconductor information search accelerators are semiconductor information search accelerators in which the search function of a process near memory (PNM) architecture semiconductor capable of searching data stored in a memory adjacent to the data to be searched is implemented in an FPGA or ASIC.

Images