Information processing apparatus, information processing method, and program

Abstract

An apparatus and method are provided that perform search processing on data in a category different from a category of a text using a query and word-level weight information. The information processing apparatus includes a data processing unit to which a query composed of a text to be applied to data search processing is input, and the data processing unit performs data search processing based on the query. Along with the query, word-level weight information that constitutes the query is input into the data processing unit, and the data processing unit performs search processing on data in a category different from the text using the query and the word-level weight information. Search processing is performed with increased influence on a word to which a positive weight is assigned, and search processing is performed with decreased influence on a word to which a negative weight is assigned.

Description

Technical Field

[0001] This disclosure relates to information processing apparatus, information processing methods, and procedures. More specifically, this disclosure relates to information processing apparatus, information processing methods, and procedures for facilitating the acquisition of search results that reflect user intent in data search processing. Background Technology

[0002] In data search processing of data in databases or networks, users can perform searches by entering keywords or queries (search statements) as free descriptions expressed in text.

[0003] For example, in image search processing, users generate keywords representing the features of the image to be searched or execute a search query (search statement).

[0004] For example, by entering "dog photos" as a query (search statement) and performing a data search, you can selectively retrieve image data of dog photos from a large amount of image data in a database or on the web.

[0005] Note that, for example, Patent Document 1 (Japanese Patent Application No. 2008-243024) discloses prior art for data search processing using keywords.

[0006] However, when using keywords or queries (search statements) in data searches, search results are often obtained that differ from the user's intent.

[0007] Specifically, users unfamiliar with data search processing have difficulty generating optimal keywords or queries (search statements) and obtaining the expected search results.

[0008] Reference List

[0009] Patent documents

[0010] Patent Document 1: Japanese Patent Application No. 2008-243024 Summary of the Invention

[0011] The problem to be solved by the present invention

[0012] For example, this disclosure is made in view of the above problems, and its purpose is to provide an information processing apparatus, information processing method and program that facilitate the acquisition of search results that reflect user intent in data search processing.

[0013] Solution to the problem

[0014] The first aspect of this disclosure is

[0015] Information processing equipment, including a data processing unit, wherein the data processing unit: Retrieve queries that include the text to be applied to data search processing; and Based on this query, data search processing is performed, where, The data processing unit Along with the query, obtain the weight information for each word or word string used to constitute the query, and Using this query and the weight information for each word or word string, perform search processing on data in categories different from the text category.

[0016] Furthermore, the second aspect of this disclosure is

[0017] Information processing methods include: Retrieve queries that include the text to be applied to data search processing, and Weight information for each word or word string used to constitute the query; and Using this query and the weight information for each word or word string, perform search processing on data in categories different from the text category.

[0018] Furthermore, the third aspect of this disclosure is: A program that enables an information processing device to execute information processing, wherein the program enables the data processing unit of the information processing device to: Retrieve queries that include the text to be applied to data search processing; and Weight information for each word or word string used to constitute the query; and Using this query and the weight information for each word or word string, perform search processing on data in categories different from the text category.

[0019] It should be noted that, for example, the program disclosed herein can be provided to an information processing device or computer system capable of executing various program codes via a storage medium or communication medium provided in a computer-readable format. By providing such a program in a computer-readable format, processing corresponding to the program is implemented on the information processing device or computer system.

[0020] Other objects, features, and advantages of this disclosure will become apparent from the detailed description and accompanying drawings based on embodiments of this disclosure described later. Note that in this specification, a system is a logical collection configuration of multiple devices and is not limited to a system in which devices with various configurations are housed in the same enclosure.

[0021] According to one embodiment of the present disclosure, an apparatus and method are implemented for performing search processing of data in a category different from the category of text, using query and weight information for each word.

[0022] Specifically, for example, it includes a data processing unit that receives a query containing text applicable to data search processing and performs data search processing based on the query. The data processing unit receives weight information for each word constituting the query along with the query, and performs search processing on data in categories different from the text using the query and the weight information for each word. Search processing with increased influence is performed on words with positive weights, and search processing with decreased influence is performed on words with negative weights.

[0023] This configuration enables devices and methods for performing search processing of data in categories different from the categories of text, using query and weight information for each word.

[0024] It should be noted that the effects described in this specification are merely illustrative and not limited, and additional effects may be provided. Attached Figure Description

[0025] Figure 1 This is a diagram illustrating an overview of data search processing using queries (search statements) as free descriptions.

[0026] Figure 2 This is a diagram showing a configuration example and an input example of the user interface (UI) where weighted queries can be entered.

[0027] Figure 3 This is a diagram showing a configuration example and an input example of the user interface (UI) where weighted queries can be entered.

[0028] Figure 4 This is a diagram showing a configuration example and an input example of the user interface (UI) where weighted queries can be entered.

[0029] Figure 5 This is a diagram showing a configuration example and an input example of the user interface (UI) where weighted queries can be entered.

[0030] Figure 6 This is a diagram showing a configuration example and an input example of the user interface (UI) where weighted queries can be entered.

[0031] Figure 7 This is a diagram illustrating a specific example of data search processing using weighted queries based on numerical inputs.

[0032] Figure 8 This is a diagram illustrating a specific example of data search processing using weighted queries based on numerical inputs.

[0033] Figure 9 This is a diagram illustrating a specific example of data search processing using a weighted query with scrollbars.

[0034] Figure 10 This is a diagram illustrating a specific example of data search processing using a weighted query with scrollbars.

[0035] Figure 11 This is a diagram illustrating a specific example of data search processing using queries based on Japanese text.

[0036] Figure 12 This is a diagram illustrating a specific example of data search processing where music data is set as the search target.

[0037] Figure 13 This is a diagram illustrating an example configuration of the information processing apparatus of this disclosure.

[0038] Figure 14 It is a diagram showing the detailed configuration of the data processing unit and the details of the processing to be performed.

[0039] Figure 15 This is a diagram illustrating the overview of a deep learning model that uses an attention mechanism.

[0040] Figure 16 This is a diagram illustrating a specific example of the processing performed by the weighted query corresponding feature vector generation unit.

[0041] Figure 17 This is a diagram showing an example of data stored in a database.

[0042] Figure 18 This is a diagram illustrating a configuration example including a content encoder within the data processing unit of an information processing device.

[0043] Figure 19 This is a flowchart illustrating a sequence of processes performed by the information processing apparatus of this disclosure.

[0044] Figure 20 This is a flowchart illustrating a sequence of processes performed by the information processing apparatus of this disclosure.

[0045] Figure 21 This is a diagram illustrating an example of query input in the process of extracting a specific scene desired by a user from movie content.

[0046] Figure 22 This is a diagram illustrating the processing by the data processing unit in extracting a specific scene desired by the user from the movie content.

[0047] Figure 23 This is a flowchart illustrating the process sequence used to describe the extraction of a specific scene desired by a user from movie content.

[0048] Figure 24This is a diagram illustrating an example of query input in the process of extracting a specific scene desired by the user from moving video content.

[0049] Figure 25 This is a diagram illustrating the processing by the data processing unit in extracting a specific scene desired by the user from moving video content.

[0050] Figure 26 It is a graph showing the data stored in the motion data database.

[0051] Figure 27 This is a diagram illustrating an example of query input in the process of extracting desired motion data from motion data.

[0052] Figure 28 This is a diagram illustrating the process by which the data processing unit extracts the motion data desired by the user from the motion data.

[0053] Figure 29 A diagram illustrating an example of the hardware configuration of the information processing apparatus of this disclosure. Detailed Implementation

[0054] In the following, the information processing apparatus, information processing method, and procedure according to this disclosure will be described in detail with reference to the accompanying drawings. Note that the description will be given according to the following items.

[0055] 1. Overview of data search processing using queries (search statements) as free descriptions.

[0056] 2. Configuration examples for the user interface (UI) that allows input of weighted queries and input examples for weighted queries.

[0057] 2-(a) Configuration of query input area with weight settings based on numerical input and query input example

[0058] 2-(b) Configuration of query input area and query input example that can set weights based on parentheses (brackets) input

[0059] 2-(c) Configuration of the query input area that allows for weight setting using scrollbars and an example of the query input.

[0060] 2-(d) Configuration of query input areas that can use color display (heatmap) for weight setting and query input examples

[0061] 3. Specific examples of data search processing using weighted queries

[0062] 3-(1) Specific examples of data search processing using weighted queries based on numerical input

[0063] 3-(2) Specific examples of data search processing using weighted queries with scrollbars

[0064] 3-(3) Specific examples of data search processing using queries in languages ​​other than English

[0065] 3-(4) Specific examples of data search processing using music data as the search target

[0066] 4. Details of the configuration and processing of the information processing equipment disclosed herein

[0067] 5. A sequence of processes performed by the information processing equipment of this disclosure.

[0068] 6. Application examples of data search and processing using the information processing equipment disclosed herein

[0069] 6-(1) Example of processing to extract specific scenes desired by users from movie content

[0070] 6-(2) Example of extracting specific scenes desired by users from motion video content

[0071] 6-(2) Example of extracting specific scenes desired by users from motion video content

[0072] 7. Hardware configuration example of the information processing device disclosed herein

[0073] 8. Overview of the configuration disclosed herein

[0074] [1. An overview of data search processing using queries (search statements) as free descriptions]

[0075] First, refer to Figure 1 The accompanying figures describe an overview of the data search process using queries (search statements) as free descriptions.

[0076] Figure 1 This is a diagram illustrating an example of a user interface (UI) displayed on a display unit of the information processing device of this disclosure, such as a user interface (UI) for data search processing of data in a database or network.

[0077] Figure 1 The UI shown is an example of a data search UI displayed on the display unit of an information processing device (e.g., PC, smartphone, tablet, etc.) disclosed herein.

[0078] In the case of performing a data search, firstly, in step S01, the user enters the query (search statement) into... Figure 1 The UI shown above is the [Query] area, which is the query (search statement) input area 11.

[0079] Figure 1 An example is shown, where

[0080] The query is "people running on the beach".

[0081] enter.

[0082] That is, this is an example of entering "people running on the beach" as a query.

[0083] When the query input to the query input area 11 is completed, then in step S02, the search (search start) icon 12 is operated (touched).

[0084] This user action initiates a data search for the target data (e.g., data in a specific database or network).

[0085] When the search process is complete, Figure 1 The search results (Result) display area 13 at the bottom of the UI shown displays an image as a search result.

[0086] In the example shown in the attached figure, the image corresponding to the query input in query input area 11 is, i.e. query="people running on the beach", That is, the image of “people running on the beach” is displayed as a search result in the search results display area 13.

[0087] However, the search results displayed in search results display area 13 may differ from the search results the user wants.

[0088] In other words, use input to Figure 1 The query (search statement) in query input area 11 shown in the figure represents the user's search intent for performing a data search, that is, query="people running on the beach", The search intent for this string, which includes multiple words, varies depending on the user.

[0089] For example, assuming there are multiple users performing searches using the above query, suppose there are, for example, the following intents as the search intents of multiple users.

[0090] User A's search intent is to search for "images of many people running on a beach".

[0091] User B's search intent is to search for "images of a person running on a quiet beach".

[0092] User C's search intent is to search for "images of people running on a busy beach".

[0093] User D's search intent = wants to search for "images of people running on a gentle beach".

[0094] As mentioned above, the images desired are often different among search users, and... Figure 1 The search results shown in area 13 may differ from the user's intent.

[0095] In such a situation, in order to obtain the search results that the user wants, the user needs to repeatedly perform the process of creating different queries and performing re-searches.

[0096] This disclosure solves such a problem.

[0097] Specifically, even when a search is performed based on a query (search statement), the information processing device of this disclosure facilitates the acquisition of search results that reflect the user's intent by enabling a search to be performed, in which weights are assigned to each of the words constituting the query (search statement) or to a single word string comprising multiple words.

[0098] [2. Configuration examples for the user interface (UI) that allows input of weighted queries and input examples for weighted queries]

[0099] Next, we will describe a configuration example of the user interface (UI) that allows input of weighted queries and an input example of a weighted query.

[0100] As described above, when performing a search based on a query (search statement) as a free description, the information processing apparatus of this disclosure enables a search in which a weight is assigned to each word or word string that includes multiple words constituting the query (search statement).

[0101] Note that the weights of each constituent word and word string in the query (search statement) are used, for example, to generate the feature vector to be applied to the search process.

[0102] That is, for example, by applying similarity analysis of feature vectors, data search processing performed in the information processing apparatus of this disclosure is carried out.

[0103] Specifically, the similarity between the query-corresponding feature vector generated based on the query and the content-corresponding feature vector corresponding to each piece of data (e.g., each piece of content, such as an image) that is the search target is analyzed, and the content (image, etc.) with the content-corresponding feature vector that has a high similarity to the query-corresponding feature vector is output as the search result.

[0104] It should be noted that data search processing based on feature vector similarity analysis is itself prior art. However, the information processing apparatus of this disclosure not only performs data search processing based on a general "query-corresponding feature vector" generated from a query (search statement) including words without weight settings, but also performs data search processing based on a new feature vector (i.e., a "weighted query-corresponding feature vector") reflecting the weights of the constituent words of the query (search statement) set by the user.

[0105] The information processing apparatus of this disclosure analyzes the similarity between the "weighted query corresponding feature vector" and the content corresponding feature vector (e.g., each piece of content such as an image) corresponding to each piece of data that is the search target, and outputs the content (image, etc.) with the content corresponding feature vector that has a high similarity to the "weighted query corresponding feature vector" as the search result.

[0106] This process allows us to obtain search results that accurately reflect user intent.

[0107] It should be noted that a specific example of the generation process for "weighted query corresponding feature vector" will be described in detail later.

[0108] Reference Figure 2 The accompanying drawings describe configuration examples of a user interface (UI) for inputting weighted queries in the information processing device of this disclosure, and input examples of weighted queries.

[0109] It should be noted that, for example, the information processing device disclosed herein is an information processing device such as a PC, smartphone, or tablet terminal.

[0110] Similar to the reference above Figure 1 The described data search user interface (UI) is displayed on the display unit of the information processing device disclosed herein.

[0111] That is, the user interface (UI) that displays data search processing for data in a database or network.

[0112] The difference between the data search processing UI displayed on the display unit of the information processing device disclosed herein and the conventional UI is the configuration of the [Query] area (i.e., the query (search statement) input area 11).

[0113] The data processing unit of the information processing device of this disclosure displays a query input area 11 on a display unit. The query input area 11 has a configuration in which the user can set a "weight" for each of the constituent words (including word strings) of a query (search statement).

[0114] It should be noted that "weight" is reflected in the generation of the above "weighted query corresponding feature vector".

[0115] Specifically, words with higher “weights” have a greater impact on data search.

[0116] Reference Figure 2 The accompanying drawings describe a configuration example of the query input area 11 in the data search UI displayed on the display unit of the information processing device of this disclosure, and an input example of a weighted query in the query input area.

[0117] Note that the following description will sequentially describe the configuration of the following four types of query input areas and query input examples.

[0118] (a) Configuration of the query input area that allows weight settings based on numerical input and an example of query input.

[0119] (b) Configuration of query input area and query input example that allows weight settings based on parentheses (round brackets) input.

[0120] (c) Configuration of the query input area that allows for weight settings using scrollbars and an example of the query input.

[0121] (d) Configuration of query input areas that can use color display (heatmap) for weight setting and query input examples

[0122] (2-(a) Configuration of query input area that can set weights based on numerical input and query input example)

[0123] First, we will describe the configuration of the query input area that allows weight settings based on numerical inputs and provide a query input example.

[0124] Figure 2 This is a diagram illustrating the configuration of the [Query] area (i.e., the query input area) and a query input example in the data search UI displayed on the display unit of the information processing device of this disclosure.

[0125] Figure 2 Examples of input for three types of queries (search statements) are shown: (1) Normal search query; (a1) Example 1 of a weighted search query based on numerical input; and (a2) Example 2 of weighted search query based on numerical input.

[0126] The example shown in "(1) Normal Search Query" is a traditional query, that is, query="people running on the beach", The query input field 11 in the data search UI is entered, as shown above. Figure 1 As described.

[0127] The example shown in “(a1) Weighted Search Query Example 1 Based on Numerical Input” is an example of the configuration of the query input area 11a in the data search UI displayed on the display unit of the information processing device of this disclosure and the query input to the query input area 11a.

[0128] In the query input area 11a, you can enter a query that sets a "weight" for each of the constituent words of the query.

[0129] For example, you can enter the following weighted query.

[0130] query="(people:2.0) running on the beach"

[0131] In the above query, "(people:2.0)" means that the weight of the word "people" that constitutes the query is set to 2.0.

[0132] The query includes the following five words: “people”; “running”; “on”; “the”; and "beach".

[0133] query="(people:2.0) running on the beach"

[0134] The weighted query is the following query: The weight of "people" was set to 2.0, and The other four words, “running,” “on,” “the,” and “beach,” were weighted at 1.0.

[0135] When a weighted query is input, the information processing device of this disclosure performs data search processing, wherein the weight of "people" is set to twice the weight of the other four words "running", "on", "the" and "beach".

[0136] That is, data search processing is performed, where the influence of "people" is set to twice the influence of the other four words.

[0137] Specifically, the information processing device of this disclosure generates a "weighted query corresponding feature vector" whose influence is set to twice that of the other four words, and performs similarity determination processing between the generated "weighted query corresponding feature vector" with high influence of "people" and the content corresponding feature vector of images, etc.

[0138] In the similarity determination process, images with high similarity are output as search results.

[0139] As a result, the search results for "people" showed high influence.

[0140] "(a2) Weighted Search Query Example 2 Based on Numerical Input" is another example of query input in the query input area 11a of the data search UI displayed on the display unit of the information processing device of this disclosure. That is, the following input example: query="(people:2.0) running on the (beach:3.0)" As shown.

[0141] "(people:2.0)" means that the weight of the word "people" that constitutes the query is set to 2.0.

[0142] "(beach:3.0)" means that the weight of the word "beach" that makes up the query is set to 3.0 times.

[0143] When a weighted query is input, the information processing device of this disclosure performs data search processing, wherein the weight of "people" is set to 2.0 times and the weight of "beach" is set to 3.0 times.

[0144] That is, data search processing is performed, where the search influence of "people" and "beach" relative to the three unweighted words (running on the) is set to 2x and 3x respectively.

[0145] Specifically, the information processing device of this disclosure generates "weighted query corresponding feature vectors" with the influence of "people" and "beach" relative to the three unweighted words (running on the) set to 2 times and 3 times respectively, and performs similarity determination processing between the generated "weighted query corresponding feature vectors" and the content corresponding feature vectors of images, etc.

[0146] In the similarity determination process, images with high similarity are output as search results.

[0147] As a result, search results with high impact for "people" and "beach" were obtained.

[0148] Note, refer to Figure 2 The example of the weighted query described is an example of how numerical values ​​indicating weights are associated with each word and input.

[0149] For example, the range of values ​​for the weight assigned to each word is a specified range, such as: Weights range from -1.0 to +3.0.

[0150] The range of weight values ​​can be a predefined range or can be freely set by the user.

[0151] Words with a weight of 0 or less (such as weight = -1.0) are words that have a lower impact on search processing compared to other words.

[0152] For example, Weighted query = "(people:-1.0) running on the beach" It is a query, in which The weight of "people" was set to -1.0, and The other four words, “running,” “on,” “the,” and “beach,” were weighted at 1.0.

[0153] When this weighted query is entered, the data search processing for "people" is performed with a weight set to -1.0.

[0154] That is, the search impact of "people" relative to unweighted words is set to -1.0 times in data search processing.

[0155] Specifically, the weight of "people" is set to -1.0 times the "weighted query corresponding feature vector", and the similarity determination process between the generated "weighted query corresponding feature vector" and the content corresponding feature vector of images, etc. is performed.

[0156] In the similarity determination process, images with high similarity are output as search results.

[0157] As a result, search results with low influence of the word "people" were obtained.

[0158] (2-(b) Configuration of query input area and query input example that can set weights based on parentheses (brackets) input)

[0159] Next, we will describe the configuration of the query input area that allows weight settings based on parentheses (or parentheses) input, and provide a query input example.

[0160] Similar to the above Figure 2 , Figure 3 This is a diagram illustrating the configuration of the query input area and a query input example in the data search UI displayed on the display unit of the information processing device of this disclosure.

[0161] Figure 3 Examples of input for three types of queries (search statements) are shown: (1) Normal search query; (b1) Example 1 of a weighted search query based on parentheses (parentheses) input; and (b2) Example 2 of weighted search query based on parentheses (parentheses) input.

[0162] The example shown in "(1) Normal Search Query" is a traditional query, that is, query="people running on the beach", The query input field 11 in the data search UI is entered, as shown above. Figure 1 As described.

[0163] "(b1) Example 1 of weighted search query based on bracket (parentheses) input" is an example of the configuration of the query input area 11b and the query input to the query input area 11b in the data search UI displayed on the display unit of the information processing device of this disclosure.

[0164] In the query input area 11b, you can enter a query that sets a "weight" for each of the constituent words of the query.

[0165] For example, you can enter the following weighted query.

[0166] query="(people) running on the beach"

[0167] This is an example of query input.

[0168] "(people)" means that the weight of the word "people" that constitutes the query is set to 1.1.

[0169] Parentheses (round brackets) indicate that the weight of the word within the parentheses () is set to 1.1.

[0170] Note that double brackets ((())) mean that the weight of the word in double brackets ((())) is set to (1.1)² times.

[0171] Furthermore, triple brackets (((()))) mean that the weight of the word in triple brackets (((()))) is set to (1.1)³ times.

[0172] Figure 3 The query shown in (b1) consists of five words, namely: “people”; “running”; “on”; “the”; and "beach".

[0173] query="(people) running on the beach"

[0174] The weighted query is the following query: The weight of "people" was set to 1.1 times, and The other four words, “running,” “on,” “the,” and “beach,” were weighted at 1.0.

[0175] When a weighted query is input, the information processing device of this disclosure performs data search processing, wherein the weight of "people" is set to 1.1 times the weight of the other four words "running", "on", "the" and "beach".

[0176] That is, the data search processing that performs the "people" influence is set to 1.1 times the influence of the other four words.

[0177] Specifically, the information processing device of this disclosure generates a "weighted query corresponding feature vector" whose influence is set to 1.1 times that of the other four words, and performs similarity determination processing between the generated "weighted query corresponding feature vector" with high influence of "people" and the content corresponding feature vector of images, etc.

[0178] In the similarity determination process, images with high similarity are output as search results.

[0179] As a result, the search results for "people" showed high influence.

[0180] "(b2) Weighted Search Query Example 2 Based on Parentheses (Parentheses) Input" is another example of query input in the query input area 11b of the data search UI displayed on the display unit of the information processing device of this disclosure. That is, the following input example: query="(people) running on the ((beach))" As shown.

[0181] "(people)" means that the weight of the word "people" that constitutes the query is set to 1.1.

[0182] "((beach))" means that the weight of the word "beach" that constitutes the query is set to (1.1)² times.

[0183] When a weighted query is input, the information processing device of this disclosure performs data search processing, wherein the weight of “people” is set to 1.1 times and the weight of “beach” is set to (1.1)² times.

[0184] That is, perform data search processing, where the search influence of “people” and “beach” relative to the three unweighted words (running on the) is set to 1.1 times and (1.1)² times, respectively.

[0185] Specifically, the information processing device of this disclosure generates "weighted query corresponding feature vectors" with the influence of "people" and "beach" relative to the three unweighted words (running on the) set to 1.1 times and (1.1)² times respectively, and performs similarity determination processing between the generated "weighted query corresponding feature vectors" and the content corresponding feature vectors of images, etc.

[0186] In the similarity determination process, images with high similarity are output as search results.

[0187] As a result, search results with high impact for "people" and "beach" were obtained.

[0188] Note, refer to Figure 3 An example of a weighted query is described by parentheses (round brackets) indicating the weights, which are associated with each word and input.

[0189] For example, the brackets for each word are within the specified range: Single brackets (parentheses) to triple brackets (parentheses).

[0190] However, the allowed levels of bracket nesting can be set to a predefined range, or it can be configured to allow users to freely set the levels.

[0191] (2-(c) Configuration of the query input area that allows weight settings using scrollbars and query input examples)

[0192] Next, we will describe the configuration of the query input area that allows for weight settings using scrollbars and provide a query input example.

[0193] Similar to the above Figure 2 and Figure 3 , Figure 4 This is a diagram illustrating the configuration of the query input area and a query input example in the data search UI displayed on the display unit of the information processing device disclosed herein.

[0194] Figure 4 Examples of input for two types of queries (search statements) are shown: (1) Normal search query; and (c1) Example 1 of weighted search query using scrollbar.

[0195] The example shown in "(1) Normal Search Query" is a traditional query, that is, query="people running on the beach", The query input field 11 in the data search UI is entered, as shown above. Figure 1 As described.

[0196] "(c1) Weighted search query example 1 using scroll bar" is a diagram describing the configuration of query input area 11c, which can be used in the information processing device of this disclosure to input a weighted search query using scroll bar, and an example of query input.

[0197] As shown in the figure, the query input area 11c includes a query input section 21 and a word weight setting section 22.

[0198] The word weight setting section 22 displays the word weight setting input section 23 and the weight adjustment scroll bar 24.

[0199] Regular queries, for example, query="people running on the beach", It is entered into the query input section 21.

[0200] The user can input the words whose weights are adjusted from the words in the query input section 21 to the weight setting word input section 23 of the word weight setting section 22.

[0201] The attached image shows an example of entering "people".

[0202] The weight adjustment scroll bar 24 of the word weight setting unit 22 is used to set the weight of the words input to the weight setting word input unit 23. Users can manipulate the scroll bar to set the weight of the words input to the weight setting word input unit 23 (in this example, "people").

[0203] In the example shown in the accompanying drawings, the weight setting range of the weight adjustment scroll bar 24 is -1.0 to 2.0. However, this is an example, and the weight setting range of the weight adjustment scroll bar 24 can be set differently. Furthermore, the weight range that the user can freely set can be adjustable.

[0204] In the example shown in the attached figure, the weight adjustment scroll bar 24 is set at a position where the weight is 1.5 times.

[0205] In this case, the weight of "people" is set to 1.5.

[0206] When a weighted query is input, the information processing device of this disclosure performs data search processing, wherein the weight of "people" is set to 1.5 times the weight of the other four words "running", "on", "the" and "beach".

[0207] That is, perform data search processing, where the influence of "people" is set to 1.5 times the influence of each of the other four words.

[0208] Specifically, the information processing device of this disclosure generates a "weighted query corresponding feature vector" whose influence is set to 1.5 times that of the other four words, and performs similarity determination processing between the generated "weighted query corresponding feature vector" with high influence of "people" and the content corresponding feature vector of images, etc.

[0209] In the similarity determination process, images with high similarity are output as search results.

[0210] As a result, the search results for "people" showed high influence.

[0211] Figure 5 Example 2 of a weighted search query using a scrollbar is shown in "(c2)".

[0212] The query input area 11c shown in (c2) of Figure 55 includes a query input section 21 and two word weight setting sections 22-1 to 2.

[0213] The upper word weight setting section 22-1 includes a weight setting word input section 23-1 and a weight adjustment scroll bar 24-1.

[0214] The lower word weight setting unit 22-2 also includes a weight setting word input unit 23-2 and a weight adjustment scroll bar 24-2.

[0215] Regular queries, for example, query="people running on the beach", It is entered into the query input section 21.

[0216] The superordinate word weight setting section 22-1 and the subordinate word weight setting section 22-2 are areas used to set the weights of different words.

[0217] The example shown in the attached figure illustrates an example where “people” is input to the weight setting word input unit 23-1 of the upper word weight setting unit 22-1 and “running” is input to the weight setting word input unit 23-2 of the lower word weight setting unit 22-2.

[0218] Set the weight adjustment scroll bar 24-1 of the upper word weight setting section 22-1 to the position where the weight is 1.5 times.

[0219] On the other hand, the weight adjustment scroll bar 24-2 of the lower word weight setting section 22-2 is set at a position where the weight is -0.5 times.

[0220] in this case, The weight of "people" was set to 1.5 times, and The weight of "running" was set to -0.5.

[0221] When such a weighted query is input, the information processing device of this disclosure performs data search processing, wherein the weight of "people" is set to 1.5 times the weight of the other three words "on", "the" and "beach", and the weight of "running" is set to -0.5 times the weight of the other three words "on", "the" and "beach".

[0222] That is, perform data search processing, where the influence of "people" is set to 1.5 times the influence of the three unweighted words, and the influence of "running" is set to -0.5 times the influence of the three unweighted words.

[0223] Specifically, the information processing device of this disclosure generates a "weighted query corresponding feature vector" in which the influence of "people" is set to 1.5 times the influence of the three unweighted words and the influence of "running" is set to -0.5 times the influence of the three unweighted words, and performs similarity determination processing between the generated "weighted query corresponding feature vector" and the content corresponding feature vector of the image, etc.

[0224] In the similarity determination process, images with high similarity are output as search results.

[0225] As a result, the search results showed that "people" had a high influence and "running" had a low influence.

[0226] Note that, although Figure 5 The example shown illustrates setting two word weight setting units 22, but the word weight setting units 22 can be sequentially increased and displayed until they are equal to the number of words constituting the query input to the query input area 11c.

[0227] That is, users can set individual weights for all words that make up the query input into the query input area 11c.

[0228] (2-(d) Configuration of query input area and query input example that can use color display (heatmap) to set weights)

[0229] Next, we will describe the configuration of the query input area that can use color display (heatmap) to set weights and provide a query input example.

[0230] Similar to the above Figures 2 to 5 , Figure 6 This is a diagram illustrating the configuration of the query input area and a query input example in the data search UI displayed on the display unit of the information processing device of this disclosure.

[0231] Figure 6 Examples of input for two types of queries (search statements) are shown: (1) Normal search query; and (d1) Example 1 of a weighted search query using color display (heatmap).

[0232] The example shown in "(1) Normal Search Query" is a traditional query, that is, query="people running on the beach", The query input field 11 in the data search UI is entered, as shown above. Figure 1 As described.

[0233] "(d1) Example 1 of weighted search query using color display (heatmap)" is an example of the configuration of the query input area 11d in the data search UI displayed on the display unit of the information processing device of this disclosure and the query entered into the query input area 11d.

[0234] In query input area 11d, you can enter a query that sets a "weight" for each of the constituent words of the query.

[0235] The attached figure shows an example of entering the following query.

[0236] query="people running on the beach"

[0237] This is an example of query input.

[0238] In addition, a dark color panel is overlaid in the text area for "people".

[0239] In addition, a light-colored color panel is overlaid in the "running" text area.

[0240] In addition, a color panel displaying the intermediate color is overlaid in the text area of ​​"beach".

[0241] These color panels are user-defined color panels, and as shown in Figure 6 As shown in the table on the left, the panel colors correspond to different weights.

[0242] like Figure 6 As shown in the table on the left, when the panel color is light, the weight is 1.0 times. The weight value increases as the panel color becomes darker, and for the darkest panel, the weight is 3.0 times.

[0243] Note that the correspondence between the color panels and weights shown in this diagram is an example, and various other settings are possible. Furthermore, users are free to configure the correspondence between the color panels and weights.

[0244] exist Figure 6 In the example shown, A dark color panel (weight = 3) is overlaid on the text area of ​​"people", and the weight of "people" is 3.

[0245] A light color panel (weight = 1.5x) is overlaid on the text area of ​​"running", with "running" having a weight of 1.5x.

[0246] The color panel of the middle color (weight = twice) is superimposed on the text area of ​​"beach", and the weight of "beach" is twice.

[0247] When a weighted query is input, the information processing device of this disclosure performs data search processing, wherein the weight of "people" is set to 3 times, the weight of "running" is set to 1.5 times, and the weight of "beach" is set to twice.

[0248] Note that the multiple is a relative multiple of the unweighted words "on" and "the".

[0249] Specifically, the information processing device of this disclosure generates a "weighted query corresponding feature vector" with an influence (weight) relative to unweighted words set to 3 times for "people", 1.5 times for "running", and twice for "beach", and performs similarity determination processing between the generated "weighted query corresponding feature vector" and the content corresponding feature vector of an image, etc.

[0250] In the similarity determination process, images with high similarity are output as search results.

[0251] As a result, it is possible to obtain search results with high impact for "people", "running", and "beach".

[0252] The above has been referenced Figures 2 to 6 Several configuration examples of a query input area in a data search user interface (UI) displayed on the display unit of the information processing device of this disclosure are described.

[0253] That is, the configuration is as follows.

[0254] (a) Configuration of query input area that allows weight settings based on numerical input

[0255] (b) Configuration of query input area that allows weight settings based on parentheses (round brackets) input

[0256] (c) Configuration of query input area that allows weight settings using scrollbars

[0257] (d) Configuration of query input areas that can use color display (heatmap) for weight setting

[0258] The information processing apparatus of this disclosure displays at least one of query input areas (a) to (d) on a data search user interface (UI).

[0259] Note that multiple query input areas (a) through (d) can be displayed together, and users can select and use the most user-friendly query input areas.

[0260] [3. A specific example of data search processing using weighted queries]

[0261] The following section will describe a specific example of data search processing using weighted queries.

[0262] For reference Figures 2 to 6 As described, the information processing apparatus of this disclosure displays a data search user interface (UI) on a display unit having a query input area to which weighted queries can be entered.

[0263] Users can perform data searches while setting various weights for the words used to construct the query.

[0264] The following section describes a specific example of data search processing performed using weighted queries.

[0265] Reference Figure 7 Specific examples of the following processes are described sequentially with the accompanying figures.

[0266] (1) A specific example of data search processing using weighted queries based on numerical input.

[0267] (2) Specific examples of data search processing using weighted queries with scrollbars

[0268] (3) Specific examples of data search processing using queries in Japanese text

[0269] (4) Specific examples of data search processing using music data as the search target

[0270] (3-(1) Specific examples of data search processing using weighted queries based on numerical input)

[0271] First, a specific example of data search processing using weighted queries based on numerical inputs will be described.

[0272] Figure 7 This is used to describe the use of the above references. Figure 2 A diagram illustrating a specific example of the data search processing described in “(a) enables the setting of weights for the query input region 11a based on numerical inputs”.

[0273] Figure 7 This example demonstrates data search processing that uses three different weight settings for the query, generated by assigning three different weights to the word "smiling" that constitutes the following query. The query is for "a photo of a smiling face".

[0274] Specifically, an example of data search using the following three weight settings is shown.

[0275] (1) The weight of “smiling” is set to 1.0 times the query weight.

[0276] (2) The weight of “smiling” was set to 0.4 times the query weight.

[0277] (3) The weight of “smiling” is set to -0.2 times the query weight.

[0278] Note that in any of (1) to (3) above, the entire text of the query initially entered into query input area 11a is

[0279] query=a photo of a smiling face, That is, the above query.

[0280] Figure 7 (1) is an example where the user enters a value of 1.0 to indicate the weight of the word “smiling” included in the query after entering the query, and changes the query to the following weighted query.

[0281] Weighted query = a photo of a (smiling:1.0) face

[0282] By changing the processing through this query, the weight of the word "smiling" was set to 1.0.

[0283] Figure 7 (2) is an example of a user entering a value of 0.4 to indicate the weight of the word “smiling” in the query after entering the query, and changing the query to the following weighted query.

[0284] Weighted query = a photo of a (smiling:0.4) face

[0285] By changing the processing through this query, the weight of the word "smiling" was set to 0.4.

[0286] Figure 7 (3) is an example where the user enters a value of -0.2 indicating the weight of the word "smiling" included in the query after entering the query, and changes the query to the following weighted query.

[0287] Weighted query = a photo of a (smiling: -0.2) face

[0288] By changing the processing through this query, the weight of the word "smiling" was set to -0.2.

[0289] Figure 7 The right side shows the image data as the result of the search based on the weighted queries through (1) to (3).

[0290] These search results are displayed in the above reference. Figure 1 The search results are displayed in area 13 of the data search UI on the display unit of the described information processing device.

[0291] exist Figure 7 The upper part shows the weighted query of (1) = a photo of a (smiling:1.0) face

[0292] In the data search processing using weighted queries, the weight for the word "smiling" was set to 1.0. As a result, facial image data with a high degree of "smiling" (i.e., a smile) was output as the search results.

[0293] In addition, Figure 7 The middle part shows the weighted query of (2) = a photo of a (smiling:0.4) face

[0294] In the data search processing using weighted queries, the weight for the word "smiling" was set to 0.4 times. As a result, facial image data with a relatively low degree of "smiling" (i.e., a smile) were output as search results.

[0295] In addition, Figure 7 The lower part shows the weighted query of (3) = a photo of a (smiling: -0.2) face

[0296] In the data search processing using weighted queries, the weight for the word "smiling" was set to -0.2. As a result, facial image data with a low degree of "smiling" (i.e., a smile) were output as search results.

[0297] As described above, the information processing device of this disclosure generates a "weighted query corresponding feature vector" based on a weighted query, and performs similarity determination processing between the generated "weighted query corresponding feature vector" and the content corresponding feature vector of an image, etc.

[0298] In the similarity determination process, images with high similarity are output as search results.

[0299] As a result, search processing is performed that reflects the weights set for each word in the weighted query.

[0300] Figure 7 The example shown illustrates the results of performing a data search by setting the weight of the word "smiling" in a weighted query to three values: 1.0, 0.4, and -0.2.

[0301] If it is possible to Figure 7 The data search results shown are understood and can perform data searches based on the weight set by the user for the word "smiling" in the query, in order to selectively retrieve images, i.e., images with different degrees of smiles.

[0302] Figure 8 This is used to describe the use of the above references. Figure 2 A diagram illustrating a specific example of the data search processing described in “(a) enables the setting of weights for the query input region 11a based on numerical inputs”.

[0303] Figure 8 This example demonstrates data search processing using three different weight settings for the query, generated by assigning three different weights to the two words "sunny" and "foggy" that constitute the following query. The query is: Trees stand quietly with flowers. It is sunny and foggy. Specifically, an example of data search using the following three weight settings is shown.

[0304] (1) Unweighted query (= Queries where the weights of “sunny” and “foggy” are set to 1.0)

[0305] (2) Queries where the weight of "sunny" is set to 1.5 and the weight of "foggy" is set to -0.5.

[0306] (3) Queries where the weight of “sunny” is set to 1.2 and the weight of “foggy” is set to 2.0.

[0307] Note that in any of (1) to (3) above, the entire text of the query initially entered into query input area 11a is

[0308] Query=Trees stand quietly with flowers.

[0309] It is sunny and foggy.

[0310] Figure 8 (1) shows an example of a search performed without setting the weight of the words included in the query after the user enters the query.

[0311] also, Figure 8 (2) is an example where the user enters a value of 1.5 indicating the weight of the word "sunny" and a value of -0.5 indicating the weight of the word "foggy" included in the query after entering the query, and changes the query to the following weighted query.

[0312] Weighted query=Trees stand quietly with flowers. It is (sunny:1.5) and(foggy:-0.5).

[0313] By changing the processing through this query, the weight of the word "sunny" was set to 1.5 times, and the weight of the word "foggy" was set to -0.5 times.

[0314] also, Figure 8 (3) is an example of a user entering a weighted query after inputting a value of 1.2 indicating the weight of the word "sunny" and a value of 2.0 indicating the weight of the word "foggy" included in the query, and changing the query to the following weighted query.

[0315] Weighted query=Trees stand quietly with flowers. It is (sunny:1.2) and(foggy:2.0).

[0316] By changing the processing based on this query, the weight of the word "sunny" was set to 1.2 times, and the weight of the word "foggy" was set to 2.0 times.

[0317] Figure 8 The right side shows the image data as the result of the search based on the weighted queries through (1) to (3).

[0318] These search results are displayed in the above reference. Figure 1 The search results are displayed in area 13 of the data search UI on the display unit of the described information processing device.

[0319] exist Figure 8 The query (1) shown above, that is, Query=Trees stand quietly with flowers. It is sunny and foggy. In the data search processing using this unweighted query, the weights of all words constituting the query are set to the same (1.0 times). As a result, the output words "sunny" and "foggy" are represented equally by image data as search results.

[0320] In addition, Figure 8 The weighted query of (2) shown in the middle is, i.e. Weighted query=Trees stand quietly with flowers. It is (sunny:1.5) and(foggy:-0.5). In the data search processing using this weighted query, the weight of the word "sunny" was set to 1.5 times and the weight of the word "foggy" was set to -0.5 times.

[0321] As a result of this search process, image data with a high degree of "sunny" and a low degree of "foggy" are output as search results.

[0322] In addition, Figure 8 The lower part shows the weighted query of (3), that is, Weighted query=Trees stand quietly with flowers. It is (sunny:1.2) and(foggy:2.0). In the data search processing using this weighted query, the weight of the word "sunny" was set to 1.2 times and the weight of the word "foggy" was set to 2.0 times.

[0323] As a result of this search process, image data with a relatively low degree of "sunny" and a high degree of "foggy" are output as search results.

[0324] As described above, the information processing device of this disclosure generates a "weighted query corresponding feature vector" based on a weighted query, and performs similarity determination processing between the generated "weighted query corresponding feature vector" and the content corresponding feature vector of an image, etc.

[0325] In the similarity determination process, images with high similarity are output as search results.

[0326] As a result, search processing is performed that reflects the weights set for each word in the weighted query.

[0327] exist Figure 8In the example shown in (2), a "weighted query corresponding feature vector" is generated based on the weight setting information that the weight of the word "sunny" is 1.5 times and the weight of the word "foggy" is -0.5 times, and the similarity determination processing between the generated "weighted query corresponding feature vector" and the content corresponding feature vector of the image, etc. is performed.

[0328] As a result, image data with a high degree of "sunny" and a low degree of "foggy" was obtained as search results.

[0329] exist Figure 8 In the example shown in (3), a "weighted query corresponding feature vector" is generated based on the weight setting information that the weight of the word "sunny" is 1.2 times and the weight of the word "foggy" is 2.0 times, and the similarity determination processing between the generated "weighted query corresponding feature vector" and the content corresponding feature vector of the image, etc. is performed.

[0330] In the similarity determination process, images with high similarity are output as search results.

[0331] As a result, image data with a relatively low degree of "sunny" and a high degree of "foggy" were output as search results.

[0332] (3-(2) Specific examples of data search processing using weighted queries with scrollbars)

[0333] Next, we will describe a specific example of data search processing using a weighted query with a scrollbar.

[0334] Figure 9 This shows the use of the above reference. Figure 4 and Figure 5 A diagram illustrating a specific example of the data search process described in "(c) query input region 11c, enabling the use of scrollbars to set weights".

[0335] like Figure 9 As shown, the query input area 11c includes a query input section 21 and a word weight setting section 22.

[0336] The word weight setting section 22 displays the word weight setting input section 23 and the weight adjustment scroll bar 24.

[0337] The following query, namely, The query is "A bear with a ribbon 3D style painting". The query input section 21 is input into the query input area 11c.

[0338] Furthermore, when “3D” is input into the word weight setting unit 22, the state of the weight adjustment scroll bar 24 of the word weight setting unit 22 is displayed, indicating that the user is adjusting the weight adjustment scroll bar 24 of the word weight setting unit 22.

[0339] In the example shown in the attached figure, the weight setting range of the weight adjustment scroll bar 24 is -1.0 to 2.0.

[0340] Users can adjust the weight of the word input unit 23 by moving the weight adjustment scroll bar 24 left and right, within a range of -1.0x to 2.0x.

[0341] Figure 9 The lower part shows an example of the search results output by adjusting the weight of scroll bar 24 by moving it left and right.

[0342] Note that the actual search results are combinations of different bear images, but for illustration purposes, one type of bear image is shown as a sample here.

[0343] For example, when the user moves the weight adjustment scroll bar 24 to the left, data search processing is performed, wherein the weight of “3D” input to the weight setting word input unit 23 of the word weight setting unit 22 is set to -1.0 times.

[0344] That is, data search processing is performed, wherein the weight of the word “3D” in the query (i.e., query = A bear with a ribbon 3D style painting) input into the query input section 21 in the query input area 11c is set to -1.0 times.

[0345] The result is as follows: Figure 9 The lower left end shows images of bears with low 3D detail (i.e., image data of bears that are essentially 2D images) as search results.

[0346] On the other hand, when the user moves the weight adjustment scroll bar 24 to the right, data search processing is performed, wherein the weight of “3D” in the weight setting word input unit 23 input to the word weight setting unit 22 is set to 2.0 times.

[0347] That is, data search processing is performed, wherein the weight of the word “3D” in the query (i.e., query = A bear with a ribbon 3D style painting) input into the query input section 21 in the query input area 11c is set to 2.0 times.

[0348] The result is as follows: Figure 9The lower right end shows a highly detailed 3D bear image (i.e., image data of a bear that is essentially a 3D image) as the search result.

[0349] Users can adjust the weight of the word input unit 23 by moving the scroll bar 24 left and right, within a range of -1.0x to 2.0x, and can obtain image data with a 3D degree corresponding to each weight as search results.

[0350] It should be noted that, in the same example, the information processing device of this disclosure generates a "weighted query corresponding feature vector" based on a weighted query, and performs similarity determination processing between the generated "weighted query corresponding feature vector" and the content corresponding feature vector of an image, etc.

[0351] exist Figure 9 In the example shown, the weight of the word "3D" is changed from -1.0 to 2.0 to generate a "weighted query corresponding feature vector" for each weight. As the result of the similarity determination between the generated "weighted query corresponding feature vector" and the content corresponding feature vector of the image, the image with high similarity is output as the search result.

[0352] Therefore, it is possible to obtain images of different 3D levels (i.e., image data from 2D to 3D) as search results based on the weight of "3D".

[0353] Similar to Figure 9 , Figure 10 This is used to describe the use of the above references. Figure 4 and Figure 5 A diagram illustrating a specific example of the data search process described in "(c) query input region 11c, enabling the use of scrollbars to set weights".

[0354] like Figure 10 As shown, the query input area 11c includes a query input section 21 and a word weight setting section 22.

[0355] The word weight setting section 22 displays the word weight setting input section 23 and the weight adjustment scroll bar 24.

[0356] In the query input section 21 of the query input area 11c, enter the following query, namely: The query is "A photo of crowded spaces".

[0357] enter.

[0358] Furthermore, when “crowded” is input into the word weight setting unit 22’s weight setting word input unit 23, it shows the state of the weight adjustment scroll bar 24 of the word weight setting unit 22 being adjusted by the user.

[0359] In the example shown in the attached figure, the weight setting range of the weight adjustment scroll bar 24 is -1.0 to 2.0.

[0360] Users can adjust the weight of the word input in the word input unit 23 by moving the scroll bar 24 left and right, within a range of -1.0x to 2.0x.

[0361] Figure 10 The lower part shows an example of the search results output by adjusting the weight of scroll bar 24 by moving it left and right.

[0362] For example, when the user moves the weight adjustment scroll bar 24 to the left, data search processing is performed, wherein the weight of “crowded” input to the weight setting word input unit 23 of the word weight setting unit 22 is set to -1.0 times.

[0363] That is, data search processing is performed, wherein the weight of the word “crowded” in the query (i.e., query = A photo of crowded spaces) input into the query input section 21 in the query input area 11c is set to -1.0 times.

[0364] The result is as follows: Figure 10 The lower left end shows image data with low congestion as the search result.

[0365] On the other hand, when the user moves the weight adjustment scroll bar 24 to the right, data search processing is performed, wherein the weight of “crowded” input to the weight setting word input unit 23 of the word weight setting unit 22 is set to 2.0 times.

[0366] That is, data search processing is performed, wherein the weight of the word “crowded” in the query (i.e., query = A photo of crowded spaces) input into the query input section 21 in the query input area 11c is set to 2.0 times.

[0367] The result is as follows: Figure 10 The image data with a high degree of congestion shown at the lower right end is used as the search result.

[0368] By moving the weight adjustment scroll bar 24 left and right, the user can adjust the weight of "crowded" in the word input unit 23 within a range of -1.0x to 2.0x, and can obtain image data of the crowding level corresponding to each weight as search results.

[0369] It should be noted that, in the same example, the information processing device of this disclosure generates a "weighted query corresponding feature vector" based on a weighted query, and performs similarity determination processing between the generated "weighted query corresponding feature vector" and the content corresponding feature vector of an image, etc.

[0370] exist Figure 10 In the example shown, the weight of the word "crowded" is changed from -1.0 to 2.0 to generate a "weighted query corresponding feature vector" for each weight. As the result of the similarity determination between the generated "weighted query corresponding feature vector" and the content corresponding feature vector of the image, the image with high similarity is output as the search result.

[0371] Therefore, it is possible to obtain image data with different levels of crowding corresponding to the weight of "crowded" as search results.

[0372] (3-(3) Specific examples of data search processing using queries in languages ​​other than English)

[0373] The following section will describe specific examples of data search processing for queries in languages ​​other than English.

[0374] Although reference Figures 7 to 10 The specific example of the search processing described is using English sentences as search queries, but search queries or weighted queries can also be generated using languages ​​other than English to perform data searches.

[0375] Figure 11 This is a diagram illustrating a specific example of data search processing using queries based on Japanese text.

[0376] Figure 11 This is a diagram used to describe a processing example where a query generated using Japanese text is applied to perform the process described above. Figure 7 The described data search process is the same as the search process, and Figure 11 Using the above reference Figure 2 A specific example of the data search processing described in “(a) query input region 11a that can set weights based on numerical input”.

[0377] Figure 11This example demonstrates data search processing that uses three different weight settings for the query, generated by assigning three different weights to the word "smiling" that constitutes the following query. query=a photo of a smiling face.

[0378] Specifically, an example of data search using the following three weight settings is shown.

[0379] (1) The weight of “smiling” is set to 1.0 times the query weight.

[0380] (2) The weight of “smiling” was set to 0.4 times the query weight.

[0381] (3) The weight of “smiling” is set to -0.2 times the query weight.

[0382] Note that in any of (1) to (3) above, the entire text of the query initially entered into query input area 11a is: query=a photo of a smiling face.

[0383] Figure 11 (1) is an example where the user enters a value of 1.0 to indicate the weight of the word “smiling” included in the query after entering the query, and changes the query to the following weighted query.

[0384] Weighted query = a photo of a (smiling:1.0) face

[0385] By changing the processing through this query, the weight of the word "smiling" was set to 1.0.

[0386] Figure 11 (2) is an example of a user entering a value of 0.4 to indicate the weight of the word “smiling” in the query after entering the query, and changing the query to the following weighted query.

[0387] Weighted query = a photo of a (smiling:0.4) face

[0388] By changing the processing through this query, the weight of the word "smiling" was set to 0.4.

[0389] Figure 11 (3) is an example where the user enters a value of -0.2 indicating the weight of the word "smiling" included in the query after entering the query, and changes the query to the following weighted query.

[0390] Weighted query = a photo of a (smiling: -0.2) face

[0391] By changing the processing through this query, the weight of the word "smiling" was set to -0.2.

[0392] Figure 11 The right side shows the image data as the result of the search based on the weighted queries through (1) to (3).

[0393] These search results are displayed in the above reference. Figure 1 The search results are displayed in area 13 of the data search UI on the display unit of the described information processing device.

[0394] exist Figure 11 The weighted query shown at the top (1) = a photo of a (smiling:1.0) face

[0395] In the data search processing using weighted queries, the weight for the word "smiling" was set to 1.0. As a result, facial image data with a high degree of "smiling" (i.e., a smile) was output as the search results.

[0396] In addition, Figure 11 The weighted query shown in the middle of the table (2) = a photo of a (smiling:0.4) face

[0397] In the data search processing using weighted queries, the weight for the word "smiling" was set to 0.4 times. As a result, facial image data with a relatively low degree of "smiling" (i.e., a smile) were output as search results.

[0398] In addition, Figure 11 The lower part shows the weighted query (3) = a photo of a (smiling: -0.2) face

[0399] In the data search processing using weighted queries, the weight for the word "smiling" was set to -0.2. As a result, facial image data with a low degree of "smiling" (i.e., a smile) were output as search results.

[0400] As described above, the information processing device of this disclosure generates a "weighted query corresponding feature vector" based on a weighted query, and performs similarity determination processing between the generated "weighted query corresponding feature vector" and the content corresponding feature vector of an image, etc.

[0401] In the similarity determination process, images with high similarity are output as search results.

[0402] As a result, search processing is performed that reflects the weights set for each word in the weighted query.

[0403] Figure 11 The example shown illustrates the results of performing a data search by setting the weight of the word "smiling" in a weighted query to three values: 1.0, 0.4, and -0.2.

[0404] As Figure 11 The data search results from (1) to (3) output images corresponding to the weights set by the user for the word "smiling" in the query, i.e., images with different degrees of smile.

[0405] (3-(4) Specific examples of data search processing using music data as the search target)

[0406] Next, we will describe a specific example of data search processing where the search target is music data.

[0407] In the specific examples described above, an example of search processing using image data as the data search target has been described. However, in the information processing apparatus of this disclosure, the category of search target data is not limited to images. For example, data searches can be performed not only on images, but also on various types of data such as documents, music, buildings, nature, and organisms that are search targets.

[0408] Reference Figure 12 This describes an example of search processing when music is set as the search target.

[0409] Figure 12 This is to illustrate the use of the above references. Figure 2 The diagram illustrates a specific example of the data search processing described in "(a) query input area 11a which can be weighted based on numerical input", and an example of the search processing for music as the search target.

[0410] Figure 12 This example demonstrates data search processing that uses three different weight settings for the query, generated by assigning three different weights to the two words "bright" and "lively" that constitute the following query. The query is "bright and lively music".

[0411] Specifically, an example of data search using the following three weight settings is shown.

[0412] (1) The weight of “bright” was set to 1.5 times the query weight.

[0413] (2) Queries where the weight of "bright" is set to 2.0 and the weight of "lively" is set to 2.0.

[0414] (3) Queries where the weight of "bright" is set to 1.0 and the weight of "lively" is set to -1.0.

[0415] Note that in any of (1) to (3) above, the entire text of the query initially entered into query input area 11a is: query=bright and lively music.

[0416] Figure 12 (1) is an example where the user enters a value of 1.5 to indicate the weight of the word "bright" included in the query after entering the query, and changes the query to the following weighted query.

[0417] Weighted query = (bright:1.5) and lively music

[0418] By changing the processing through this query, the weight of the word "bright" was set to 1.5 times.

[0419] also, Figure 12 (2) is an example of a user entering a value of 2.0 indicating the weight of the word "bright" and a value of 2.0 indicating the weight of the word "lively" included in the query after entering the query, and changing the query to the following weighted query.

[0420] Weighted query = (bright:2.0) and (lively:2.0) music

[0421] By changing the processing through this query, the weight of the word "bright" was set to 2.0 times, and the weight of the word "lively" was also set to 2.0 times.

[0422] also, Figure 12 (3) is an example of a user entering a value of 1.2 indicating the weight of the word "bright" and a value of -1.0 indicating the weight of the word "lively" included in the query after entering the query, and changing the query to the following weighted query.

[0423] Weighted query = (bright:1.2) and (lively:-1.0) music

[0424] By changing the processing through this query, the weight of the word "bright" was set to 1.2 times, and the weight of the word "lively" was also set to -1.0 times.

[0425] Figure 12 The right side shows the image data as the result of the search based on the weighted queries through (1) to (3).

[0426] These search results are displayed in the above reference. Figure 1 The search results are displayed in area 13 of the data search UI on the display unit of the described information processing device.

[0427] exist Figure 12 The upper part shows the weighted query (1) = (bright:1.5) and lively music.

[0428] In the data search processing using weighted queries, the word "bright" was given a weight of 1.5. As a result, music data with high brightness was output as the search results.

[0429] exist Figure 12 The weighted query (2) shown in the middle of the string is (bright:2.0) and (lively:2.0) music.

[0430] In the data search processing using weighted queries, the weight of the word "bright" was set to 2.0 times, and the weight of the word "lively" was also set to 2.0 times. As a result, music data with high brightness and high activity was output as the search results.

[0431] In addition, Figure 12 The lower part shows the weighted query (3) = (bright:1.2) and (lively:-1.0)music

[0432] In the data search processing using weighted queries, the weight of the word "bright" was set to 1.2 times and the weight of the word "lively" was set to -1.0 times. As a result, music data with high brightness and low activity was output as the search results.

[0433] As described above, the information processing device of this disclosure generates a "weighted query corresponding feature vector" based on a weighted query, and performs similarity determination processing between the generated "weighted query corresponding feature vector" and the content corresponding feature vector of an image, etc.

[0434] In the similarity determination process, images with high similarity are output as search results.

[0435] As a result, search processing is performed that reflects the weights set for each word in the weighted query.

[0436] Figure 12 The example shown illustrates the results of performing a data search while changing the weights of the two words "bright" and "lively" in a weighted query.

[0437] As Figure 12 The data search results from (1) to (3) output music corresponding to the weights set by the user for the words "bright" and "lively" in the query, i.e. music with different brightness or activity levels.

[0438] [4. Details of the configuration and processing of the information processing equipment disclosed herein]

[0439] The following will describe the details of the configuration and processing of the information processing device disclosed herein.

[0440] The information processing apparatus disclosed herein is a device capable of performing data search processing, and is, for example, a device such as a PC, smartphone, or tablet terminal.

[0441] It should be noted that the information processing device disclosed herein may be configured as a single device or as multiple devices.

[0442] Figure 13 An example configuration of the information processing device 100 of this disclosure is shown.

[0443] It should be noted that Figure 13 The configuration diagram shown does not show the overall configuration of an information processing device such as a PC, but rather the configuration diagram of the main components used for the processing in this disclosure.

[0444] will describe Figure 13 The configuration of the information processing device 100 shown in the figure.

[0445] like Figure 13 As shown, the information processing device 100 includes a user interface (UI) 101, a data processing unit 102, and a communication unit 103.

[0446] The user interface (UI) 101 includes an input unit 111 and an output unit 112. The output unit 112 includes, for example, outputting display data (such as the data mentioned above). Figure 1 The data search UI described herein is a display unit. The input unit 111 includes at least one of an input device comprising a touch panel, a keyboard, and a mouse.

[0447] Users can input queries (search statements) and weight information in units of words constituting the query via the input unit 111 of the user interface (UI) 101.

[0448] The above references can be used Figures 2 to 12 The various types of query input areas 11a to 11c are described for inputting queries (search statements) and weight information in units of words that constitute the query.

[0449] like Figure 13 As shown, this input information is input from input unit 111 to data processing unit 102. The input information is... Figure 13 The query (search statement) and weight information are shown in the image.

[0450] The data processing unit 102 receives the query (search statement) and weight information, which are input by the input unit 111 via the user interface (UI) 101, in units of words constituting the query, and performs data search processing using these input information.

[0451] The data that can be searched is, for example, data on the network, such as data stored in databases 200-1 to n that are accessible via communication unit 103.

[0452] The data processing unit 102 performs content search processing, such as searching for images, based on user-input query and weight information, accessible via the communication unit 102. The search results generated during the search processing are output to the output unit of the user interface (UI) 101. The search results are... Figure 13 The search results shown are in the image.

[0453] Search results are displayed, for example, in the reference above. Figure 1 The search results for the data search UI described are displayed in area 13.

[0454] When performing data search processing, the data processing unit 102 generates a "weighted query corresponding feature vector" based on the query and weight information input by the user, and performs similarity determination processing between the generated "weighted query corresponding feature vector" and the feature vector generated corresponding to content such as an image accessible via the communication unit 102.

[0455] In the similarity determination process, images with high similarity are the search results.

[0456] Reference Figure 14 The detailed configuration of the data processing unit 102 and the details of the processing to be performed are described.

[0457] like Figure 14 As shown, the data processing unit 102 includes a text encoder 121, a weighted query corresponding feature vector generation unit 122, and a feature vector similarity analysis unit 123.

[0458] The text encoder 121 receives a query (search statement) input via the input unit 111 and performs encoding processing using a learning model.

[0459] As a specific example of a text encoder 121, it is possible to use a Transformer, which is a deep learning model (Deep Neural Network) that uses a focus mechanism to analyze the correlation between words.

[0460] Reference Figure 15 This section describes an outline of a deep learning model that uses a attention mechanism that can be used as a text encoder 121.

[0461] The attention mechanism is a mechanism that analyzes the relevance between each word that makes up a query and other words.

[0462] Figure 15 Examples of the following two attention mechanisms are shown.

[0463] (a) One-way focus (Transformer)

[0464] (b) Two-way attention (BERT)

[0465] When analyzing the relevance of each word that makes up a query (search statement) to other words, "(a) One-way focus" analyzes the relevance of that word only to the words that follow it. That is, the relevance analysis between words is performed only in one direction, from beginning to end.

[0466] The “(a) one-way attention” is used as a Transformer, which is a deep neural network.

[0467] When analyzing the relevance of each word that constitutes a query (search statement) to other words, "(b) Two-way focus" analyzes the relevance of that word not only to the words following it, but also to the words preceding it. That is, it analyzes the two-way relevance between words, not only from the previous word to the next word, but also from the next word to the previous word.

[0468] The “(b) Bidirectional Attention” is used as the Bidirectional Encoder Representation (BERT) from the Transformer, which is a derivative of the Transformer.

[0469] Note that this is an example of a query (search statement) to be analyzed. Figure 15 Examples (a) and (b) shown illustrate the following queries (i.e., The query (search statement) equals "dog running". enter.

[0470] Note that [SOS] indicates the beginning of a sentence, [EOS] indicates the end of a sentence, and [PAD] indicates padding data used for adjusting the data length.

[0471] In either case of "(a) one-way attention" or "(b) two-way attention", Five elements, namely [SOS], dog, running, [EOS], and [PAD], are used as input elements, and processing is performed individually on each of the five elements from [SOS] to [PAD].

[0472] That is, each processing object element is input into a deep learning model that uses an attention mechanism, and as output, the output is a feature vector indicating the features of the query (search statement), namely the "query-corresponding feature vector".

[0473] It should be noted that in the following text, elements such as words used as input and processing units of deep learning models with attention mechanisms will be referred to as “tokens”.

[0474] exist Figure 15 In the examples (a) and (b) shown, each of the five elements [SOS], dog, running, [EOS], and [PAD] is a token.

[0475] like Figure 15 As shown, in “(a) One-way focus”, the “query-corresponding feature vector” representing the features of the query (search statement) as output is output from the output unit of the token [EOS].

[0476] In “(a) One-way focus”, the “query corresponding feature vector” output from the output unit of token [EOS] is generated as a feature vector that includes the analysis results for tokens located before token [EOS] (i.e., words such as [SOS], dog, and running), that is, the analysis results of the one-way (from front to back) token relevance for all tokens [SOS], dog, running, and [EOS].

[0477] On the other hand, such as Figure 15 As shown, in “(b) Two-way attention”, firstly, the [token-corresponding vectors] output individually from the output units of all tokens [SOS], dog, running, [EOS], and [PAD] are obtained, and arithmetic processing (e.g., average vector calculation processing) is performed on multiple [token-corresponding vectors] corresponding to all tokens, and the “query-corresponding feature vector” indicating the features of the query (search statement) as the final output is output.

[0478] Note that, similarly in "(a) One-way attention", internally, as in "(b) Two-way attention", the computation of the "token correspondence vector" corresponding to all tokens [SOS], dog, running, [EOS], and [PAD] is performed. In "(a) One-way attention", only the "token correspondence vector" output from the output unit of [EOS] is selected from all "token correspondence vectors" and then selected and output as the "query correspondence feature vector".

[0479] The "Query Corresponding Feature Vector" output from each of "(a) One-way Focus" and "(b) Two-way Focus" is a multi-dimensional vector indicating the features of the query (search).

[0480] However, since the direction of the correlation analysis between words is different in (a) and (b), slightly different vectors are produced.

[0481] exist Figure 14 The text encoder 121 in the data processing unit 102 of the information processing apparatus 100 of this disclosure shown in the figure is configured to use in Figure 15 The encoder for “(a) one-way attention” shown in (a).

[0482] This is used to set Figure 14 The calculation and processing of “weighted query corresponding feature vector 131” in the weighted query corresponding feature vector generation unit 122 in the later stage of the text encoder 121 shown.

[0483] exist Figure 14 The weighted query correspondence feature vector generation unit 122 in the later stage of the text encoder 121 shown calculates the difference between adjacent tokens in the "token correspondence vector" generated by the text encoder 121, and performs a process of multiplying the calculated difference by the weight corresponding to the word (token) set by the user, etc. to calculate the "weighted query correspondence feature vector 131".

[0484] exist Figure 15 In “(a) One-way attention” shown in (a), since the “token correspondence vector” corresponding to a token such as each word is calculated by one-way token correlation analysis, the difference between the “token correspondence vectors” of adjacent tokens is data that reflects the difference between the feature vectors of tokens (i.e., words).

[0485] However, in “(b) Two-way attention”, because the “token correspondence vector” corresponding to each token such as a word is calculated through two-way token correlation analysis, the difference in the “token correspondence vector” between adjacent tokens does not directly reflect the difference in the feature vectors between tokens (i.e., words).

[0486] Therefore, in Figure 14The text encoder 121 in the data processing unit 102 of the information processing apparatus 100 of this disclosure shown in the figure is configured as follows: Figure 15 The encoder of type “(a) one-way attention” shown in (a).

[0487] like Figure 14 As shown, the weighted query corresponding feature vector generation unit 122 is provided after the text encoder 121 in the data processing unit 102 of the information processing device 100 of this disclosure.

[0488] As described above, the weighted query corresponding feature vector generation unit 122 calculates the difference between adjacent tokens in the "token corresponding vector" generated by the text encoder 121, and performs a process of multiplying the calculated difference by the weight corresponding to the word (token) set by the user, etc., to calculate the "weighted query corresponding feature vector 131".

[0489] Reference Figure 16 A specific example of the processing performed by the weighted query corresponding feature vector generation unit 122 is described.

[0490] exist Figure 16 The diagram shown on the left is a diagram illustrating the detailed configuration of the processing performed by the text encoder 121 and the weighted query corresponding feature vector generation unit 122.

[0491] Figure 16 The right side shows the sequence of processes performed by the weighted query corresponding feature vector generation unit 122.

[0492] exist Figure 16 The lower left side shows the detailed configuration of the processing performed by the text encoder 121, and the upper part shows the detailed configuration of the processing performed by the weighted query corresponding feature vector generation unit 122.

[0493] Note that the query (search statement) to be analyzed is indicated as follows: Figure 15 Enter the following query as described (i.e., Query (search statement) = dog running, Example of handling the situation.

[0494] In addition, as referenced Figure 15 As described, each of the five elements [SOS], dog, running, [EOS], and [PAD] is set as a token for processing.

[0495] exist Figure 16 The text encoder 121 shown in the lower left part is a reference. Figure 15 (a) describes an encoder of the “(a) one-way attention” type.

[0496] As described above, the text encoder 121, which is an encoder of the "(a) one-way attention" type, performs the processing of calculating the "token correspondence vector" corresponding to all tokens of [SOS], dog, running, [EOS] and [PAD].

[0497] Figure 16 The upper left part shows the detailed configuration of the processing performed by the weighted query corresponding feature vector generation unit 122.

[0498] The weighted query corresponding feature vector generation unit 122 calculates the difference between adjacent tokens in the "token corresponding vector" generated by the text encoder 121, and performs a process of multiplying the calculated difference by the weight corresponding to the word (token) set by the user, etc., to calculate the "weighted query corresponding feature vector 131".

[0499] The weighted query corresponding feature vector generation unit 122 is executed sequentially in the following steps. Figure 16 The right side shows the processing of steps S01 and S02 to calculate the "weighted query corresponding feature vector 131".

[0500] The processes in steps S01 and S02 will be described.

[0501] (Step S01)

[0502] First, in step S01, the weighted query corresponding feature vector generation unit 122 performs the token corresponding vector difference (D) operation. X ) calculation processing.

[0503] The weighted query corresponding feature vector generation unit 122 calculates the difference between adjacent tokens in the "token correspondence vector" generated by the text encoder 121 as the token correspondence vector difference (D). X ).

[0504] That is, according to the following expression (Expression 1), the token-corresponding vector difference (D) is... X ) is calculated as the vector corresponding to adjacent tokens (V) X and V X The difference (Diff) between -1).

[0505] D x =V X -V X-1 ... (Expression 1)

[0506] exist Figure 16 In the example shown on the left, The token-to-vector difference (D1) corresponding to the token (dog) is calculated as the difference between the following two: The token-to-vector (V1) corresponding to the token (dog), and The token-to-vector (V0) corresponding to the token (SOS), that is, D1 = V1 - V0.

[0507] Furthermore, the token-corresponding vector difference (D2) corresponding to the token (running) is calculated as the difference between the token-corresponding vector (V2) corresponding to the token (running) and the token-corresponding vector (V1) corresponding to the token (dog), that is, D2 = V2 - V1.

[0508] Similarly, in the following text, the vectors (V) corresponding to adjacent tokens up to the token (EOS) are calculated sequentially. X and V X-1 The difference (Diff) between ).

[0509] If the number of tokens from SOS to EOS is k, Calculate the vector difference (D1 to D) between the k tokens. k ).

[0510] Note that it is possible to use the token-to-vector difference (D) X The following represents the vector (V0 to V) corresponding to each token (word). N ).

[0511] V1 = V0 + D1

[0512] V2 = V1 + D2 = V0 + D1 + D2 : V K =V K-1 +D K =V0+D1+...+D K Note that V K It is the feature vector corresponding to the query-corresponding feature vector (Vec), which is determined by the user Figure 15 (a) Encoder computation for a Transformer of the one-way attention type.

[0514] (Step S02)

[0515] Next, in step S02, the weighted query corresponding feature vector generation unit 122 calculates the "weighted query corresponding feature vector V'". K ".

[0516] The weight (scale) of each word (x) input by the user is set to s. x And calculate the weighted query corresponding feature vector V' according to the following expression (expression 2). K ".

[0517] V' K =V0+s1D1+...+s K D K ... (Expression 2)

[0518] However, V0 is the vector corresponding to the first token output by the encoder. s n It is the weight of the nth word in the constituent words of the query, and D n It is the difference between the token vector of the nth word in the query and the token vector of the (n-1)th word.

[0519] Note that (Expression 2) is the weight (s) corresponding to the second and subsequent tokens (x). x ) and the vector difference (D) corresponding to the token X All multiplication values ​​(s) X D X Arithmetic processing of the vector corresponding to the first token added to the element (token) of the processing object.

[0520] Through this arithmetic process, the "weighted query corresponding feature vector V'" is calculated. K ".

[0521] The weighted query corresponding feature vector V' is calculated in this way. K " is a feature vector that reflects the weights set by the user for each word.

[0522] For example, if the input query (search statement) is the following query, Query (search statement) = dog running, The word (dog) has a weight of 2.0, and The word (running) has a weight of 1.0. Based on the above (Expression 2), the weighted query corresponding feature vector V' is calculated. K "It is generated as a vector in which the word (dog) has a more important meaning than the word (running)."

[0523] In addition, for example, the above references Figure 2 The query (search statement) described, that is, The query (search statement) is: (people:2.0) running on the beach. With the word "people" weighted at 2.0 times and other words (running, on, the, and beach) weighted at 1.0 times, Based on the above (Expression 2), the weighted query corresponding feature vector V' is calculated. K "It is generated as a vector of the word (people) with a more important meaning than other words (running, on, the, and beach).

[0524] return Figure 14 The description of the configuration of the data processing unit 102 of the information processing device 100 of this disclosure will continue.

[0525] As described above, the weighted query corresponding feature vector generation unit 122 calculates the difference between adjacent tokens in the "token corresponding vector" generated by the text encoder 121, and performs a process of multiplying the calculated difference by the weight corresponding to the word (token) set by the user, etc., to calculate the "weighted query corresponding feature vector 131".

[0526] That is, the weight (scale) of each word (x) input by the user is set to s. x And calculate the weighted query corresponding feature vector V' according to the following expression (expression 2). K 131.

[0527] V' K =V0+s1D1+...+s K D K ... (Expression 2)

[0528] like Figure 14 As shown, the "weighted query corresponding feature vector 131" calculated by the weighted query corresponding feature vector generation unit 122 is input to the feature vector similarity analysis unit 123.

[0529] The feature vector similarity analysis unit 123 performs a similarity determination between the "weighted query corresponding feature vector 131" calculated by the weighted query corresponding feature vector generation unit 122 and the feature vector corresponding to the large amount of data included in the search target data (i.e., the content corresponding feature vector 132).

[0530] The target data to be searched is, for example, data in a network that can be accessed via the communication unit 103 of the information processing device 100, such as content like image data. Figure 14 The databases 200-1 to n shown store content such as image data.

[0531] Note that in databases 200-1 to n, content identifiers (image IDs, etc.) and feature vectors (content-corresponding feature vectors) corresponding to the features of the pre-analyzed content (images, etc.) are recorded in association with content such as image data.

[0532] Reference Figure 17 Example describing the data stored in databases 200-1 to n.

[0533] Figure 17 The example shown is an example of a database storing image data. Figure 17 The image data shown in the upper left part is data stored in database 200-x.

[0534] First, the management device of database 200-x sets an identifier (ID) for each image data.

[0535] Next, each image is input into the content encoder 211, and the content-corresponding feature vector (Vec.x) of the image features is calculated.

[0536] As a content encoder 211, a deep learning model (DeepNeural Network) is used to perform image analysis and compute feature vectors.

[0537] Specifically, similar to the Transformer mentioned above, image analysis can be performed using VisionTransformer (ViT), Swin Transformer, ResNet, etc., which utilize attention mechanisms.

[0538] Content-corresponding feature vectors (Vecx) indicating the features of an image computed by the content encoder 211 are stored in the database in association with the content (image, etc.) and the content ID (image ID).

[0539] Figure 14 The feature vector similarity analysis unit 123 of the data processing unit 102 of the information processing device 100 shown in the figure obtains the content-corresponding feature vectors 132-1 to n of the content (image) from the databases 200-1 to n in the network via the communication unit 103.

[0540] The feature vector similarity analysis unit 123 analyzes the similarity between the feature vectors 132-1 to n corresponding to the content and the "weighted query corresponding feature vector 131" calculated by the weighted query corresponding feature vector generation unit 122.

[0541] Note that similarity analysis between feature vectors is performed by calculating similarity indices such as cosine similarity or distance between vectors (Euclidean distance).

[0542] The feature vector similarity analysis unit 123 selects content (images) that have a high similarity to the "weighted query corresponding feature vector 131" calculated by the weighted query corresponding feature vector generation unit 122, and outputs the selected content to the output unit 112 as the search result 133.

[0543] Search result 133 was displayed Figure 1 The search results are displayed in area 13 of the data search UI shown in the image.

[0544] It should be noted that the feature vector similarity analysis unit 123 can perform a process that restricts the content (image) with high similarity that is output to the output unit 112 as the search result 133 to the content (image) with similarity equal to or higher than a preset threshold.

[0545] In addition, sorting processing can be performed to arrange the content (images) to be output to the output unit 112 as search results 133 in descending order of similarity, and processing can be performed to display the content in descending order of similarity.

[0546] In addition, such as Figure 18 As shown, a configuration can also be adopted in which a content encoder 124 is set inside the data processing unit 102 of the information processing device 100, and the processing of the image and other content input via the communication unit 103 is performed to calculate the feature vector corresponding to the content.

[0547] The content encoder 124 uses a deep learning model (deep neural network) to perform image analysis to compute feature vectors, similar to the reference above. Figure 17 The content described is encoder 211.

[0548] Specifically, similar to the Transformer mentioned above, image analysis can be performed using VisionTransformer (ViT), Swin Transformer, ResNet, etc., which utilize attention mechanisms.

[0549] [5. A sequence of processes performed by the information processing apparatus of this disclosure]

[0550] Next, the sequence of processes performed by the information processing apparatus of this disclosure will be described.

[0551] Reference Figure 19 and Figure 20 The flowchart shown describes a sequence of processes performed by the information processing device of this disclosure.

[0552] It should be noted that, according to the program stored in the storage unit of the information processing device of this disclosure, under the control of a data processing unit including a CPU with program execution capabilities, the program is executed according to... Figure 19 and Figure 20 The process shown in the flowchart.

[0553] In the following text, the sequence will be described in order. Figure 19 and Figure 20 The process shown in the diagram is the handling of each step.

[0554] (Step S101)

[0555] First, in step S101, the data processing unit 102 of the information processing device 100 of this disclosure activates the search processing execution screen (UI) and displays the screen on the display unit.

[0556] For example, in Figure 1 The data search UI shown is displayed on the display unit of the information processing device 100.

[0557] (Step S102)

[0558] Next, in step S102, the data processing unit 102 of the information processing device 100 of this disclosure receives a search query (search statement).

[0559] For example, the user will enter... Figure 1 The search query (search statement) in the query input area 11 of the data search UI shown in the figure is input into the data processing unit 102.

[0560] (Step S103)

[0561] Next, in step S103, the data processing unit 102 of the information processing device 100 of this disclosure determines whether there is a weight setting in the input search query (search statement).

[0562] That is, determining whether a search query (search statement) is a query that sets the weight of each constituent word.

[0563] Note that queries that set weights for each constituent word are, for example, referenced. Figure 2 The following queries are for any of the following weight settings described in the accompanying figures.

[0564] (a) Weight settings based on numerical input

[0565] (b) Weight settings based on parentheses (parentheses) input

[0566] (c) Using scrollbar weight settings

[0567] (d) Weight settings using color display (heatmap)

[0568] If the input query (search statement) is determined to be an unweighted query, the process proceeds to step S111.

[0569] On the other hand, if it is determined that the input query (search statement) is a weighted query, the process proceeds to step S121.

[0570] (Step S111)

[0571] If it is determined in step S103 that the input query (search statement) is an unweighted query, then the following processing is performed in step S111.

[0572] In step S111, the data processing unit 102 performs encoding processing of the input search query (search statement) and calculates the feature vector corresponding to the query.

[0573] This processing is handled by Figure 14 The weighted query corresponding feature vector generation unit 122 of the data processing unit 102 shown is executed.

[0574] It should be noted that references such as those above can be used. Figure 15 (a) describes a Transformer model of unidirectional focus type to perform the computation of the feature vector corresponding to the query.

[0575] That is, this is in Figure 15 The calculation process of "query corresponding feature vector (Vec)" shown in (a) is in Figure 14 The vector generated in the text encoder 121 of the data processing unit 102 shown in the figure.

[0576] exist Figure 14 The weighted query-corresponding feature vector generation unit 122 of the data processing unit 102 shown in the figure directly outputs the "query-corresponding feature vector (Vec)" input from the text encoder 121 as... Figure 14 The “weighted query corresponding feature vector 131” is shown in the figure.

[0577] However, in this case, “weighted query corresponding feature vector 131” is a feature vector with all words weighted at 1.0, and is equal to the unweighted “query corresponding feature vector (Vec)”.

[0578] It should be noted that the weighted query corresponding feature vector generation unit 122 outputs the "query corresponding feature vector (Vec)" input from the text encoder 121 as is, so that the above reference can be omitted. Figure 16 The arithmetic processing described in steps S01 and S02.

[0579] However, the above reference can be executed even in the case of an unweighted query (search statement). Figure 16 The arithmetic processing described in steps S01 and S02.

[0580] That is, all weights s of the word (x) input by the user. x Set to 1.0, and calculate the "weighted query corresponding feature vector V'" according to the above formula (expression 2). K 131.

[0581] V' K =V0+s1D1+...+s K D K ... (Expression 2)

[0582] If the user does not input the weight (scale) for each word (x), all weights (s) in the above (Expression 2) are... x =1.0.

[0583] In this case, the weighted query corresponding to the feature vector V' in the above (expression 2) K The calculation expression is as follows (expression 3).

[0584] V' K =V0+D1+...+D K ... (Expression 3)

[0585] In the above (expression 3), because

[0586] D1=V1-V0

[0587] D2=V2-V1 : D K =V K -V K-1 , The above (expression 3) becomes V' K =V0+D1+...+D K =V0+(V1-V0)+(V2-V1)+...(V K -V K -1) =V K , as well as The weighted query corresponding feature vector V' calculated using the above (expression 3) K "become with" Figure 15 The vector that matches the "Query corresponding feature vector (Vec)" shown in (a) of (a).

[0589] As described above, the weighted query corresponding feature vector generation unit 122 outputs...

[0590] (a) The "query-corresponding feature vector (Vec)" input from text encoder 121 is output as is, as in the case of an unweighted input query (search statement). Alternatively, (b) All weights s of the word (x) input by the user xSet to 1.0, and calculate the "weighted query corresponding feature vector V'" according to the above formula (expression 2). K 131.

[0591] V' K =V0+s1D1+...+s K D K ... (Expression 2)

[0592] Any configuration is acceptable as long as either of the two types of processing described above (a) or (b) is performed.

[0593] When the calculation of the feature vector corresponding to the unweighted query (search statement) is completed in step S111, the process proceeds to step S112.

[0594] (Step S112)

[0595] Next, in step S112, the data processing unit 102 of the information processing device 100 of this disclosure acquires or calculates the feature vector corresponding to the search target content (image, etc.).

[0596] As referenced above Figure 14 and Figure 17 As described, the content ID and feature vector are stored together with content such as images in databases 200-1 to n, which store content such as images that can be accessed by the information processing device 100 via the communication unit 103.

[0597] In step S112, the data processing unit 102 of the information processing device 100 of this disclosure obtains the stored feature vector associated with the image from the database 200-1 to n of the stored content.

[0598] It should be noted that, in the absence of image-associated feature vectors stored in the databases 200-1 to n, the content encoder 124 in the data processing unit 102 of the information processing device 100 analyzes the images obtained from the databases 200-1 to n and performs the operations as described above. Figure 18 The aforementioned processing of the calculated feature vectors.

[0599] (Step S113)

[0600] Next, in step S113, the data processing unit 102 of the information processing device 100 of this disclosure analyzes the similarity between the query-corresponding feature vector calculated in step S111 and the content (image, etc.)-corresponding feature vector obtained or calculated in step S112.

[0601] As mentioned above, similarity indices such as cosine similarity or vector distance (Euclidean distance) are used for similarity analysis between these feature vectors.

[0602] (Step S131)

[0603] Next, in step S131, the data processing unit 102 of the information processing device 100 of this disclosure selects content (images) with feature vectors that have a similarity to the query corresponding feature vectors equal to or higher than a specified threshold, based on the similarity analysis results of the feature vectors in step S113.

[0604] (Step S132)

[0605] Next, in step S132, the data processing unit 102 of the information processing device 100 of this disclosure sorts the content (image) selected in step S131 that has a similarity to the feature vector corresponding to the query that is equal to or higher than a specified threshold as the search results in descending order of similarity, and displays it on the display unit of the information processing device 100.

[0606] That is, these images are displayed as search results in the above reference. Figure 1 The search results are displayed in area 13 of the data search UI on the display unit of the described information processing device.

[0607] Next, it will be described in Figure 19 The following processing in step S121 is performed if the determination in step S103 is yes (i.e., if the input query (search statement) is determined to be a weighted query).

[0608] (Step S121)

[0609] If it is determined in step S103 that the input query (search statement) is a weighted query, then the following processing is performed in step S121.

[0610] First, in step S121, the input search query (search statement) is encoded to calculate the token-corresponding vector.

[0611] Note that this processing is based on the above references. Figure 16 The text encoder 121 described performs the processing.

[0612] like Figure 16 As shown in the lower left portion, the text encoder 121 calculates the token-corresponding vector (V0 to V). N ).

[0613] (Step S122)

[0614] Next, in step S122, the data processing unit 102 performs arithmetic processing that applies the token-corresponding vector calculated by the text encoder 121 in step S121 and the weight set by the user for each word, and calculates the weighted query-corresponding feature vector.

[0615] This process is based on the above references. Figure 16 The weighted query described corresponds to the arithmetic processing performed by the feature vector generation unit 122, and is... Figure 16 The processing steps S01 to S02 are shown in the figure.

[0616] The weighted query corresponding feature vector generation unit 122 first calculates the difference between adjacent tokens in the "token correspondence vector" generated by the text encoder 121 as the token correspondence vector difference (D). X ).

[0617] That is, according to the following expression (Expression 1), the token-corresponding vector difference (D) is... X ) is calculated as the vector corresponding to adjacent tokens (V) X and V X-1 The difference (Diff) between ).

[0618] Dx=V X -V X-1 ... (Expression 1)

[0619] Next, the weight (scale) of each word (x) input by the user is set to s. x And calculate the weighted query corresponding feature vector V' according to the following expression (expression 2). K ".

[0620] V' K =V0+s1D1+...+s K D K ... (Expression 2)

[0621] The weighted query corresponding feature vector generation unit 122 performs arithmetic processing to calculate the weighted query corresponding feature vector V'. K ".

[0622] (Step S123)

[0623] Next, in step S123, the data processing unit 102 of the information processing device 100 of this disclosure acquires or calculates the feature vector corresponding to the search target content (image, etc.).

[0624] This process is the same as the process described in step S112 above.

[0625] That is, the data processing unit 102 of the information processing device 100 of this disclosure obtains the stored feature vector associated with the image from the database 200-1 to n of the stored content.

[0626] Alternatively, if the stored image-associated feature vectors are not stored in databases 200-1 to n, the content encoder 124 in the data processing unit 102 of the information processing device 100 analyzes the images obtained from databases 200-1 to n and performs the operations as described above. Figure 18 The process described for calculating eigenvectors.

[0627] (Step S124)

[0628] Next, in step S124, the data processing unit 102 of the information processing device 100 of this disclosure analyzes the similarity between the weighted query corresponding feature vector calculated in step S122 and the content (image, etc.) corresponding feature vector obtained or calculated in step S123.

[0629] As mentioned above, similarity indices such as cosine similarity or vector distance (Euclidean distance) are used for similarity analysis between these feature vectors.

[0630] (Steps S131 and S132)

[0631] The processing in steps S131 to S132 is similar to the processing described above.

[0632] That is, in step S131, based on the similarity analysis results of the feature vectors in step S124, the content (image) with the feature vector corresponding to the content that has a similarity to the feature vector corresponding to the weighted query that is equal to or higher than a specified threshold is selected.

[0633] Next, in step S132, the content (image) selected in step S131 that has a similarity to the feature vector corresponding to the weighted query that is equal to or higher than a specified threshold is sorted in descending order of similarity and output side by side on the display unit of the information processing device 100 as the search results.

[0634] That is, these images are displayed as search results in the above reference. Figure 1 The search results are displayed in area 13 of the data search UI on the display unit of the described information processing device.

[0635] By performing such processing, search results that reflect the weight of each word set by the user can be obtained, and the search results desired by the user can be obtained efficiently.

[0636] Note that in the above embodiments, an example of image data being the search target has been described; however, the content or data being the search target is not limited to images and can be other types of data. That is, the category of data that can be the search target of the information processing apparatus of this disclosure is not limited to images. For example, data searches can be performed not only on images but also on various types of data such as documents, music, buildings, nature, and organisms that are the search targets.

[0637] [6. Application examples of data search and processing using the information processing equipment of this disclosure]

[0638] Next, an application example of data search processing using the information processing device disclosed herein will be described.

[0639] The following three processing examples are described as application examples of data search processing using the information processing device disclosed herein.

[0640] (1) Example of extracting specific scenes desired by users from movie content

[0641] (2) Example of extracting specific scenes desired by users from motion video content

[0642] (3) Example of processing to extract the motion data that the user wants from motion data

[0643] (6-(1) Example of processing to extract specific scenes desired by users from movie content)

[0644] First, we will describe an example of the process of extracting specific scenes that users want from movie content.

[0645] Films are often distributed worldwide, but regulations vary from country to country and region to region. Specifically, for example, there may be requirements to remove violent scenes. However, the standards for depicting violence differ from country to region, making it impossible to apply a uniform standard.

[0646] In this case, by performing the process described in this disclosure, namely, scene extraction by assigning weights to the words constituting the query (search statement), it is possible to extract scenes according to the standards of each region.

[0647] For example, such as Figure 21 As shown, this is a query (search statement). Query = extreme or violent scene The input is processed, and the search is performed while assigning various weights to the words "extreme" and "violent" included in the query.

[0648] By performing this process, scenes can be extracted based on the standards of each region.

[0649] For example, in the case of extracting the scene based on the criteria of region A, such as Figure 21 As shown, by setting

[0650] The weight of "extreme" is 1.5 times, and

[0651] The weight of "violent" is 1.8 times. Scenes can be extracted based on the standards of region A.

[0652] Furthermore, when extracting scenes based on criteria different from those of region B (region A), the weight settings are changed. For example, by setting...

[0653] The weight of "extreme" is 1.0, and

[0654] The weight of "violent" is 1.2 times. Scenes can be extracted based on the criteria of region B.

[0655] Reference Figure 22 Describes the processing when performing scene extraction processing based on the actual situation of the region.

[0656] Figure 22 The data processing unit 102 shown above is the reference. Figure 14 and Figure 18 The configuration of the data processing unit 102 of the information processing device 100 of this disclosure is described.

[0657] Note that the dashed line indicates that content encoder 124 may or may not be used.

[0658] Various movie contents are stored in the movie database 210.

[0659] With the scene-corresponding feature vectors associated with each scene of the movie content already stored in the movie database 210, the content encoder 124 is unnecessary, and the scene-corresponding feature vectors 132-1 to n are obtained from the movie database 210.

[0660] On the other hand, in the absence of scene-corresponding feature vectors associated with each scene of the movie content stored in the movie database 210, the content encoder 124 analyzes each scene of the movie content and calculates scene-corresponding feature vectors 132-1 to n.

[0661] The text encoder 121 and the weighted query corresponding feature vector generation unit 122 use the query (search statement) input via the input unit 111 (i.e., query = extreme or violent scene) and reference... Figure 21 The weight settings for each word in the description are used to calculate the "weighted query corresponding feature vector 131".

[0662] The vector similarity analysis unit 123 analyzes the similarity between the "weighted query corresponding feature vector 131" generated by the weighted query corresponding feature vector generation unit 122 and the scene corresponding feature vectors 132-1 to n, and outputs the scenes with high similarity to the output unit 112 as search results.

[0663] The user examines the output, and if the desired search results are not obtained, the weighting of the constituent words in the query is adjusted. The user then performs a re-search. By repeating this process, the desired search results can be obtained.

[0664] Figure 23 The flowchart shown is a flowchart used to describe the sequence of the above processing.

[0665] The sequence will be described in Figure 23 The core steps of the flowchart shown in the figure are processed.

[0666] (Step S201)

[0667] First, the data processing unit 102 of the information processing device 100 of this disclosure receives the following query (search statement) via the input unit 111. Query = extreme or violent scenarios And the weight information for each word.

[0668] For example, using reference Figure 21 The UI is described for inputting queries and weight information.

[0669] (Step S202)

[0670] Next, in step S202, the data processing unit 102 of the information processing device 100 of this disclosure performs the following data search processing: generating a weighted query corresponding feature vector based on the input query and the weight information of each word, and extracting scenes of movies that have scene corresponding feature vectors with high similarity to the generated weighted query corresponding feature vectors.

[0671] (Step S203)

[0672] Next, in step S203, the data processing unit 102 of the information processing device 100 of this disclosure outputs the scene of the movie that has a scene corresponding feature vector that has a high similarity to the weighted query corresponding feature vector obtained in the search processing in step S202 as the search result to the output unit 112.

[0673] These search results are displayed in the above reference. Figure 1 The search results are displayed in area 13 of the data search UI on the display unit of the described information processing device.

[0674] (Step S204)

[0675] Step S204 is the user's search result confirmation process.

[0676] If it is determined that a search result matching the user's intent has been obtained, the search process is terminated.

[0677] On the other hand, if it is determined that no search results matching the user's intent have been obtained, the process proceeds to step S205.

[0678] (Step S205)

[0679] If it is determined in step S204 that no search results matching the user's intent have been obtained, then the process in step S205 is executed.

[0680] The user sets up the following query as is and enters...

[0681] Query = extreme or violent scene

[0682] At the same time, the weight information of each word is changed.

[0683] Then, in step S204, the processing of S202 and subsequent steps is repeated.

[0684] By repeating the weight change process and search process as described above, the process ends when the determination in step S204 is yes, that is, when it is determined that the search results desired by the user have been obtained.

[0685] As described above, by applying the processing disclosed herein, the search results desired by the user can be obtained efficiently by performing weight adjustments on each component word of the query.

[0686] (6-(2) Example of processing to extract specific scenes desired by users from motion video content)

[0687] Next, we will describe a processing example of extracting specific scenes desired by the user from motion video content.

[0688] For example, in generating highlighted motion images that extract only the scenes of interest from video content of a football match, the following scene extraction process might be required: (a) The scenario of scoring a goal; (b) Scenes of athletes celebrating after scoring a goal; and (c) A scene where the goalkeeper makes a super save.

[0689] For example, even when extracting such scenes from video content of a football match, the processing disclosed herein can be applied to effectively extract scenes that match the user's intent.

[0690] For example, such as Figure 24 As shown, this is a query (search statement). The query is for scenes in which a player strikes a celebratory pose after scoring a goal. The input is used to assign various weights to the words "scoring a goal" and "celebratory pose" (word strings) included in the query, and then the search is performed.

[0691] For example, such as Figure 24 As shown, by setting weights,

[0692] The weighting for "scoring a goal" is 1.5 times, and

[0693] The weight of "celebratory pose" is -0.2. It can perform scene extraction centered on the scene of a goal scored.

[0694] By performing this process, scenarios that match the user's intent can be extracted.

[0695] It should be noted that the information processing device 100 of this disclosure is capable of performing weight settings not only on a single word, but also on word strings comprising multiple words as units.

[0696] Reference Figure 25 Describe the processing configuration when performing this scenario extraction process.

[0697] Figure 25 The data processing unit 102 shown above is the reference. Figure 14 and Figure 18 The configuration of the data processing unit 102 of the information processing device 100 of this disclosure is described.

[0698] Note that the dashed line indicates that content encoder 124 may or may not be used.

[0699] Various sports video content is stored in the sports video database 220.

[0700] With the scene-corresponding feature vectors associated with each scene of the sports video content already stored in the sports video database 220, the content encoder 124 is unnecessary, and the scene-corresponding feature vectors 132-1 to n are obtained from the sports video database 220.

[0701] On the other hand, in the case that the scene-corresponding feature vector associated with each scene of the sports video content is not stored in the sports video database 220, the content encoder 124 analyzes each scene of the sports video content and calculates the scene-corresponding feature vectors 132-1 to n.

[0702] The text encoder 121 and the weighted query corresponding feature vector generation unit 122 use the query (search statement) input via the input unit 111 (i.e., Query=scenes in which a player strikes a celebratory pose afterscoring a goal) and reference Figure 24 The weight settings for each word (word string) are described, and the "weighted query corresponding feature vector 131" is calculated.

[0703] The vector similarity analysis unit 123 analyzes the similarity between the "weighted query corresponding feature vector 131" generated by the weighted query corresponding feature vector generation unit 122 and the scene corresponding feature vectors 132-1 to n, and outputs the scenes with high similarity to the output unit 112 as search results.

[0704] The user examines the output, and if the desired search results are not obtained, the weighting of the constituent words in the query is adjusted. The user then performs a re-search. By repeating this process, the desired search results can be obtained.

[0705] (6-(3) Example of processing to extract the motion data that the user wants from motion data)

[0706] Next, we will describe a processing example of extracting the motion data that the user wants from motion data.

[0707] For example, in the case of processing a character to perform a specific action in game content, the following processing can be performed: retrieve the motion data that the user wants from a motion database that stores various data related to the action, i.e., motion data, and use the retrieved motion data to control the character's actions.

[0708] However, the following problem exists: due to the accumulation of a large amount of different sports data in the sports database, it is difficult to search for sports data that contain the sports that the user wants.

[0709] Figure 26 An example of motion data stored in motion database 230 is shown.

[0710] The examples shown are examples of "run" motion data and "run and jump" motion data.

[0711] In addition, a large amount of motion data is stored in the motion database 230.

[0712] By applying the processing disclosed herein, the motion data desired by the user can be effectively extracted.

[0713] Figure 27 Examples of two queries (search statements) for searching motion data are shown.

[0714] The query (search statement) for (a) is

[0715] The query is for "motion data of a running movement as if tired". This is an example of performing a search while assigning weights to the word "as if tired" included in the query.

[0716] The query (search statement) for (a) is

[0717] The query is for "motion data of a running movement performed quickly". This is an example of performing a search while assigning weights to the word "quickly" included in the query.

[0718] By adjusting the weight of each query, in case (a) we can select exercise data with different levels of fatigue, and in case (b) we can select exercise data with different speeds.

[0719] By performing such processing, motion data that matches the user's intent can be effectively extracted.

[0720] Reference Figure 28 Describes the processing configuration when performing this motion data extraction process.

[0721] Figure 28 The data processing unit 102 shown above is the reference. Figure 14 and Figure 18 The configuration of the data processing unit 102 of the information processing device 100 of this disclosure is described.

[0722] Note that the dashed line indicates that content encoder 124 may or may not be used.

[0723] The sports data database 230 stores sports data for various sports.

[0724] Since the motion data feature vectors associated with each piece of motion data are already stored in the motion data database 230, the content encoder 124 is unnecessary, and the motion data feature vectors 132-1 to n are obtained from the motion data database 230.

[0725] On the other hand, when the motion data corresponding feature vector associated with each piece of motion data is not stored in the motion data database 230, the content encoder 124 analyzes the motion data to calculate the motion data corresponding feature vectors 132-1 to n.

[0726] For example, the text encoder 121 and the weighted query corresponding feature vector generation unit 122 use the query (search statement) input via the input unit 111 (i.e., Query=motion data of a running movement as if tired) and reference Figure 27 The weight settings for each word (word string) are described, and the "weighted query corresponding feature vector 131" is calculated.

[0727] The vector similarity analysis unit 123 analyzes the similarity between the "weighted query corresponding feature vector 131" generated by the weighted query corresponding feature vector generation unit 122 and the motion data corresponding feature vectors 132-1 to n, and outputs the motion data with high similarity to the output unit 112 as the search result.

[0728] The user examines the output and, if the desired search result is not obtained, adjusts the weighting of the constituent words in the query. The user then performs a re-search. By repeating this process, the desired motion data can be obtained as the search result.

[0729] [7. Hardware configuration example of the information processing device disclosed herein]

[0730] Next, we will refer to Figure 29 This disclosure describes an example of the hardware configuration of the information processing device.

[0731] It should be noted that Figure 29 The hardware of the information processing device shown corresponds to the hardware of devices such as PCs, smartphones, and tablets, which are examples of information processing devices in this disclosure.

[0732] The following will describe the composition. Figure 29 The various hardware components of the information processing device shown.

[0733] The central processing unit (CPU) 301 serves as a control unit or data processing unit that performs various processes according to a program stored in the read-only memory (ROM) 302 or the storage unit 308. For example, the CPU 301 performs processing according to the sequence described in the above embodiments. The random access memory (RAM) 303 stores the programs, data, etc., executed by the CPU 301. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304.

[0734] The CPU 301 is connected to the input / output interface 305 via the bus 304, and includes an input unit 306 for various switches, buttons, touch panels, sensors, cameras, microphones, etc., and an output unit 307 for executing data output to the display unit, speakers, etc., connected to the input / output interface 305. For example, the CPU 301 performs various processes in response to commands input from the input unit 306, and outputs the processing results to the output unit 307.

[0735] The storage unit 308, connected to the input / output interface 305, stores the program and data executed by the CPU 301. The communication unit 309 serves as a transmission / reception unit for Wi-Fi communication, Bluetooth (registered trademark) communication, and other data communication via networks such as the Internet or a local area network, and communicates with external devices.

[0736] The driver 310 connected to the input / output interface 305 drives a removable medium 311, such as a semiconductor memory (such as a memory card), and records or reads data.

[0737] [8. Overview of the configuration disclosed herein]

[0738] The embodiments of this disclosure have been described in detail above with reference to specific examples. However, it will be apparent to those skilled in the art that modifications or substitutions can be made to the embodiments without departing from the spirit of this disclosure. That is, this disclosure is provided by way of illustration and should not be interpreted in a limiting manner. The claims should be considered in order to determine the essence of this disclosure.

[0739] It should be noted that the technology disclosed in this specification may have the following configurations.

[0740] (1) An information processing device, comprising: a data processing unit, wherein the data processing unit: Retrieve queries that include the text to be applied to data search processing; and Based on this query, data search processing is performed, where, The data processing unit Along with the query, obtain the weight information for each word or word string used to constitute the query, and Using this query and the weight information for each word or word string, perform search processing on data in categories different from the text category.

[0741] (2) According to the information processing device of (1), the data in the category different from the category of text is image data or music data.

[0742] (3) According to the information processing device of (1) or (2), the weight information of each word or word string used to constitute the query is: Positive weights or negative weights, and The data processing unit performs search processing on words or word strings with positive weights, increasing their influence in the search process. Perform search processing on words or word strings that have been given negative weights, resulting in a reduced impact on the search process.

[0743] (4) The information processing device according to any one of (1) to (3), further comprising: The display unit displays a data search user interface, wherein the data processing unit displays the data search user interface on the display unit. The data search user interface includes a query input area that allows input of weight information for each word or word string.

[0744] (5) The information processing device according to (4), wherein the data processing unit displays a data search user interface on the display unit, the data search user interface including a query input area, the query input area enabling numerical input of weight information for each word or word string.

[0745] (6) The information processing device according to (4) or (5), wherein the data processing unit displays a data search user interface on the display unit, the data search user interface including a query input area, the query input area enabling the input of weight information for each word or word string using brackets.

[0746] (7) An information processing device according to any one of (4) to (6), wherein the data processing unit displays a data search user interface on a display unit, the data search user interface including a query input area that enables input of weight information for each word or word string by operating a scroll bar.

[0747] (8) An information processing device according to any one of (4) to (7), wherein the data processing unit displays a data search user interface on a display unit, the data search user interface including a query input area, the query input area enabling the use of color display (heatmap) to input weight information for each word or word string.

[0748] (9) An information processing device according to any one of (1) to (8), wherein the data processing unit uses the query and the weight information for each word or word string to generate a weighted query-corresponding feature vector, and

[0749] Perform content search processing to select and retrieve content with feature vectors that have a high similarity to the feature vectors corresponding to the generated weighted query.

[0750] (10) The information processing device according to (9), wherein the data processing unit includes: An encoder that generates a token-to-vector corresponding to each token, where a token is a processing unit comprising the constituent words of the query; and The weighted query corresponding feature vector generation unit generates the weighted query corresponding feature vector by performing arithmetic processing on the token corresponding vector generated by the encoder.

[0751] (11) The information processing device according to (10), wherein the encoder is an encoder using a deep learning model that utilizes an attention mechanism; and

[0752] This includes a one-way focus mechanism, which analyzes the relevance of each word in the query only to the word following it.

[0753] (12) The information processing device according to (10) or (11), wherein the weighted query corresponding feature vector generation unit calculates the token corresponding vector difference, the token corresponding vector difference being the difference between adjacent tokens in the token corresponding vector generated by the encoder, and

[0754] A weighted query-corresponding feature vector is generated by performing arithmetic processing on the calculated token-corresponding vector difference and weight information for each word or word string used to constitute the query.

[0755] (13) An information processing device according to any one of (10) to (12), wherein the weighted query corresponding feature vector generation unit calculates the weighted query corresponding feature vector V' by performing arithmetic processing according to the following formula. K : V' K =V0+s1D1+...+s K D K As used for the vector difference corresponding to the token (D) X ) and weight information (s x Arithmetic processing of tokens, corresponding vector differences (D) X ) is the difference between adjacent tokens in the token-corresponding vector generated by the encoder, weight information (s x ) is used to construct each word or word string in the query, where, V' K It is a weighted query corresponding to the feature vector. V0 is the vector corresponding to the first token output from the encoder. s n It is a weight assigned to the nth word in the constituent words of the query, and D n It is the difference between the token vector of the nth word in the query and the token vector of the (n-1)th word.

[0756] (14) An information processing device according to any one of (9) to (13), wherein the data processing unit includes a feature vector similarity analysis unit, and the feature vector similarity analysis unit selects content corresponding to the feature vector that has a high similarity to the feature vector corresponding to the weighted query.

[0757] (15) The information processing device according to (14) wherein the feature vector similarity analysis unit calculates the cosine similarity or Euclidean distance and performs vector similarity analysis, wherein the cosine similarity or Euclidean distance is the vector similarity judgment index value in the weighted query corresponding feature vector and the content corresponding feature vector similarity analysis processing.

[0758] (16) An information processing device according to any one of (1) to (15), wherein the data processing unit performs search processing using data stored in a database as search target data.

[0759] (17) According to the information processing device of (16), the content as the search target and the content-corresponding feature vector generated by the feature analysis of the content are stored in the database, and

[0760] The data processing unit performs content search processing, which selectively retrieves content from the database that has a content-corresponding feature vector that has a high similarity to the weighted query-corresponding feature vector generated using the query and the weight information for each word or word string.

[0761] (18) An information processing device according to (16) or (17), wherein the content as the search target is stored in a database, and

[0762] The data processing unit performs analysis on the content stored in the database to calculate the feature vectors corresponding to the content, and

[0763] The content search process selects content-corresponding feature vectors from the computed content-corresponding feature vectors that have high similarity to the weighted query-corresponding feature vectors generated using the query and the weight information for each word or word string, and selectively retrieves content from the database from which the selected content-corresponding feature vectors were generated as search results.

[0764] (19) An information processing method, comprising: Retrieve queries that include the text to be applied to data search processing, and Weight information used to construct each word or word string in the query; and Using this query and the weight information for each word or word string, perform search processing on data in categories different from the text category.

[0765] (20) A program that causes an information processing device to perform information processing, the program causing the data processing unit of the information processing device to: Retrieve queries that include the text to be applied to data search processing; and Weight information for each word or word string used to constitute the query; and Using this query and the weight information for each word or word string, perform search processing on data in categories different from the text category.

[0766] The series of processes described herein can be executed by hardware, software, or a combination of both. In the case of software execution, a program containing the processing sequence can be installed in the memory of a computer with dedicated hardware and executed, or the program can be installed in a general-purpose computer capable of performing various types of processing and executed. For example, the program can be pre-recorded on a recording medium. Besides being installed on a computer from a recording medium, the program can also be received via a network (such as a local area network (LAN) or the Internet) and installed on a recording medium (such as an internal hard drive).

[0767] It should be noted that the various types of processing described in this specification can be executed not only sequentially as described, but also in parallel or individually, depending on the processing capabilities of the device performing the processing or as needed. Furthermore, the system described in this specification is a logical collection configuration of multiple devices, and is not limited to a system where the devices in each configuration are housed in the same enclosure.

[0768] Industrial applicability

[0769] As described above, according to one embodiment of the present disclosure, an apparatus and method are implemented for performing search processing of data in a category different from the category of text, using query and weight information for each word.

[0770] Specifically, for example, it includes a data processing unit that receives a query containing text applicable to data search processing and performs data search processing based on the query. The data processing unit receives weight information for each word constituting the query along with the query, and performs data search processing in categories different from the text's category using the query and the weight information for each word. Search processing with increased influence is performed for words with positive weights, and search processing with decreased influence is performed for words with negative weights.

[0771] This configuration enables devices and methods for performing search processing of data in categories different from the categories of text, using query and weight information for each word.

[0772] List of reference numerals

[0773] 11. Query Input Area

[0774] 12 Search (Start Search) icon

[0775] 13 Search Results Display Area

[0776] 100 Information processing equipment

[0777] 101 User Interface (UI)

[0778] 102 Data Processing Unit

[0779] 103 Communication Unit

[0780] 111 Input Unit

[0781] 112 Output Unit

[0782] 121 Text Encoder

[0783] 122 Weighted Query Corresponding Feature Vector Generation Department

[0784] 123 Feature Vector Similarity Analysis Department

[0785] 124 Content Encoder

[0786] 131 Weighted query corresponding feature vector

[0787] 132 Content-Corresponding Feature Vector

[0788] 133 search results

[0789] Databases 200, 210, 220, 230

[0790] 211 Content Encoder

[0791] 301 CPU

[0792] 302 ROM

[0793] 303 RAM

[0794] 304 bus

[0795] 305 Input / Output Interface

[0796] 306 Input Unit

[0797] 307 Output Unit

[0798] 308 memory cells

[0799] 309 Communication Unit

[0800] 310 drive

[0801] 311 Removable media.

Claims

1. An information processing device, comprising: Data processing unit, the data processing unit: Retrieve queries that include the text to be applied to data search processing; as well as Based on the query, data search processing is performed, wherein, The data processing unit: Together with the query, obtain the weight information for each word or word string that constitutes the query, and Using the query and the weight information for each word or word string, perform search processing on data in categories different from the category of the text.

2. The information processing device according to claim 1, wherein, The data in the category that is different from the category of text is image data or music data.

3. The information processing device according to claim 1, wherein, The weight information used to constitute each word or word string in the query is: Positive weights or negative weights, and The data processing unit performs search processing on words or word strings that have been given positive weights, thereby increasing their influence in the search process. For words or word strings that have been given negative weights, a search process with reduced influence in the search process is performed.

4. The information processing device according to claim 1, further comprising: The display unit displays a data search user interface, wherein... The data processing unit: The data search user interface is displayed on the display unit. The data search user interface includes a query input area that allows input of weight information for each word or word string.

5. The information processing device according to claim 4, wherein, The data processing unit: The data search user interface is displayed on the display unit. The data search user interface includes a query input area that allows numerical input of weight information for each word or word string.

6. The information processing device according to claim 4, wherein, The data processing unit: The data search user interface is displayed on the display unit. The data search user interface includes a query input area that allows weight information for each word or word string to be entered using brackets.

7. The information processing device according to claim 4, wherein, The data processing unit: The data search user interface is displayed on the display unit. The data search user interface includes a query input area that allows input of weight information for each word or word string by operating a scroll bar.

8. The information processing device according to claim 4, wherein, The data processing unit: The data search user interface is displayed on the display unit. The data search user interface includes a query input area that enables the input of weight information for each word or word string using color display (heatmap).

9. The information processing device according to claim 1, wherein, The data processing unit: The query and the weight information for each word or word string are used to generate a weighted query-corresponding feature vector, and Perform content search processing to select and obtain content corresponding feature vectors that have a high similarity to the generated weighted query corresponding feature vectors.

10. The information processing device according to claim 9, wherein, The data processing unit includes: An encoder that generates a token-to-vector corresponding to each token, wherein the token is a processing unit comprising the constituent words of the query; and The weighted query corresponding feature vector generation unit generates the weighted query corresponding feature vector by performing arithmetic processing on the token corresponding vector generated by the encoder.

11. The information processing device according to claim 10, wherein, The encoder: It is an encoder that uses a deep learning model that leverages attention mechanisms, and This includes a one-way focus mechanism, which analyzes the relevance of each word constituting the query only to the word following it.

12. The information processing device according to claim 10, wherein, The weighted query corresponds to the feature vector generation part: Calculate the token-to-vector difference, which is the difference between adjacent tokens in the token-to-vector generated by the encoder, and A weighted query-corresponding feature vector is generated by performing arithmetic processing on the calculated token-corresponding vector difference and the weight information used to constitute each of the words or word strings in the query.

13. The information processing device according to claim 10, wherein, The weighted query corresponds to the feature vector generation part: The weighted query-corresponding feature vector V' is calculated by performing arithmetic operations according to the following equation. K : V' K = V0 + s1D1 +... + s K D K As used for the vector difference corresponding to the token (D) X ) and weight information (s x The arithmetic processing of the tokens, the corresponding vector difference (D) X The weight information (s) is the difference between adjacent tokens in the token-corresponding vector generated by the encoder. x ) are used to constitute each of the words or word strings in the query, wherein, V' K It is the feature vector corresponding to the weighted query. V0 is the first token-corresponding vector output from the encoder. s n It is the weight set for the nth word in the constituent words of the query, and D n It is the difference between the token vector of the nth word in the constituent words of the query and the token vector of the (n-1)th word.

14. The information processing device according to claim 9, wherein, The data processing unit: It includes a feature vector similarity analysis unit, which selects content whose feature vector has a high similarity to the feature vector corresponding to the weighted query.

15. The information processing apparatus according to claim 14, wherein, The feature vector similarity analysis unit: Calculate cosine similarity or Euclidean distance, and perform vector similarity analysis. The cosine similarity or Euclidean distance is a vector similarity judgment index value in the similarity analysis and processing between the weighted query corresponding feature vector and the content corresponding feature vector.

16. The information processing device according to claim 1, wherein, The data processing unit: The search process uses data stored in the database as the target data for the search.

17. The information processing device according to claim 16, wherein, The content used as the search target and the corresponding feature vectors generated from the feature analysis of the content are stored in the database, and The data processing unit performs content search processing, which selectively retrieves from the database content corresponding to feature vectors that have a high similarity to the weighted query corresponding feature vectors generated using the query and the weight information for each word or word string.

18. The information processing device according to claim 16, wherein, The content used as the search target is stored in the database, and The data processing unit: Perform analysis on the content stored in the database to calculate feature vectors corresponding to the content, and The content search process performs a content search process that selects content-corresponding feature vectors from the calculated content-corresponding feature vectors that have a high similarity to the weighted query-corresponding feature vectors generated using the query and the weight information for each word or word string, and selectively retrieves content from the database from which the selected content-corresponding feature vectors were generated as search results.

19. An information processing method, comprising: Retrieve queries that include the text to be applied to data search processing, and Weight information used to constitute each word or word string in the query; as well as Using the query and the weight information for each word or word string, perform search processing on data in categories different from the category of the text.

20. A program that causes an information processing device to perform information processing, the program causing the data processing unit of the information processing device to: Retrieve queries that include the text to be applied to data search processing; as well as Weight information used to constitute each word or word string in the query; as well as Using the query and the weight information for each word or word string, perform search processing on data in categories different from the category of the text.

Images