Retrieval device and user terminal

Abstract

To deal with conventional difficulty in obtaining information about an object appearing in a field of view of a user taking the direction of movement of the user into consideration.SOLUTION: A retrieval device 1 includes: a specific information receiving unit 121 that accepts specific information having a query accepted during movement of a user, the query having an object direction concerning the direction in which an object is present relative to the direction of movement of the user and an object outer shape concerning an outer shape of the object and user information that can be acquired during the movement of the user; a retrieval unit 131 that acquires object explanation information including the name of the object specified by the specific information; and an output unit 14 that outputs the object explanation information acquired by the retrieval unit 131. The retrieval device can easily obtain information about the object appearing in a field of view of the user taking the direction of movement of the user into consideration.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a search device or the like that acquires information while a user is on the move. [Background technology]

[0002] Conventionally, there has been a map display device that has a function to search for and display facilities located around the vehicle (see Patent Document 1). This conventional technology aims to speed up the process of comparing and determining whether map objects constituting map data are hidden by other map objects when performing billboard processing on two-dimensional map data to display the map. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5762637 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, since the direction of movement of the user is not taken into consideration, it is difficult to obtain information about objects that are within the user's field of view. [Means for solving the problem]

[0005] The search device of the first invention is a search device that includes a specific information receiving unit that receives specific information, which is a query received while a user is moving, and which has an object direction related to the direction in which the object exists relative to the user's movement direction and an object outer shape related to the object's outer shape, and user information that can be obtained while the user is moving, a search unit that obtains object description information including the name of the object identified by the specific information, and an output unit that outputs the object description information obtained by the search unit.

[0006] With this configuration, it is possible to easily obtain information about objects that are within the user's field of view, taking into account the user's direction of movement.

[0007] Furthermore, in the search device of the second invention, compared to the first invention, the specific information receiving unit receives specific information having a query and user information having a user position that identifies the user's position and movement direction information that can identify the user's movement direction, and the search unit determines, from an object management unit in which two or more object information having the name of the object and the object position that identifies the position of the object is stored, an object that exists in the movement direction identified by the movement direction information from the user position contained in the user information, using the object position, and obtains object description information including the name of the object.

[0008] With this configuration, it is possible to easily obtain information about objects that are within the user's field of view, taking into account the user's direction of movement.

[0009] Furthermore, the search device of the third invention is a search device in which, compared to the first invention, the object management unit stores two or more pieces of object information acquired based on videos with location information acquired while one or two or more users are moving.

[0010] With this configuration, it is possible to easily obtain information about objects that are within the user's field of view, taking into account the user's direction of movement.

[0011] Furthermore, the search device of the fourth invention is a search device in which, compared to the second or third invention, the specific information receiving unit receives specific information including user information from the user terminal when the user terminal receives a query having an object direction and an object outer shape while the user is moving and is unable to obtain object description information corresponding to the query.

[0012] With this configuration, it is possible to easily obtain information about objects that are within the user's field of view, taking into account the user's direction of movement.

[0013] In addition, the search device of the fifth invention, compared to the first invention, is a search device that accepts specific information having user information which is an image taken while the user is moving, and the search unit provides a prompt including the image and the object outline to the generation AI and obtains object description information from the generation AI.

[0014] With this configuration, it is possible to easily obtain information about objects that are within the user's field of view, taking into account the user's direction of movement.

[0015] In addition, the search device of the sixth invention is a search device that, with respect to any one of the first to fifth inventions, further comprises a history storage unit that acquires and stores history information regarding the output of object description information output by the output unit.

[0016] With this configuration, it is possible to easily obtain information about objects that are within the user's field of view, taking into account the user's direction of movement.

[0017] In addition, the user terminal of the seventh invention is a user terminal that includes a terminal receiving unit that receives a query having an object direction related to the direction of the object relative to the user's position and an object outer shape related to the outer shape of the object while the user is moving, a user position acquisition unit that acquires a user position that identifies the user's position, a specific information construction unit that constructs specific information having two or more user positions or movement direction information based on two or more user positions acquired by the user position acquisition unit and the query, a terminal transmitting unit that transmits the specific information to a search device, a terminal receiving unit that receives object description information including the name of the object from the search device, and a terminal output unit that outputs the object description information received by the terminal receiving unit.

[0018] With this configuration, it is possible to easily obtain information about objects that are within the user's field of view, taking into account the user's direction of movement. [Effects of the Invention]

[0019] According to the search device of the present invention, it is possible to easily obtain information about objects that are within the field of view of a user, taking into consideration the direction of movement of the user. [Brief explanation of the drawings]

[0020] [Figure 1] Conceptual diagram of search system A in embodiment 1 [Figure 2] Block diagram of the search system A [Figure 3] A flowchart illustrating an example of the operation of the search device 1. [Figure 4] 10 is a flowchart illustrating a first example of the information acquisition process. [Figure 5] A flowchart illustrating an example of the score acquisition process [Figure 6] 10 is a flowchart illustrating a second example of the information acquisition process. [Figure 7] A flowchart illustrating an example of the history accumulation process. [Figure 8] A flowchart illustrating an example of a process for creating a same-popularity list. [Figure 9] A flowchart illustrating an example of the operation of the user terminal 2 [Figure 10] 10 is a flowchart illustrating a first example of the user information acquisition process. [Figure 11] 10 is a flowchart illustrating a second example of the user information acquisition process. [Figure 12] A diagram showing the object management table [Figure 13] Figure showing the history management table [Figure 14] FIG. 1 is a diagram illustrating the invention. [Figure 15] A diagram explaining the output example [Figure 16] An example of the prompt is shown below. [Figure 17] Block diagram of the computer system DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, embodiments of a search device and the like will be described with reference to the drawings. Note that components with the same reference numerals in the embodiments perform similar operations, and therefore repeated description may be omitted.

[0022] In this specification, information X being associated with information Y means that information Y can be obtained from information X, or information X can be obtained from information Y, and the method of association is not important. Information X and information Y may be linked, may exist in the same buffer, information X may be included in information Y, or information Y may be included in information X, etc.

[0023] Furthermore, in this specification, selecting or determining information Z means obtaining information Z, obtaining a pointer to information Z, obtaining the ID of information Z, setting a flag on information Z, etc., and it is sufficient if information Z can be accessed.

[0024] (Embodiment 1) 1 is a conceptual diagram of a search system A according to this embodiment. The search system A includes a search device 1 and one or more user terminals 2.

[0025] The search device 1 is a device that acquires information based on a query from a user. The search device 1 is usually a server, but may also be a terminal. When the search device 1 is a server, the search device 1 may be, for example, a cloud server or an ASP server, but the type does not matter. When the search device 1 is a terminal, the search device 1 may be, for example, a navigation terminal, a smartphone, a personal computer, a tablet terminal, or a drive recorder, but the type does not matter. Note that when the search device 1 is a terminal, the user terminal 2 is not required.

[0026] The user terminal 2 is a terminal used by a user. The user is the person who inputs a query. The user typically inputs a query while moving. It is preferable that the query is input by voice, but the input method is not important. It is preferable that the user moves by a moving object, but it may also be movement under the user's own power. Movement under the user's own power includes, for example, movement by walking or movement by running. The moving object is typically a car, but the type is not important, such as a bicycle, train, or bus.

[0027] The user terminal 2 may be, for example, a navigation terminal, a smartphone, a personal computer, a tablet terminal, or a drive recorder, but the type is not important.

[0028] The search device 1 and one or more user terminals 2 can communicate with each other via a network such as the Internet.

[0029] 2 is a block diagram of a search system A according to this embodiment. The search device 1 includes a storage unit 11, a reception unit 12, a processing unit 13, and an output unit 14. The storage unit 11 includes an object management unit 111 and a history management unit 112. The reception unit 12 includes a specific information reception unit 121. The processing unit 13 includes a search unit 131 and an accumulation unit 132.

[0030] The user terminal 2 includes a terminal storage unit 21, a terminal reception unit 22, a device processing unit 23, a terminal transmission unit 24, a terminal reception unit 25, and a terminal output unit 26. The terminal storage unit 21 includes an object management unit 111. The device processing unit 23 includes a user position acquisition unit 231 and a specific information configuration unit 232.

[0031] Various types of information are stored in the storage unit 11 constituting the search device 1. The various types of information include, for example, object information (to be described later) and history information (to be described later).

[0032] Two or more pieces of object information are stored in the object management unit 111. Note that the object management unit 111 may exist in an external device (not shown).

[0033] Object information is information about an object. An object is something that a user asks a question about. An object is, for example, a man-made object such as a building, a store, a park, or a bridge, or a natural object such as a mountain or a river. An object is, for example, a so-called POI (Point of Interest). Object information usually includes the name and position of the object. The object position is information that specifies the position of the object. The object position may be information that specifies the area in which the object exists. The object position is, for example, one or more (latitude, longitude), or one or more (latitude, longitude, altitude).

[0034] The object information has, for example, one or more object attribute values. The object attribute value is, for example, the object's outer shape. The object's outer shape is information obtained from the appearance of the object. The object's outer shape is, for example, the type of object (mountain, building, store), the shape of the object, the color of the object, and the size of the object. The object attribute value is, for example, object feature information. The object feature information is characteristic information of the object. The object feature information is, for example, the height of the mountain, the year the building was built, the year the store opened, and the type of store (e.g., Italian, ramen shop). The object feature information may include the object's outer shape. It is preferable that the object information be associated with an object identifier. The object identifier is an ID that identifies the object.

[0035] It is preferable that the object information of the object management unit 111 is information acquired based on a video with location information. A video with location information is a video with associated location information. For example, each still image constituting a video with location information is associated with location information. The location information is, for example, (latitude, longitude) or (latitude, longitude, altitude). A video with location information is, for example, a video captured by a drive recorder (not shown). A video with location information is, for example, a video received by the reception unit 12 from the drive recorder. Note that, for example, the processing unit 13 described later acquires the object information from the video.

[0036] The history management unit 112 acquires and accumulates history information. History information is information relating to the output of object description information by the output unit 14. The history information includes, for example, an object identifier, the name of the object, and the number of times it has been output. The history information includes, for example, one or more pieces of time information. The time information is information that specifies when the object description information was output. The time may be a date, a date and time, a time of day, or a time period.

[0037] The receiving unit 12 receives various information and instructions, such as specific information, a video with location information, and a popularity output instruction, which will be described later.

[0038] A popularity output instruction is an instruction to output information about a popular object. A popular object here is an object that has been output many times. Preferably, the popularity output instruction is an instruction to output information about a popular object among objects that exist near the location of the user or user terminal 2. Note that the popularity output instruction may have, for example, one or more user positions. The user's movement direction can be identified by two or more user positions. The popularity output instruction may have conditions related to one or more object attribute values. Such conditions are, for example, conditions related to the type of object (e.g., "restaurant"). An example of a popularity output instruction is, "Tell me about popular restaurants around here!"

[0039] The reception unit 12 receives video with location information from, for example, the user terminal 2 or a drive recorder (not shown), and stores the video with the location information.

[0040] The identification information receiving unit 121 receives the identification information. For example, the identification information receiving unit 121 receives the identification information from the user terminal 2. The identification information receiving unit 121 usually receives the identification information while the user is moving.

[0041] The specific information is information for identifying an object that matches a query issued while the user is moving, and includes, for example, a query and user information.

[0042] A query is an inquiry from a user that occurs while the user is moving, and includes an object direction and an object outer shape.

[0043] The object direction is information about the direction in which the object exists. The object direction is information about the direction of the object relative to the user's moving direction. The object direction is, for example, "front," "left," "right," or "back."

[0044] The object shape is information about the shape of the object. The object shape is, for example, the type of object, the shape of the object, or the color of the object. The type of object is, for example, a "building," "a building," "a store," "a shop," "a park," "a bridge," "a mountain," or "a river."

[0045] User information is information acquired while the user is moving. For example, the user information is user position, movement direction information, or an image. For example, the user information is user position and movement direction information, or an image.

[0046] The user position is position information that specifies the user's position. The user's position may be the position of the user terminal 2. The movement direction information is information that specifies the user's movement direction. The user's movement direction may be the movement direction of the user terminal 2. The movement direction information is, for example, an angle (for example, 45 degrees) or a direction (for example, "north," "northeast," or "south-southeast") relative to a reference direction (for example, true north). The image is, for example, a still image or video captured by the user terminal 2 or a drive recorder (not shown).

[0047] For example, when the user terminal 2 receives a query having an object direction and an object outer shape while the user is moving and is unable to acquire object description information corresponding to the query, the identification information receiving unit 121 receives identification information including user information from the user terminal 2. Such identification information includes, for example, a query having an object outer shape and user information. The query here does not need to include the object direction. The user information here is, for example, an image.

[0048] The identification information receiving unit 121 receives, for example, identification information having user information, which is an image captured while the user is moving. The identification information receiving unit 121 receives, for example, identification information having the image and the outline of an object.

[0049] The processing unit 13 performs, for example, an object information creation process. The object information creation process is a process of creating object information using a video with location information received by the receiving unit 12. The processing unit 13, for example, performs object recognition processing on one or more still images included in the video to detect one or more objects and acquire the types of the objects. Next, for each of the one or more objects, the processing unit 13 searches a map database (not shown) using the location information and type paired with the still image, and acquires the name of the object, one or more object feature information, and one or more object attribute values ​​from the map database. The processing unit 13 then generates a unique object identifier and stores the object information in the object management unit 111 in association with the object identifier. Note that the object information includes, for example, the name of the object, one or more object feature information, and one or more object attribute values. The processing unit 13 may associate a still image or video in the received video with the object information.

[0050] Processing unit 13 performs a popularity list construction process in response to the received popularity output instruction, for example. The popularity list construction process is a process for constructing a popularity list. The popularity list construction process will be described below with reference to FIG. 8.

[0051] The search unit 131 acquires object description information including the name of the object identified by the identification information received by the identification information receiving unit 121.

[0052] The search unit 131 can acquire object description information, for example, by using (1) the object management unit 111 or (2) a generation AI, as shown below. The generation AI here is software or a device that, when given a prompt, outputs an answer to the prompt. The generation AI here is software or a device that, when given a prompt including an image, outputs an answer to the object in the image. The generation AI can be, for example, "ChatGPT (registered trademark) 4.0" or "T5." The prompt may also be called a query, inquiry, or the like.

[0053] (1) Method using the object management unit 111 The search unit 131 determines, using the object positions contained in the object information of the object management unit 111, one or more objects that are objects that exist in a direction specified by the object direction (e.g., "left") relative to the movement direction (e.g., "north") specified by the movement direction information from the user position (e.g., (x1, y1)) contained in the user information included in the identification information received by the identification information receiving unit 121, and that match the object outer shape contained in the identification information, and acquires object description information including the names of each of the one or more objects. Note that it is preferable for the search unit 131 to acquire object description information for one object.

[0054] The object description information includes, for example, one or more object attribute values ​​in addition to the name of the object. The search unit 131 may acquire one or more object attribute values ​​from the object management unit 111, or may acquire one or more object attribute values ​​by searching a database (not shown) using the name of the object as a key.

[0055] (2) Using generative AI The search unit 131 gives the generation AI a prompt including, for example, an image included in the user information and the object outer shape, and acquires object description information from the generation AI.

[0056] The search unit 131, for example, acquires an image corresponding to the object direction from images included in the user information, provides a prompt including the image and the object outline to the generation AI, and acquires object description information from the generation AI. The search unit 131, for example, identifies one camera (for example, a camera that captures the image ahead of a moving object) from images acquired by two or more cameras, acquires the image captured by that camera, provides a prompt including the image and the object outline to the generation AI, and acquires object description information from the generation AI.

[0057] The search unit 131, for example, obtains an image by cutting out a partial area corresponding to the object direction from an image contained in the user information, provides a prompt including the image and the object outline to the generation AI, and obtains object description information from the generation AI.

[0058] The search unit 131, for example, obtains an image by cutting out a portion of the area corresponding to the object direction from an image included in the user information (for example, an image taken with a 360-degree camera), corrects the image, provides a prompt including the corrected image and the object outline to the generation AI, and obtains object description information from the generation AI.

[0059] The storage unit 132 acquires and stores history information. The history information is information relating to the output of object description information by the output unit 14. The history information includes, for example, an object identifier, the name of the object, and the number of times it has been output. The history information includes, for example, time information. The time information is information that specifies when the object description information was output. The time may be a date, a date and time, a time of day, or a time period. The storage unit 132 stores the history information in, for example, the history management unit 112.

[0060] The output unit 14 outputs the object description information acquired by the search unit 131. The output unit 14 transmits the object description information acquired by the search unit 131 to the user terminal 2, for example.

[0061] Various types of information are stored in the terminal storage unit 21 that constitutes the user terminal 2. The various types of information are, for example, information about two or more objects.

[0062] The object management unit 111 of the user terminal 2 and the object management unit 111 of the search device 1 may have different contents, data structures, etc. The object management unit 111 of the user terminal 2 usually has a smaller amount of data than the object management unit 111 of the search device 1.

[0063] The terminal reception unit 22 receives various types of information and instructions. The terminal reception unit 22 receives queries while the user is moving. It is preferable that the terminal reception unit 22 receives queries by voice.

[0064] A query has an object direction, which is the direction of the object relative to the user's position, and an object shape, which is the shape of the object. Examples of queries are "What is the snowy mountain in front?", "What is the store with the queue on the left?", "What is the orange tower on the right?", and "What is the building with the red walls in the background?".

[0065] The means for inputting queries etc. is preferably a microphone, but may also be a touch panel, keyboard, mouse, menu screen, etc.

[0066] The terminal processing unit 23 performs various types of processing. For example, the various types of processing are processing to convert received information, instructions, etc. into information, instructions, etc. with a structure to be transmitted. For example, the various types of processing are processing to convert received information into information with a structure to be output.

[0067] The device processing unit 23 performs speech recognition processing on the query, which is, for example, a voice, to acquire the query, which is a character string. The device processing unit 23 also acquires, for example, the object direction and the object outer shape from the query, which is a character string.

[0068] The device processing unit 23, for example, detects the presence of characters indicating directions such as "front," "left," "right," and "back" in the query, and acquires the object direction corresponding to the characters. The device processing unit 23, for example, performs morphological analysis on the character string that is the query, acquires noun words, and acquires from the words words that indicate the external appearance of the object, such as its type, shape, or color. Each of these one or more words constitutes the external appearance of the object.

[0069] For example, the device processing unit 23 cuts out an image of a direction specified by the object direction included in the received query from the captured image. In this case, the captured image is, for example, an image captured by a 360-degree camera.

[0070] The device processing unit 23, for example, selects and acquires an image of the direction specified by the object direction included in the received query from images captured by each of two or more cameras. In this case, information indicating the direction (for example, "front," "left," "right," or "rear") is stored in association with the identifier of each of the two or more cameras.

[0071] The device processing unit 23 acquires object description information, for example, using the acquired specific information. This processing may be the same as the processing of the search unit 131. However, although the device processing unit 23 performs the processing to acquire object description information, it is not necessary for it to be able to acquire object description information.

[0072] The user position acquisition unit 231 acquires a user position that identifies the user's location. The user position acquisition unit 231 acquires the user position by, for example, a GPS receiver. The user position acquisition unit 231 acquires the user position continuously, for example, periodically or irregularly.

[0073] The specific information constructing unit 232 constructs specific information having, for example, a query and one or more user positions acquired by the user position acquiring unit 231. The specific information constructing unit 232 acquires movement direction information using, for example, two or more user positions acquired by the user position acquiring unit 231, and constructs specific information having user information having the movement direction information and at least one user position, and a query.

[0074] The travel direction information may be acquired by the user terminal 2 or the search device 1.

[0075] The terminal transmitting unit 24 transmits the specific information constructed by the specific information constructing unit 232 to the search device 1. It is preferable that the terminal transmitting unit 24 transmits the specific information to the search device 1 when the device processing unit 23 is unable to acquire the object description information.

[0076] The terminal receiving unit 25 receives object description information including the name of the object from the search device 1 in response to the transmission of the identification information.

[0077] The terminal output unit 26 outputs the object description information received by the terminal receiving unit 25. The terminal output unit 26 may also output the object description information acquired by the device processing unit 23.

[0078] Here, output preferably means audio output, but the concept may also include display on a display, projection using a projector, printing on a printer, transmission to an external device, storage on a recording medium, and handing over the processing results to other processing devices or other programs.

[0079] The storage unit 11, object management unit 111, history management unit 112, and terminal storage unit 21 are preferably non-volatile recording media, but may also be realized as volatile recording media.

[0080] There is no restriction on the process by which information is stored in the storage unit 11 etc. For example, information may be stored in the storage unit 11 etc. via a recording medium, information transmitted via a communication line etc. may be stored in the storage unit 11 etc., or information input via an input device may be stored in the storage unit 11 etc.

[0081] The reception unit 12 and the specific information reception unit 121 are preferably realized by wireless or wired communication means, but may also be realized by a means for receiving broadcasts, a device driver for an input means such as a touch panel or keyboard, or control software for a menu screen.

[0082] The processing unit 13, search unit 131, history management unit 112, device processing unit 23, and specific information configuration unit 232 can usually be realized by a processor, memory, etc. The processing procedures of the processing unit 13, etc. are usually realized by software, and the software is recorded on a recording medium such as a ROM. However, they may also be realized by hardware (dedicated circuit). The processor may be a CPU, MPU, GPU, etc., and the type does not matter.

[0083] The output unit 14 is usually realized by a wireless or wired communication means, but may also be realized by driver software for an output device such as a display or speaker, or by a combination of driver software for an output device and the output device.

[0084] The terminal reception unit 22 can be realized by a device driver for an input means such as a touch panel or a keyboard, or control software for a menu screen.

[0085] The user position acquisition unit 231 is realized by, for example, a GPS receiver.

[0086] The terminal transmitting unit 24 is usually realized by a wireless or wired communication means, but may also be realized by a broadcasting means.

[0087] The terminal receiving unit 25 is usually realized by a wireless or wired communication means, but may also be realized by a means for receiving broadcasts.

[0088] The terminal output unit 26 may or may not be considered to include an output device such as a display, a speaker, etc. The terminal output unit 26 may be realized by driver software for an output device, or a combination of driver software for an output device and the output device, etc.

[0089] Next, an example of the operation of the search device 1 will be described with reference to the flowchart of FIG.

[0090] (S301) The identification information receiving unit 121 determines whether or not the identification information has been received. If the identification information has been received, the process proceeds to S302, and if the identification information has not been received, the process proceeds to S305. Here, the identification information receiving unit 121 determines whether or not the identification information has been received from, for example, the user terminal 2. However, the identification information receiving unit 121 may also receive the identification information from the user.

[0091] (S302) The search unit 131 acquires object description information corresponding to the identification information received in S301. An example of such information acquisition processing will be described with reference to the flowcharts of FIGS.

[0092] (S303) The output unit 14 outputs the object description information acquired in S302. Note that here, the output unit 14 transmits, for example, the object description information acquired in S302 to the user terminal 2 that has transmitted the identification information. Note that the output unit 14 may output the object description information acquired in S302 as audio or display it, for example.

[0093] (S304) The storage unit 132 stores the history information. Return to S301. An example of such history accumulation processing will be described with reference to the flowchart of FIG.

[0094] (S305) The reception unit 12 determines whether or not a popularity output instruction has been received. If a popularity output instruction has been received, the process proceeds to S306; if not, the process returns to S301. The reception unit 12 receives the popularity output instruction from the user terminal 2, for example. However, the reception unit 12 may also receive the popularity output instruction from the user.

[0095] (S306) Processing unit 13 creates a popularity list. An example of the popularity list creation process will be described with reference to the flowchart in FIG.

[0096] (S307) The output unit 14 outputs the popularity list created in S306. Return to S301. The output unit 14, for example, transmits the popularity list to the user terminal 2 that transmitted the popularity output instruction. However, the output unit 14 may also output the popularity list as audio or display it.

[0097] In the flowchart of FIG. 3, the process ends when the power is turned off or an interrupt occurs to end the process.

[0098] Next, a first example of the information acquisition process of S302 will be described with reference to the flowchart of Fig. 4. The first example of the information acquisition process is a process of using the object management unit 111 to acquire object description information.

[0099] (S401) The search unit 131 acquires the user position included in the identification information received in S301.

[0100] (S402) The search unit 131 acquires the movement direction information included in the identification information.

[0101] (S403) The search unit 131 acquires the object direction included in the query included in the identification information.

[0102] (S404) The search unit 131 acquires an object region using the user position, the movement direction information, and the object direction.

[0103] The object area is an area where an object exists. The object area is an area in the direction indicated by the object direction (for example, "forward," "right," or "left") relative to the position specified by the user position and the movement direction indicated by the movement direction information. The object area is, for example, an area within a threshold distance from the position specified by the user position. The area in the direction indicated by the object direction is, for example, an area within a 40-degree range or a 60-degree range. The object area is, for example, position information (for example, (latitude, longitude)) of each vertex of a rectangular area. The object area has, for example, an angle with the user position (for example, 45 degrees, 60 degrees) and orientation information indicating the orientation of the center of the area (for example, an angle with respect to true north). In such a case, the object area has, for example, a fan-shaped shape.

[0104] (S405) The search unit 131 acquires the object outer shape included in the identification information.

[0105] (S406) The search unit 131 assigns 1 to the counter i.

[0106] (S407) The search unit 131 determines whether or not the i-th object information exists in the object management unit 111. If the i-th object information exists, the process proceeds to S408, and if not, the process proceeds to S414.

[0107] (S408) The search unit 131 acquires the object position included in the i-th object information from the object management unit 111.

[0108] (S409) The search unit 131 determines whether the object position acquired in S408 is within the range of the area specified by the object area acquired in S404. If the object position is within the range of the area specified by the object area, the process proceeds to S410, and if not, the process proceeds to S413.

[0109] (S410) Search unit 131 acquires from object management unit 111 one or more object attribute values ​​(for example, type of object, shape of object, color of object) included in the i-th object information.

[0110] (S411) The search unit 131 determines whether or not one or more object attribute values ​​acquired in S410 satisfy the object contour acquired in S405. If they satisfy the object contour, the process proceeds to S412, and if they do not satisfy the object contour, the process proceeds to S413.

[0111] (S412) The search unit 131 temporarily stores the i-th object information in a buffer (not shown).

[0112] (S413) The search unit 131 increments the counter i by 1. The process returns to S407.

[0113] (S414) The search unit 131 assigns 1 to the counter j.

[0114] (S415) The search unit 131 determines whether the j-th object information exists in a buffer (not shown). If the j-th object information exists, the process proceeds to S416, and if not, the process proceeds to S418.

[0115] (S416) Search unit 131 acquires a score using the j-th object information, and stores the score in association with the j-th object information. An example of the score acquisition process will be described with reference to the flowchart in FIG.

[0116] The score is the degree of matching with the query, and is a value using the distance from the user position. Here, the shorter the distance between the object position and the user position, the higher the score acquired by the search unit 131 for the object.

[0117] (S417) The search unit 131 increments the counter j by 1. The process returns to S415.

[0118] (S418) The search unit 131 acquires one or more pieces of object information that are paired with the score acquired in S416 and that satisfies the selection condition.

[0119] (S419) The search unit 131 uses one or more pieces of object information acquired in S418 to acquire object description information that explains the object for each piece of object information, and then returns to the upper-level processing.

[0120] 4, the search unit 131 acquires object information corresponding to a score that satisfies the selection condition. However, the search unit 131 may acquire object information using another algorithm, such as selecting only objects whose object positions are closest to the user position. Furthermore, the search unit 131 may ultimately acquire all of the object information temporarily accumulated in a buffer (not shown) in S412.

[0121] Next, an example of the score acquisition process in S416 will be described with reference to the flowchart in FIG.

[0122] (S501) The search unit 131 acquires the user position.

[0123] (S502) The search unit 131 acquires the position of the object.

[0124] (S503) The search unit 131 calculates the distance between the user terminal 2 and the object using the user position and the object position.

[0125] (S504) The search unit 131 acquires the size of the object, which is an example of an object attribute value.

[0126] (S505) The search unit 131 obtains a score using the distance and size, and returns to the upper-level process.

[0127] Note that the smaller the distance, the larger the score obtained by the search unit 131. Also, the larger the size, the larger the score obtained by the search unit 131. The search unit 131 calculates the score, for example, using a decreasing function with the distance as a parameter and an increasing function with the size as a parameter.

[0128] 5, the score may be obtained using an object attribute value other than the size. The score obtaining method and algorithm are not limited.

[0129] Next, a second example of the information acquisition process of S302 will be described with reference to the flowchart of Fig. 6. The second example of the information acquisition process is a process of acquiring object description information using a generation AI.

[0130] (S601) The search unit 131 acquires the direction of the object.

[0131] (S602) The search unit 131 determines whether the direction indicated by the object direction acquired in S601 is "forward." If the direction is "forward," the process proceeds to S603, and if the direction is not "forward," the process proceeds to S604.

[0132] (S603) The search unit 131 acquires a front image from one or more images included in the received user information. For example, the search unit 131 cuts out a front image (for example, an image 30 degrees to the left and right of the reference direction (directly in front) (60 degrees in total)) from a wide-angle image (for example, an image taken with a 360-degree camera). For example, the search unit 131 selects an image from the camera capturing the front from among the two or more cameras.

[0133] (S604) The search unit 131 determines whether the direction indicated by the object direction acquired in S601 is "right." If the direction is "right," the process proceeds to S605, and if the direction is not "right," the process proceeds to S606.

[0134] (S605) The search unit 131 acquires an image of the right side from the image included in the received user information. For example, the search unit 131 cuts out an image of the right side (for example, an image from 15 degrees to 75 degrees (60 degrees in total) to the right of the reference direction (directly in front)) from a wide-angle image (for example, an image taken with a 360-degree camera). For example, the search unit 131 selects an image taken by a camera capturing the right side from among two or more cameras.

[0135] (S606) The search unit 131 determines whether the direction indicated by the object direction acquired in S601 is "left." If the direction is "left," the process proceeds to S607, and if the direction is not "left," the process proceeds to S608. Note that here, if the direction is not "left," the direction is usually "backward."

[0136] (S607) The search unit 131 acquires an image of the left side from the images included in the received user information. For example, the search unit 131 cuts out an image of the left side (for example, an image from 15 degrees to 75 degrees (60 degrees in total) to the left of the reference direction (directly in front)) from a wide-angle image (for example, an image taken with a 360-degree camera). For example, the search unit 131 selects an image from a camera capturing the left side of the image from two or more cameras.

[0137] (S608) The search unit 131 acquires a rearward image from the images included in the received user information. For example, the search unit 131 cuts out a rearward image (for example, an image from 150 degrees to 210 degrees (60 degrees in total) on the left side of the reference direction (directly ahead)) from a wide-angle image (for example, an image taken with a 360-degree camera). For example, the search unit 131 selects an image from a camera capturing a rearward image from among two or more cameras.

[0138] (S609) The search unit 131 acquires the outer shape of the object.

[0139] (S610) The search unit 131 constructs a prompt having the image acquired in S603, S605, S607, or S608 and the object outline acquired in S609. A prompt template, which is a template for constructing a prompt, is, for example, "What is the object in the image below that is <object outline>? <image>". Note that the <object outline> of the prompt is a variable into which a character string of the object outline is placed. The <image> is a variable into which an image is placed. The prompt template is stored in the storage unit 11.

[0140] (S611) The search unit 131 passes the prompt constructed in S610 to the generation AI.

[0141] (S612) The search unit 131 determines whether or not an answer has been acquired from the generation AI. If an answer has been acquired, the process proceeds to S613, and if an answer has not been acquired, the process returns to S612.

[0142] (S613) The search unit 131 acquires object description information using the answer acquired in S612, and returns to the upper-level processing.

[0143] Next, an example of the history accumulation process in S304 will be described with reference to the flowchart in FIG.

[0144] (S701) The accumulation unit 132 acquires the name of the object contained in the output object description information.

[0145] (S702) The accumulation unit 132 acquires current time information from a clock (not shown).

[0146] (S703) The accumulation unit 132 increments by 1 the output count of the history management unit 112, which is the output count paired with the name of the object acquired in S701.

[0147] (S704) The storage unit 132 stores the time information acquired in S702 in pair with the name of the object in the history management unit 112. The process returns to the upper level process.

[0148] Next, an example of the popularity list construction process in S306 will be described with reference to the flowchart in FIG.

[0149] (S801) The processing unit 13 acquires one or more user positions included in the popularity output instruction.

[0150] (S802) Using the user location acquired in S801, the processing unit 13 acquires conditions including range information that specifies the locational range of objects that can be included in the popularity list. The range information is, for example, information that specifies a range that is within a threshold distance from the user location. The range information is, for example, information that specifies a range within a threshold distance from the user location and a range in the user's movement direction. The conditions including range information are, for example, conditions included in the popularity output instruction. The conditions included in the popularity output instruction include, for example, the type of object.

[0151] (S803) The processing unit 13 assigns 1 to the counter i.

[0152] (S804) The processing unit 13 determines whether or not the i-th object information exists in the object management unit 111. If the i-th object information exists, the process proceeds to S805, and if not, the process proceeds to S809.

[0153] (S805) The processing unit 13 acquires the object position and the like contained in the i-th object information from the object management unit 111. Note that the object position and the like is the object position and one or more object attribute values.

[0154] (S806) The processing unit 13 determines whether the object position etc. acquired in S805 satisfies the condition acquired in S802. If the condition is satisfied, the process proceeds to S807, and if not, the process proceeds to S808.

[0155] Here, the processing unit 13 determines, for example, whether the object position acquired in S805 is within the range indicated by the range information acquired in S802, and whether one or more object attribute values ​​satisfy the conditions acquired in S802.

[0156] (S807) Processing unit 13 acquires the i-th object information. Processing unit 13 also acquires the number of outputs paired with the name of the object contained in the i-th object information from history management unit 112. Processing unit 13 associates the i-th object information with the number of outputs and temporarily stores them in a buffer (not shown).

[0157] (S808) The processing unit 13 increments the counter i by 1. The process returns to S804.

[0158] (S809) The processing unit 13 sorts the object information in a buffer (not shown) in descending order using the number of output times as a key.

[0159] (S810) The processing unit 13 acquires one or more pieces of object information corresponding to the number of output times that satisfy the selection condition from the object information sorted in S809, and uses the one or more pieces of object information to create a popularity list of the object. The processing unit 13 then returns to the upper processing. The selection condition may be, for example, that the ranking of the number of output times is equal to or greater than a threshold, or that the number of output times is equal to or greater than a threshold.

[0160] Next, an example of the operation of the user terminal 2 will be described with reference to the flowchart of FIG.

[0161] (S901) The terminal reception unit 22 determines whether or not a voice query has been received from the user. If a query has been received, the process proceeds to S902, and if not, the process proceeds to S912.

[0162] (S902) The specific information construction unit 232 performs speech recognition on the query received in S901 and acquires a character string. The specific information construction unit 232 acquires the object direction (for example, "front," "right," or "left") from the character string.

[0163] (S903) The specific information configuration unit 232 acquires the object outline from the character string acquired in S902.

[0164] (S904) The device processing unit 23 acquires user information. An example of the user information acquisition process will be described with reference to the flowcharts of FIGS.

[0165] (S905) The identification information constructing unit 232 constructs identification information having the object direction, the object outer shape, and user information.

[0166] (S906) The device processing unit 23 performs information acquisition processing using the specific information configured in S905. This information acquisition processing may be the same processing as the information acquisition processing described using the flowchart of Fig. 4 or 6. However, here, the device processing unit 23 may not be able to acquire the object description information.

[0167] (S907) The terminal processing unit 23 determines whether or not the object description information was acquired in S906. If the object description information was acquired, the process proceeds to S908, and if not, the process proceeds to S910.

[0168] (S908) The terminal output unit 26 outputs the object description information acquired in S907 or S911.

[0169] (S909) The terminal processing unit 23 performs a history accumulation process. This history accumulation process may be the same as the history accumulation process described using the flowchart in FIG.

[0170] (S910) The terminal transmitting unit 24 transmits the identification information constructed in S905 to the retrieval device 1.

[0171] (S911) The terminal receiving unit 25 determines whether or not it has received object description information. If it has received object description information, it proceeds to S908, and if it has not received object description information, it returns to S911.

[0172] (S912) The terminal reception unit 22 determines whether or not a popularity output instruction has been received from the user. If a popularity output instruction has been received, the process proceeds to S913, and if not, the process returns to S901.

[0173] (S913) The device processing unit 23 composes a popularity output instruction to be transmitted. The terminal transmitting unit 24 transmits the popularity output instruction to the search device 1.

[0174] (S914) Terminal receiving unit 25 determines whether or not the popularity list has been received. If the popularity list has been received, the process proceeds to S915; if not, the process returns to S914.

[0175] (S915) Device processing unit 23 creates a popularity list to be output. Device output unit 26 outputs the popularity list. Return to S901.

[0176] In the flowchart of FIG. 9, it is preferable that the history accumulation process be performed by the search device 1.

[0177] In the flowchart of FIG. 9, the process ends when the power is turned off or an interrupt occurs to end the process.

[0178] Next, a first example of the user information acquisition process in S904 will be described with reference to the flowchart in FIG.

[0179] (S1001) The user position acquisition unit 231 acquires two or more user positions that are consecutive in time series.

[0180] (S1002) The identification information configuration unit 232 acquires movement direction information using two or more user positions. Note that the two or more user positions are user positions in time series.

[0181] (S1003) The specific information configuration unit 232 configures user information having the latest user position and the movement direction information acquired in S1002, and returns to the upper level processing.

[0182] In the flowchart of Figure 10, the identification information configuration unit 232 may use known navigation technology to identify the road on which the vehicle is traveling, and use information about the road to obtain travel direction information based on the map database used by the navigation system.

[0183] Next, a second example of the user information acquisition process in S904 will be described with reference to the flowchart in FIG.

[0184] (S1101) The identification information configuration unit 232 acquires the object direction acquired in S902.

[0185] (S1102) The identification information construction unit 232 determines a camera corresponding to the object direction. Note that here, if there is one camera, the identification information construction unit 232 determines that one camera. If there are two or more cameras, each camera corresponds to the object direction.

[0186] (S1103) The specific information configuration unit 232 acquires the image captured by the camera determined in S1102. Note that it is preferable that the image here is a still image.

[0187] (S1104) The identification information configuration unit 232 determines whether it is necessary to cut out an image in the direction that identifies the object direction from the image acquired in S1103. If it is necessary to cut out an image, the process proceeds to S1105; if it is not necessary, the process returns to the upper processing.

[0188] (S1105) The specific information configuration unit 232 cuts out an image of a predetermined range from the image acquired in S1103, with the direction indicated by the object direction as the center, and returns to the upper-level processing.

[0189] Specific examples of the operation of the search system A in this embodiment will be described below. Two specific examples will be described below. Specific example 1 is a case where the object management unit 111 is used. Specific example 2 is a case where a generation AI is used.

[0190] (Example 1) The object management unit 111 of the search device 1 stores an object management table shown in FIG. 12. The object management table is a table for managing object information. The object management table includes "ID," "object identifier," "name," "type," and "object attribute value." The "object attribute value" includes "object position," "object outer shape," and "object feature information." The "object outer shape" includes "color" and "size." The "object feature information" is information that indicates the characteristics of an object. The type of information included in the "object feature information" differs depending on the type of object. The type of information included in the "object feature information" may differ depending on the object. The information included in the "object feature information" may include information classified as object outer shape.

[0191] The history management unit 112 stores a history management table shown in Fig. 13. The history management table is a table for managing history information. The history management table has "ID," "object identifier," "name," "number of outputs," "time information," "object position," and "size."

[0192] In such a situation, suppose that a user U driving a car becomes curious about a mountain he sees ahead while traveling in the car, and utters, "What is that snow-covered mountain ahead?" as shown in Figure 14.

[0193] Then, the terminal reception unit 22 of the user terminal 2 (for example, a navigation terminal) mounted on the vehicle receives the voice "What is that snow-covered mountain ahead?". Next, the device processing unit 23 performs voice recognition processing on the voice and obtains the character string "What is that snow-covered mountain ahead?". Next, the device processing unit 23 analyzes the character string and constructs a query "<object direction> front <object outer shape> snow <type> mountain".

[0194] For example, the device processing unit 23 acquires the object direction "front" when the character string contains the character "front" (e.g., "forward"), acquires the object direction "left" when the character string contains the character "left" (e.g., "left side", "left hand"), acquires the object direction "right" when the character string contains the character "right" (e.g., "right side", "right hand"), and acquires the object direction "back" when the character string contains the character "back" (e.g., "backward").

[0195] Furthermore, the user position acquisition unit 231 successively acquires two or more user positions. Next, the specific information configuration unit 232 acquires movement direction information (for example, an angle of "90 degrees" with respect to true north = eastward) using the two or more user positions. The specific information configuration unit 232 acquires the latest user position (x n ,y n ,z n ) and the acquired moving direction information "90 degrees" and the user information "<user position> (x n ,y n ,z n ) <Movement direction information> 90 degrees.

[0196] Next, the specific information configuration unit 232 generates specific information “<object direction> forward <object outer shape> snow <type> mountain <user position> (x n ,y n ,z n ) <Movement direction information> 90 degrees". Note that "<Type> Mountain" can also be considered to be included in the object outline.

[0197] Next, the device processing unit 23 performs information acquisition processing using the specified information. Here, it is assumed that the device processing unit 23 first performs an information search for the local object management unit 111 held by the user terminal 2, but is unable to acquire object description information.

[0198] Next, the terminal transmitting unit 24 transmits the specific information “<direction of object> forward <outline of object> snow <type> mountain <user position> (x n ,y n ,z n) <Movement direction information> 90 degrees" is transmitted to the retrieval device 1.

[0199] Next, the specific information receiving unit 121 of the search device 1 receives the specific information. Next, the search unit 131 performs information acquisition processing using the specific information (see the flowchart in FIG. 4).

[0200] The search unit 131 then searches the object management table (FIG. 12) and determines that the record (object information) of "ID=1" matches the received identification information. Then, using the "name" and "object characteristic information" of "ID=1", the search unit 131 obtains the object description information "Mount Fuji. Its elevation is 3,776 m and the mine is currently closed."

[0201] Next, the output unit 14 transmits the object description information to the user terminal 2 that has transmitted the identification information.

[0202] Next, the terminal receiving unit 25 of the user terminal 2 receives the object description information "Mount Fuji. Its elevation is 3,776 m, and the mountain is currently closed." Next, the device processing unit 23 converts the object description information into voice. Next, the terminal output unit 26 outputs the object description information as voice (see FIG. 15). Note that it is preferable that the device processing unit 23 acquires an image when accepting a query. It is also preferable that the terminal output unit 26 display the image, as in 1501 of FIG. 15. It is also preferable that the terminal output unit 26 clearly indicate the object in the image on the display, as in 1502 of FIG. 15.

[0203] The storage unit 132 of the search device 1 also stores the history information described using the flowchart in Fig. 7. That is, the storage unit 132 acquires the name of the object, "Mount Fuji," contained in the output object description information. The storage unit 132 also acquires current time information from a clock (not shown). The storage unit 132 also increments the output count paired with the object name, "Mount Fuji," in the history management table (Fig. 13) by 1 to "21,822."

[0204] (Example 2) 13 is stored in the history management unit 112. In this case, the search device 1 does not have the object management unit 111.

[0205] In such a situation, suppose that a user U driving a car becomes curious about a mountain he sees ahead while traveling in the car, and utters, "What is that snow-covered mountain ahead?" as shown in Figure 14.

[0206] Then, the terminal reception unit 22 of the user terminal 2 (for example, a navigation terminal) mounted on the vehicle receives the query "What is that snow-covered mountain ahead?". Next, the device processing unit 23 performs speech recognition processing on the speech and acquires the query string "What is that snow-covered mountain ahead?".

[0207] Next, the search unit 131 acquires the object direction "front" from the character string "What is the snow-covered mountain ahead?".

[0208] Next, the device processing unit 23 performs the user information acquisition process described using the flowchart in FIG. 11. That is, the device processing unit 23 acquires an image captured by an in-vehicle camera (for example, a drive recorder (not shown) or a camera included in the user terminal 2). Here, the image is, for example, an image captured by a 360-degree camera. Next, the device processing unit 23 cuts out an image of an area "forward" from the image, the area having a 60-degree angle of view. Next, the device processing unit 23 corrects the cut-out image and acquires the corrected image. Note that this corrected image is the user information here.

[0209] Next, the specific information constructing unit 232 constructs specific information having a query string "What is that snow-covered mountain ahead?" including the object direction and the object outline, and user information which is the corrected image.

[0210] Next, the terminal output unit 26 transmits the specified information to the search device 1.

[0211] Next, the specific information receiving unit 121 of the search device 1 receives the specific information. Next, the search unit 131 performs the search process described using the flowchart in FIG.

[0212] That is, the search unit 131 acquires the query "What is that snow-covered mountain ahead?" contained in the specific information. The search unit 131 also acquires the image contained in the specific information. Next, the search unit 131 constructs a prompt including the query and the image. An example of such a prompt is shown in FIG. 16. Next, the search unit 131 passes the prompt to the generation AI.

[0213] Next, the search unit 131 receives a response from the generation AI: "This image shows Mount Fuji, a symbol of Japan, due to its distinctive shape and snow-capped peak. Mount Fuji is known for its beauty and can be seen in many photos regardless of the season. In this image, Mount Fuji stands majestically against the backdrop of a clear blue sky, and the road in the foreground accentuates its beauty." Here, this response is considered to be object description information.

[0214] Next, the output unit 14 transmits the object description information to the user terminal 2.

[0215] Next, the terminal receiving unit 25 of the user terminal 2 receives the object description information. Next, the terminal processing unit 23 converts the object description information into voice. Next, the terminal output unit 26 outputs the object description information as voice.

[0216] The accumulation unit 132 of the search device 1 accumulates the history information described using the flowchart in Fig. 7. That is, the accumulation unit 132 acquires the name of the object "Mount Fuji" contained in the output object description information. The accumulation unit 132 also acquires current time information from a clock (not shown). The accumulation unit 132 also increments the output count paired with the object name "Mount Fuji" in the history management table (Fig. 13) by 1 to "21822".

[0217] As described above, according to this embodiment, it is possible to easily obtain information about an object that is within the user's field of view, taking into consideration the user's moving direction.

[0218] The processing in this embodiment may be implemented by software. This software may be distributed by software download or the like. Furthermore, this software may be recorded on a recording medium such as a CD-ROM and distributed. This also applies to other embodiments in this specification. The software implementing the search device 1 in this embodiment is the following program. That is, this program causes a computer to function as: a specific information receiving unit that receives a query received while a user is moving, the query having an object direction relating to the direction in which an object exists relative to the user's moving direction and an object outer shape relating to the outer shape of the object; and specific information having user information that can be acquired while the user is moving; a search unit that acquires object description information including the name of the object identified by the specific information; and an output unit that outputs the object description information acquired by the search unit.

[0219] The software that realizes the user terminal 2 is the following program. That is, this program causes a computer to: a terminal reception unit that receives, while the user is moving, a query having an object direction relating to the direction of the object relative to the user's position and an object outline relating to the outline of the object; a user position acquisition unit that acquires a user position that identifies the location of the user; a specific information configuration unit that configures specific information including the query and two or more user positions or movement direction information based on the two or more user positions acquired by the user position acquisition unit; a terminal transmitting unit that transmits the identification information to the search device according to any one of claims 1 to 6; a terminal receiving unit that receives object description information including the name of the object from the search device; The program causes the terminal receiving unit to function as a terminal output unit that outputs the object description information received by the terminal receiving unit.

[0220] FIG. 17 is a block diagram of a computer system 300 that executes the programs described in this specification to realize the search device 1 or the user terminal 2 of the various embodiments described above.

[0221] In FIG. 17, a computer system 300 includes a computer 301 including a CD-ROM drive, a keyboard 302, a mouse 303, and a monitor 304.

[0222] 17, computer 301 includes, in addition to CD-ROM drive 3012, MPU 3013, bus 3014 connected to CD-ROM drive 3012 etc., ROM 3015 for storing programs such as a boot-up program, RAM 3016 connected to MPU 3013 for temporarily storing instructions of application programs and providing temporary storage space, and hard disk 3017 for storing application programs, system programs, and data. Although not shown here, computer 301 may further include a network card for providing connection to a LAN.

[0223] A program that causes computer system 300 to execute functions such as the search device of the above-described embodiments may be stored on CD-ROM 3101, inserted into CD-ROM drive 3012, and then transferred to hard disk 3017. Alternatively, the program may be transmitted to computer 301 via a network (not shown) and stored on hard disk 3017. The program is loaded into RAM 3016 when executed. The program may also be loaded directly from CD-ROM 3101 or the network.

[0224] The program does not necessarily include an operating system (OS) or a third-party program that causes the computer 301 to execute the functions of the search device and the like of the above-described embodiments. The program only needs to include instructions that call appropriate functions (modules) in a controlled manner to achieve the desired results. How the computer system 300 operates is well known, and a detailed description thereof will be omitted.

[0225] In addition, in the above program, the steps of transmitting information and receiving information do not include processing performed by hardware, such as processing performed by a modem or interface card in the transmission step (processing that can only be performed by hardware).

[0226] The computer that executes the program may be a single computer or a plurality of computers, that is, it may perform centralized processing or distributed processing.

[0227] Furthermore, in each of the above embodiments, it goes without saying that two or more communication means present in one device may be physically realized by one medium.

[0228] Furthermore, in each of the above embodiments, each process may be realized by centralized processing in a single device, or may be realized by distributed processing in a plurality of devices.

[0229] The present invention is not limited to the above-described embodiment, and various modifications are possible, and it goes without saying that these modifications are also included within the scope of the present invention. [Industrial Applicability]

[0230] As described above, the search device 1 of the present invention has the effect of easily obtaining information about objects within the user's field of view, taking into account the user's direction of movement, and is useful as a server that provides information to a user terminal 2, etc. [Explanation of symbols]

[0231] A Search System 1 Search device 2. User terminal 11 Storage area 12 Reception 13 Processing section 14 Output section 21 Terminal storage section 22 Terminal Reception 23 Terminal processing section 24 Terminal transmitter 25 Terminal receiving section 26 Terminal Output Unit 111 Object Management Department 112 History Management Department 121 Specific Information Reception Department 131 Search Department 132 Storage Unit 132 Sakube 231 User location acquisition unit 232 Specific information component

Claims

1. a specific information receiving unit that receives a query received while a user is moving, the query having an object direction related to a direction in which an object exists relative to the user's moving direction and an object outer shape related to the outer shape of the object, and specific information having user information that can be acquired while the user is moving; a search unit that acquires object description information including the name of the object identified by the identification information; an output unit that outputs the object description information acquired by the search unit.

2. The specific information receiving unit receiving the query and the identification information having user information including a user position that identifies the position of the user and movement direction information that can identify the movement direction of the user; The search unit The search device of claim 1, wherein an object management unit stores two or more pieces of object information each having a name of an object and an object position that specifies the position of the object, and determines an object that exists in the movement direction specified by the movement direction information from the user position stored in the user information using the object position, and obtains the object description information including the name of the object.

3. The object management unit The search device according to claim 1 , wherein two or more pieces of object information are stored, the object information being acquired based on video with location information acquired while one or more users are moving.

4. The specific information receiving unit A search device as described in claim 2 or claim 3, wherein when a user terminal receives a query having an object direction and an object shape while the user is moving and is unable to obtain the object description information corresponding to the query, the search device receives specific information including the user information from the user terminal.

5. receiving the specific information having user information that is an image taken while the user is moving; The search unit 2. The search device according to claim 1, wherein a prompt including the image and the object outline is given to a generating AI, and the object description information is obtained from the generating AI.

6. a terminal receiving unit that receives a query having an object direction relating to a direction of an object relative to a position of the user and an object outline relating to an outline of the object while the user is moving; a user position acquisition unit that acquires a user position that identifies the location of the user; a specific information configuration unit that configures specific information including the query and two or more user positions or movement direction information based on the two or more user positions acquired by the user position acquisition unit; a terminal transmitting unit that transmits the specific information to the search device according to any one of claims 1 to 5; a terminal receiving unit that receives object description information including the name of the object from the search device; a terminal output unit that outputs the object description information received by the terminal receiving unit.

Images