Search system, search method, and search device

Abstract

This search system acquires a search condition that is at least one condition related to an object and stochastically represented, and extracts information on an object satisfying the search condition from virtual space information including information on one or more objects in a target virtual space. The virtual space information stochastically represents the position of each object in the target virtual space.

Description

Search system, search method, and search device

[0001] The present disclosure relates to a search system, a search method, and a search device.

[0002] Technologies for representing objects in a virtual space have been developed. For example, Patent Literature 1 discloses a technology for capturing an image of an object using a camera and generating a three-dimensional model of the object using image data obtained by capturing the image.

[0003] Japanese Patent Application Laid-Open No. 2004-185123

[0004] In generating 3D model data in the invention of Patent Document 1, the position and size of an object are determined as specific values. Therefore, there is a possibility that information about errors that occur in the process of generating the 3D model, such as camera observation errors, may be lost. The present disclosure has been made in consideration of the above-mentioned problems, and one of its purposes is to provide a new technology for handling information about objects in a virtual space.

[0005] The search system of the present disclosure includes an acquisition unit that acquires search conditions in which at least one condition related to an object is expressed probabilistically, and a search execution unit that extracts information related to an object that satisfies the search conditions from virtual space information that includes information on one or more objects in a target virtual space. The virtual space information probabilistically represents the position of each of the objects in the target virtual space.

[0006] A search method according to the present disclosure is executed by a computer. The search method includes an acquisition step of acquiring search criteria in which at least one condition related to an object is probabilistically expressed, and a search execution step of extracting information related to an object that satisfies the search criteria from virtual space information that includes information on one or more objects in a target virtual space. The virtual space information probabilistically expresses the position of each of the objects in the target virtual space.

[0007] The search device of the present disclosure includes an acquisition unit that acquires search conditions in which at least one condition related to an object is expressed probabilistically, and a search execution unit that extracts information related to an object that satisfies the search conditions from virtual space information that includes information on one or more objects in a target virtual space, the virtual space information probabilistically representing the position of each of the objects in the target virtual space.

[0008] According to the present disclosure, a new technique for handling information about objects in a virtual space is provided.

[0009] FIG. 1 is a diagram illustrating an overview of the operation of a search system. FIG. 2 is a block diagram illustrating an example of the functional configuration of a search system. FIG. 3 is a block diagram illustrating an example of the functional configuration of a search device. FIG. 4 is a block diagram illustrating an example of the hardware configuration of a computer that realizes the search system. FIG. 5 is a flowchart illustrating an example of the flow of processing executed by the search system. FIG. 6 is a diagram illustrating an example of the configuration of position information. FIG. 7 is a diagram illustrating an example of the configuration of state information. FIG. 8 is a diagram illustrating an example of a probability distribution representing the orientation of an object on a horizontal plane. FIG. 9 is a diagram illustrating an example of the configuration of attribute information. FIG. 10 is a diagram illustrating an example of the spatial distribution of object attributes. FIG. 11 is a diagram illustrating an example of a situation in which virtual space information is generated.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and duplicate explanations will be omitted as necessary for clarity. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0011] <Overview> Fig. 1 is a diagram illustrating an example of an overview of the operation of the search system 2000. Here, Fig. 1 is a diagram for facilitating understanding of the overview of the search system 2000, and the operation of the search system 2000 is not limited to the operation shown in Fig. 1. Furthermore, the configuration of the virtual space information 10 shown in Fig. 1 is not limited to the configuration shown in Fig. 1.

[0012] The search system 2000 extracts information about objects that satisfy search conditions from the virtual space information 10. The search conditions represent conditions for extracting information from the virtual space information 10. The virtual space information 10 indicates, for each of one or more objects, information about one or more objects in a predetermined virtual space (hereinafter, referred to as a target virtual space).

[0013] The target virtual space may be a space that imitates real space, or a space that has been modified from a virtual space that imitates real space. In the former case, for example, objects in real space are observed by various sensors, and the target virtual space is constructed using the observation results. On the other hand, in the latter case, the target virtual space is constructed by adding objects that do not exist in real space to a virtual space generated using the observation results from the sensors, or by changing part of the information about objects that exist in real space. Here, the method of constructing the target virtual space will be described in detail later.

[0014] The virtual space information 10 indicates at least position information 20. The position information 20 indicates the position of one or more objects in the target virtual space probabilistically. Here, one method of expressing information probabilistically is to express the information as a probability distribution. The position of each object is indicated for each of a plurality of time points. For example, in FIG. 1, the position information 20 indicates the probability distribution of the positions of objects A001 and A002 for each of a plurality of time points.

[0015] The virtual space information 10 may further include state information 30 indicating the state of each object, attribute information 40 indicating the attributes of each object, or both. The state of an object indicated in the state information 30 may be posture, speed, or orientation. The attributes of an object indicated in the attribute information 40 may be type, shape, size, color, or brightness. Note that the virtual space information 10 may further indicate information other than the position, state, or attributes of each object.

[0016] In the state information 30, the state of an object may be represented deterministically or probabilistically. In the former case, the speed of the object is represented by a specific value such as "1.0 m / sec." In the latter case, the speed of the object is represented by a probability distribution as shown in FIG. 1.

[0017] The state information 30 may represent both a deterministic state and a probabilistic state. For example, if the state information 30 represents the speed and orientation of an object, the state information 30 may represent the speed of the object deterministically while representing the orientation of the object probabilistically. The state information 30 may also represent one or more states as information that changes over time.

[0018] Similarly, the attributes of an object may be represented deterministically or probabilistically in the attribute information 40. The attribute information 40 may indicate both attributes that are represented deterministically and attributes that are represented probabilistically. The attribute information 40 may also represent one or more attributes as information that changes over time.

[0019] The search system 2000 extracts information about objects that satisfy search conditions from the virtual space information 10. Search conditions related to information that is probabilistically represented in the virtual space information 10 can be expressed using probability.

[0020] For example, suppose a user of the search system 2000 wishes to obtain information about an object moving eastward at a speed of 1.0 m / sec or greater. In this case, the user specifies a search condition such as "a speed of 1.0 m / sec or greater and a direction toward the east that is met with a probability of 50% or greater." An object that satisfies this search condition is an object for which the joint probability of having a speed of 1.0 m / sec or greater and a direction toward the east is 50% or greater. Hereinafter, the probability threshold (50% in the above example) used in a search using the search system 2000 is referred to as the probability threshold. Note that, as will be described later, a time point or a period may also be specified as a search condition.

[0021] The probability threshold may be predetermined in the search system 2000 or may be specified at the time of the search. Furthermore, if the search conditions include multiple conditions, the probability threshold may be specified individually for each condition or for a combination of multiple conditions. Further explanation of the probability threshold will be provided later.

[0022] <Example of Effects> In the 3D model data of Patent Document 1, the positions of objects and the like are deterministically represented. Therefore, Patent Document 1 does not anticipate a situation in which the positions of objects in a virtual space are represented probabilistically.

[0023] In this regard, the virtual space information 10 probabilistically represents the position of each object in the target virtual space. The virtual space information 10 can also probabilistically represent the state and attributes of the object. The search system 2000 acquires search conditions expressed using probability and extracts information about objects that satisfy the search conditions from the virtual space information 10. As a result, the search system 2000 can obtain information about objects that satisfy specified conditions in a situation where information about objects in a virtual space is expressed probabilistically. In this way, the search system 2000 provides a new technology for handling objects in a virtual space.

[0024] One of the reasons why the positions of objects and the like are represented probabilistically in the virtual space information 10 is to manage the information about the objects while taking into account various errors that occur when generating the virtual space information 10. For example, when an object in real space is observed by a sensor, the sensor data obtained from the sensor may contain observation errors. Furthermore, errors may also be included in the identification process performed using the sensor data and the process of converting the identification results into output values. In order to be able to take these errors into account, it is preferable to treat the information obtained using the observation results from the sensor as probabilistic information.

[0025] However, when object information is expressed probabilistically, searching for information about an object without considering the probability can result in too many objects satisfying the search criteria. For example, suppose a search is performed with the search criteria "location is (x1, y1, z1)." This search criteria will be satisfied by all objects whose probability of being at (x1, y1, z1) is greater than 0. However, this is thought to be different from the search results desired by the user.

[0026] In this regard, the search system 2000 accepts search conditions expressed using probabilities, such as "the probability that the location is (x1, y1, z1) is 50% or more." This makes it possible to search for information about objects expressed probabilistically using conditions that reflect the user's intention in more detail or more accurately. Therefore, the search system 2000 can more accurately provide information desired by the user from information about objects expressed probabilistically.

[0027] The search system 2000 of this embodiment will be described in more detail below.

[0028] 2 is a block diagram illustrating an example of the functional configuration of the search system 2000. The search system 2000 has an acquisition unit 2020 and a search execution unit 2040. The acquisition unit 2020 acquires search conditions. The search execution unit 2040 extracts information about objects that satisfy the search conditions from the virtual space information 10.

[0029] Here, the search system 2000 may be realized by a single device. The device that realizes the search system 2000 is called a search device. Fig. 3 is a block diagram illustrating an example of the functional configuration of the search device. The search device 3000 has an acquisition unit 2020 and a search execution unit 2040.

[0030] <Example of Hardware Configuration> Each functional component of search system 2000 may be realized by hardware that realizes the respective functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a case where each functional component of search system 2000 is realized by a combination of hardware and software will be further described.

[0031] 4 is a block diagram illustrating an example of the hardware configuration of a computer 1000 that realizes the search system 2000. The computer 1000 is any computer. For example, the computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 1000 may be a portable computer such as a smartphone or a tablet terminal. The computer 1000 may be a dedicated computer designed to realize the search system 2000, or may be a general-purpose computer.

[0032] For example, by installing a predetermined application on the computer 1000, each function of the search system 2000 is realized on the computer 1000. The application is configured as a program for realizing each functional component of the search system 2000. Note that any method for acquiring the program is possible. For example, the program can be acquired from a storage medium (such as a DVD disc or USB memory) on which the program is stored. Alternatively, the program can be acquired by downloading the program from a server device that manages the storage device on which the program is stored.

[0033] The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to bus connection.

[0034] The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device realized using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, a read only memory (ROM), or the like.

[0035] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0036] The network interface 1120 is an interface for connecting the computer 1000 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0037] The storage device 1080 stores a program (a program that realizes the above-mentioned application) that realizes each functional component of the search system 2000. The processor 1040 reads this program into the memory 1060 and executes it to realize each functional component of the search system 2000.

[0038] The search system 2000 may be realized by one computer 1000 or by multiple computers 1000. In the former case, it can be said that the search device 3000 is realized by one computer 1000 that realizes the search system 2000. In the latter case, the configurations of the computers 1000 do not need to be the same, and can be different from each other.

[0039] 5 is a flowchart illustrating the flow of processing executed by the search system 2000. The acquisition unit 2020 acquires search conditions (S102). The search execution unit 2040 extracts information about objects that satisfy the search conditions from the virtual space information 10.

[0040] <Regarding the Virtual Space Information 10> The virtual space information 10 will be described in more detail. As described above, the virtual space information 10 indicates position information 20 that represents the position of each of one or more objects. In the position information 20, the positions of the objects are represented probabilistically. The virtual space information 10 also includes state information 30 that represents the state of each object, attribute information 40 that represents the attributes of each object, or both. For ease of understanding, in the following description, unless otherwise specified, the virtual space information 10 is assumed to include both the state information 30 and the attribute information 40.

[0041] First, the position information 20 will be described. FIG. 6 is a diagram illustrating an example of the configuration of the position information 20. In FIG. 6, the position information 20 indicates presence probability information 50 for each pair (t, P) of time t and position P. The presence probability information 50 corresponding to the pair of time t and position P indicates the presence probability of each object at position P for time t. The presence probability of a certain object at time t and position P indicates the probability that the object exists at position P at time t. If the target virtual space is a three-dimensional space consisting of X, Y, and Z axes, position P can be expressed as (x, y, z). Therefore, the position information 20 in FIG. 6 can also be said to indicate the presence probability of each object for each tuple (t, x, y, z). Note that a tuple represents a set of multiple elements arranged in a predetermined order. In the present disclosure, a pair is a tuple with two elements.

[0042] The presence probability information 50 associates an object identifier 51 with a presence probability 52. ​​The object identifier 51 represents an identifier assigned to an object. The presence probability 52 represents the presence probability of the corresponding object. In FIG. 6, the presence probability information 50 corresponding to (ti, Pj) represents "at time ti, an object with identifier A001 exists at position Pj with a 73% probability" and "at time ti, an object with identifier A002 exists at position Pj with a 30% probability." i and j are the identifiers of the time and position, respectively.

[0043] In Figure 6, the position information 20 represents the spatial distribution of the existence probability of each object at each time point. Therefore, for example, by extracting all of the existence probability information 50 for time point t, the spatial distribution of the existence probability of each object at time point t can be obtained. The spatial distribution of the existence probability of an object at time point t can also be expressed as the probability distribution of the object's position at time point t. In Figure 6, a graph 60 represents the probability distribution of the positions of object A001 and object A002 at time point ti.

[0044] The existence probability information 50 may indicate the existence probabilities of all objects shown in the virtual space information 10, or may indicate the existence probabilities only for objects that satisfy a predetermined condition. In the latter case, for example, the existence probability information 50 indicates the existence probability of only objects whose existence probability is greater than a threshold. For example, if 0 is used as the threshold, information about objects that are certain not to exist at a certain (t, P) can be omitted from the existence probability information 50 for that (t, P).

[0045] The configuration of the position information 20 shown in FIG. 6 is an example, and the configuration of the position information 20 is not limited to the structure shown in FIG. 6. For example, in the virtual space information 10, the position information 20 can be defined for each object. Specifically, the position information 20 for an object is indicated in association with the identifier of the object. The position information 20 indicates a probability distribution of the position of the corresponding object for each of a plurality of points in time. That is, in this case, the probability distribution of the position of object A at point in time t is indicated in association with a pair (A, t) of object A and point in time t.

[0046] It should be noted that the intervals between the multiple time points handled by the virtual space information 10 may or may not be constant.

[0047] Next, a description will be given of the state information 30. Fig. 7 is a diagram illustrating an example of the configuration of the state information 30. In Fig. 7, the state information 30 indicates a posture 31, a speed 32, and a direction 33.

[0048] The posture 31 indicates the posture of the object. For example, when the object is represented as a set of joint points, the posture of the object represents the spatial distribution of the probability of existence of each joint point. More specifically, the posture can be represented as a heat map or a three-dimensional vector field of the positions of each joint point. Furthermore, the set of joint points may be recognized and processed, and expressed using a model name such as a human or a horse and a label name such as an upright posture or a jumping posture, such as a 90% probability of indicating an upright posture for a human and a 10% probability of indicating a jumping posture for a horse.

[0049] The speed 32 indicates the speed of the object. If the speed of the object is represented deterministically, for example, the speed 32 indicates a specific value representing the speed of the object. If the speed of the object is represented probabilistically, for example, the speed 32 indicates a probability distribution of speed values.

[0050] Orientation 33 indicates the orientation of an object. The orientation of an object is expressed, for example, as the orientation on a horizontal plane (in other words, the orientation in a planar view). The orientation on a horizontal plane is expressed, for example, as a bearing. Alternatively, the orientation on a horizontal plane can be expressed, for example, as the magnitude of the angle with a reference direction (for example, the x-axis direction).

[0051] When the orientation of an object is deterministically represented, for example, the orientation 33 indicates a specific bearing or a specific angle of the orientation of the object on a horizontal plane. When the orientation of an object is probabilistically represented, for example, the orientation 33 indicates a probability distribution of the orientation of the object on a horizontal plane or a probability distribution of the size of the angle with a reference direction.

[0052] 8 illustrates a probability distribution representing the orientation of an object on a horizontal plane. Graph 70 shows the probability that the orientation of the object on a horizontal plane is in each of eight different orientations. Graph 80 shows the probability distribution of the magnitude of the angle between the orientation of the object on a horizontal plane and a reference direction. The range of the angle with the reference direction is set to, for example, 0 degrees or more and less than 360 degrees.

[0053] The orientation of an object may further indicate the magnitude of the angle (i.e., elevation angle) formed with the horizontal plane in addition to the orientation on the horizontal plane. When the orientation of an object is expressed deterministically, for example, the orientation 33 indicates the magnitude of the angle formed with the horizontal plane by the orientation of the object with a specific value. When the orientation of an object is expressed probabilistically, for example, the orientation 33 indicates a probability distribution regarding the angle formed with the horizontal plane by the object. The range of the angle formed with the horizontal plane by the object is set to, for example, from 0 degrees to 90 degrees.

[0054] 9 is a diagram illustrating an example of the configuration of the attribute information 40. In FIG. 9, the attribute information 40 indicates a type 41, a shape 42, a size 43, a color 44, and a brightness 45.

[0055] The type 41 indicates the type of object. Various types of objects can be used as the type of object, such as a human, a dog, a cat, a car, a motorcycle, or a bicycle. When the type of object is deterministically represented, for example, the type 41 indicates a specific type. When the type of object is probabilistically represented, for example, the type 41 indicates a probability distribution for the type of object.

[0056] Here, object types may be defined hierarchically. For example, types such as humans, dogs, and cats are defined below the type of animals. Also, types such as cars, motorbikes, and bicycles are defined below the type of vehicles.

[0057] When types are defined hierarchically in this way, for example, type 41 indicates a type corresponding to a leaf in the hierarchical tree. For example, as described above, assume that types such as human, dog, and cat are defined below a type called animal, and that types such as car, motorcycle, and bicycle are defined below a type called vehicle. When types are expressed deterministically, type 41 indicates one of human, dog, cat, car, motorcycle, and bicycle. When types are expressed probabilistically, type 41 indicates the probability that the object type is human, the probability that the object type is dog, the probability that the object type is cat, the probability that the object type is car, the probability that the object type is motorcycle, and the probability that the object type is bicycle, respectively.

[0058] The shape 42 indicates the shape of an object. The shape of the object is approximated by, for example, a polygon. In this case, the shape of the object can be expressed, for example, by a set of vertices of the polygon. More specifically, the shape 42 represents the spatial distribution (heat map) of the existence probability of each vertex of the polygon approximating the object. Furthermore, for example, the shape 42 indicates a group of candidate points estimated using a segmentation technique, and the shape and probability value of the region containing them.

[0059] The size 43 indicates the size of the object. The size of the object is represented, for example, by any one of the width, depth, and vertical lengths. Alternatively, for example, the size of the object may be represented by a combination of any two of the width, depth, and vertical lengths. Alternatively, for example, the size of the object may be represented by any combination of the width, depth, and vertical lengths. However, the size of the object may be represented without using any of the width, depth, or vertical lengths.

[0060] When the size of an object is represented deterministically, for example, size 43 indicates a specific value for the width direction length, etc. On the other hand, when the size of an object is represented probabilistically, for example, size 43 indicates a probability distribution of values ​​representing the size of the object. For example, assume that the size of an object is represented by all combinations of the width direction length, the depth direction length, and the vertical direction length. In this case, size 43 indicates a probability distribution for the width direction length of the object, a probability distribution for the depth direction length of the object, and a probability distribution for the vertical direction length of the object. Alternatively, for example, size 43 may indicate a probability distribution for combinations of the width direction length, the depth direction length, and the height direction length.

[0061] The size of an object may be represented using a predefined label, such as small size, medium size, and large size.

[0062] If size is represented deterministically, e.g., size 43 indicates a particular label. If size of an object is represented probabilistically, e.g., size 43 indicates a probability distribution over labels representing size.

[0063] Color 44 indicates the color of an object. The color of an object can be expressed, for example, by coordinates in a predetermined color space. The color space can be, for example, an RGB space or a CMYK space. More specifically, when the RGB space is used as the color space, the color of the object is expressed as a combination of the values ​​of the R component, the G component, and the B component.

[0064] When the color of an object is represented deterministically, for example, color 44 indicates a specific coordinate in a color space. When the color of an object is represented probabilistically, for example, color 44 indicates a probability distribution for the coordinate in the color space. Alternatively, for example, color 44 may indicate a probability distribution for each value along each coordinate axis of the color space. For example, when an RGB space is used as the color space, color 44 may indicate a probability distribution for the R component of the object's color, a probability distribution for the G component of the object's color, and a probability distribution for the B component of the object's color.

[0065] Colors may be represented by predefined labels, such as red, blue, etc. If colors are represented deterministically, e.g., color 44 indicates a particular label representing the color. If colors are represented probabilistically, e.g., color 44 indicates a probability distribution over the labels representing the color.

[0066] The brightness 45 indicates the brightness of an object. The brightness of an object is expressed, for example, using brightness. When the brightness is expressed deterministically, for example, the brightness 45 indicates a specific brightness. When the brightness is expressed probabilistically, for example, the brightness 45 indicates a probability distribution of brightness.

[0067] Here, by looking at the probability distribution of the state and attributes of an object together with the probability distribution of the object's position, the state and attributes of the object can be expressed as a spatial distribution. FIG. 10 is a diagram illustrating an example of the spatial distribution of object attributes. In FIG. 10, graph 90 shows the probability distribution of the type of object A001. Specifically, graph 90 shows the probability that the type of object A001 is a human, the probability that the type of object A001 is a cat, and the probability that the type of object A001 is a dog. Graph 100 shows the probability distribution of the position of object A001.

[0068] By multiplying graph 90 and graph 100 (in other words, by obtaining the joint probability of type and location), graph 110 can be obtained. Graph 110 represents the spatial distribution of the existence probability of object A001 for each of multiple types. Specifically, graph 110 shows the spatial distribution of the existence probability of object A001 whose type is human, the spatial distribution of the existence probability of object A001 whose type is cat, and the distribution of the existence probability of object A001 whose type is dog.

[0069] <Method of generating virtual space information 10> Here, an example of a method of generating virtual space information 10 will be described. In one example, the virtual space information 10 is generated to imitate real space. Hereinafter, the function unit that generates the virtual space information 10 will be referred to as a generation unit. The generation unit may be included in the search system 2000, or may be provided outside the search system 2000. In the latter case, the device that includes the generation unit has the hardware configuration shown in FIG. 4, similar to the computer that realizes the search system 2000.

[0070] Fig. 11 is a diagram illustrating a situation in which virtual space information 10 is generated. In the example of Fig. 11, a target virtual space 130 represented by the virtual space information 10 is a space that imitates a real space 120. A virtual space that imitates a real space in this way is also called a digital twin.

[0071] The real space 120 is any space in the real world. For example, the real space 120 is a space where some work is carried out, such as a factory or a warehouse. In this case, objects existing in the real space 120 are people, transport vehicles, work robots, etc.

[0072] Various sensors such as a camera, a ranging sensor, or a microphone are provided in the real space 120. The ranging sensor is, for example, a LiDAR (Light Detection and Ranging) sensor. These sensors perform sensing periodically or irregularly and generate sensing data representing the sensing results. The sensing data generated by the camera is, for example, image data. The sensing data generated by the ranging sensor is, for example, point cloud data. The sensing data generated by the microphone is, for example, audio data.

[0073] The generation unit 200 acquires sensing data. Here, each sensor may provide the sensing data directly to the generation unit 200, or may store the sensing data in a storage unit. In the latter case, the generation unit 200 acquires the sensing data by accessing the storage unit in which the sensing data is stored.

[0074] The generation unit 200 uses the sensor data to detect objects and analyze the states and attributes of the detected objects, thereby generating virtual space information 10.

[0075] For example, the generation unit 200 uses current and past sensor data to generate virtual space information 10 that represents the position, state, and attributes of each object at each of multiple current and past points in time. Furthermore, the generation unit 200 may use current and past sensor data to predict the position, state, and attributes of each object at each of multiple future points in time, and include the results of the prediction in the virtual space information 10. In this way, the predicted future real space 120 can be represented in the target virtual space 130.

[0076] When the virtual space information 10 indicates the position of each object at each of multiple future time points, the search conditions may be configured to specify future time points or time periods. This allows the user to use the search system 2000 to obtain predicted object information (e.g., predicted speed) for future time points.

[0077] <Acquisition of Search Conditions: S102> The acquisition unit 2020 acquires the search conditions (S102). More specifically, the acquisition unit 2020 acquires a search query indicating the search conditions. Here, in addition to the search conditions, the search query may further indicate the type of information about the object to be extracted from the virtual space information 10.

[0078] There are various methods for the acquisition unit 2020 to acquire a search query. For example, the acquisition unit 2020 provides a search screen on which a user of the search system 2000 can input search conditions. In this case, the acquisition unit 2020 acquires the result of input on the search screen as a search query indicating the search conditions.

[0079] The search screen may be displayed on a display device provided in the search system 2000, or on a display device provided in another device communicatively connected to the search system 2000. In the latter case, for example, the search system 2000 provides a website where a search can be performed on the virtual space information 10. A user of the search system 2000 accesses the website provided by the search system 2000 using a PC, a smartphone, or the like. The above-mentioned search screen is provided as one of the web pages included in the website. When the user inputs information on the web page representing the search screen, a request representing a search query is sent to the search system 2000. The acquisition unit 2020 acquires the search query from this request.

[0080] The user may select a desired search rule from a plurality of predefined search rules. In this case, a plurality of search rules each representing a different search condition are stored in advance in a storage unit accessible from the search system 2000.

[0081] For example, the acquisition unit 2020 provides the user with a search screen on which one or more search rules can be selected. The user selects a search rule on the search screen. As a result, a request indicating the selected search rule is transmitted to the search system 2000. The acquisition unit 2020 acquires the search query by acquiring the search rule indicated in the request from the storage unit. By allowing the user to select a predetermined search rule in this way, the user can easily search for desired information.

[0082] Alternatively, for example, the acquisition unit 2020 may acquire a search query stored in advance in a storage unit accessible from the search system 2000. For example, the search system 2000 is configured to periodically perform a search using predetermined search conditions by periodically acquiring the search query from the storage unit. This method makes it possible to periodically obtain information about objects that satisfy the predetermined conditions. Therefore, for example, it becomes possible to periodically monitor the real space represented by the virtual space information 10.

[0083] <Executing a Search: S104> The search execution unit 2040 extracts information about objects that satisfy the search conditions from the virtual space information 10 (S104). As described above, the search execution unit 2040 identifies objects that satisfy the search conditions from among the objects shown in the virtual space information 10, and obtains information about the identified objects.

[0084] For example, search conditions indicate conditions related to location, status, or attributes. Location conditions are expressed as a specific location or a specific area. In the former case, the location condition is a condition such as "the location is (x1, y1, z1)." In the latter case, the location condition is a condition such as "the location is included in area R."

[0085] Here, there are various methods for specifying a specific area of ​​the target virtual space in the search conditions. For example, a specific area is specified by the type of area and the value of a parameter that can identify an area of ​​that type. If the specific area is a rectangular area, the parameters for identifying the area may be a pair of the positions of the upper left corner and the lower right corner, or a combination of the upper left corner, width, and height. Furthermore, the area may be specified using a map of the target virtual space displayed on the search screen.

[0086] Alternatively, a region may be specified by a range of positions along one or more coordinate axes. For example, a search criterion such as "position z>1.0m" specifies a region representing all ranges where z>1.0m.

[0087] A search condition related to a state can be specified using a parameter value representing the state or a range of values ​​for that parameter. For example, a search condition related to the posture of an object can be specified using a label such as "standing upright." In addition, a search condition related to the posture of an object can be specified based on the positional relationship of multiple joint points. For example, it is possible to specify conditions such as "the neck is lower than the shoulder" or "the angle between the line connecting the wrist and elbow and the line connecting the shoulder and elbow is 90° or less." Note that "the angle between the line connecting the wrist and elbow and the line connecting the shoulder and elbow is 90° or less" can also be expressed as "the elbow is bent by 90° or more."

[0088] Search conditions related to the speed of an object can be specified as "the speed is 1.0 m / sec" or "the speed is 1.0 m / sec or more", and the direction of the object can be specified as "east" or "between east and northeast".

[0089] As mentioned above, the method of specifying the orientation is not limited to the method using the azimuth direction. For example, the search condition for the orientation of an object can be specified as "a direction of 30° to 60° in the horizontal plane" or "an angle of 15° to 45° with respect to the horizontal plane."

[0090] A search condition related to an attribute can be specified using a parameter value representing the attribute or a range of values ​​for that parameter. A search condition related to an object type can be specified, for example, as "type is human."

[0091] As described above, object types can be defined hierarchically. In this case, by specifying a type that is not a leaf in the hierarchical tree in the search criteria, multiple attributes that are subordinate to that type can be specified. For example, suppose three types are defined below the type "vehicle": automobile, motorcycle, and bicycle. In this case, the search criteria "type is vehicle" can be used to express the search criteria "type is automobile, or type is motorcycle, or type is bicycle." By utilizing the hierarchical structure of types in this way, it becomes easy to specify multiple types.

[0092] The condition for the shape of an object can be expressed as a condition related to a polygon that approximates the object. For example, the condition for the shape of an object can be specified based on the type of polygon that approximates the object. More specifically, it can be specified as "the shape is hexagonal." The shape of an object can also be specified based on the positional relationship of multiple vertices in the polygon that approximates the object. For example, it can be specified as "the first vertex is below the second vertex." Here, it is assumed that each vertex is assigned a number according to a predetermined rule.

[0093] The size condition of an object can be specified using the parameter value that represents the size or the range of the parameter value. For example, if the size is expressed by a combination of width, depth, and height, you can specify a search condition such as "width is 2.0 m or less, depth is 4.5 m or less, and height is 1.5 m or less."

[0094] When the size is represented by a combination of a plurality of parameters, in the search condition, only some of the parameters may be used. For example, when the size is represented by a combination of width, depth, and height, a search condition such as "the width is 2.0 m or less" can be specified. In this case, the depth and height are arbitrary.

[0095] When the size is represented using a label such as S size, the label can be used in the search condition. For example, a search condition such as "the size is S size" can be specified. By using the label in this way, the search condition regarding the size can be easily specified.

[0096] The condition of the color of an object can be specified using the value of the parameter representing the color or the range of the value of the parameter. For example, the parameter representing the color is the coordinate in the color space. As a more specific example, when the RGB space is used as the color space, the parameters representing the color are the R value, the G value, and the B value.

[0097] The range of the value of the parameter representing the color can be represented by a region in the color space. For example, a search condition such as "the range of the color is r1 < R < r2, g1 < G < g2, and b1 < B < b2" can be used to specify the range of the color. When the color is represented by a combination of a plurality of parameters, only the conditions regarding some of the parameters may be specified.

[0098] Also, the parameter representing the color may be a label such as red or blue. In this case, the search condition can be specified using the color label. For example, a search condition such as "it is red" can be specified.

[0099] The condition of the brightness of an object can be specified using the value of the parameter representing the brightness or the range of the value of the parameter. For example, the parameter representing the brightness is the lightness. Therefore, for example, a search condition such as "the lightness is 3 or more and 5 or less" can be specified.

[0100] A time point or a period may be specified in the search condition. An example of a search condition that specifies a time point or a period is "the speed is 1.0 m / sec or more from time t1 to time t2."

[0101] As described above, the virtual space information 10 may indicate the position, state, or attributes of each object in the future. When the virtual space information 10 indicates the position of each object in the future, a future time or period may be specified as a search condition. This allows the user to use the search system 2000 to obtain predicted object information (e.g., predicted speed) for a future time.

[0102] A search condition can be expressed as a combination of multiple conditions. Hereinafter, each of the multiple conditions that make up a search condition will be referred to as a partial condition. Multiple partial conditions can be combined using any logical symbol (such as and, or, or xor).

[0103] The search performed by the search execution unit 2040 uses a probability threshold. For example, a probability threshold is specified for each partial condition in the search criteria. An example of a search criteria in which a probability threshold is specified for each partial condition is "(the probability that the speed is 1.0 m / sec or greater is 50% or greater) and (the probability that the width is 1.7 m or less is 70% or greater)."

[0104] A probability threshold may be specified for a combination of partial conditions in a search condition. When a probability threshold is specified for a combination of partial conditions, for example, a probability threshold such as "probability of 50% or more" is specified for the partial conditions "(speed is 1.0 m / sec or more) and (width is 1.7 m or less)." The probability threshold specified for a combination of partial conditions represents a threshold for the probability that multiple partial conditions are satisfied simultaneously (i.e., joint probability).

[0105] The probability threshold may be specified by the user or may be predetermined in the search system 2000. In the latter case, the search conditions indicated in the search query do not include a probability threshold, such as "(speed is 1.0 m / sec or more) and (width is 1.7 m or less)." The acquisition unit 2020 generates search conditions that include a probability threshold based on the search conditions indicated in the search query and a predetermined probability threshold. In this way, the acquisition unit 2020 acquires search conditions that include a probability threshold.

[0106] For example, suppose the search conditions indicated in the search query are "(speed is 1.0 m / sec or more) and (width is 1.7 m or less)" and the predetermined probability threshold is 50%. In this case, the acquisition unit 2020 acquires the search condition "the probability that ((speed is 1.0 m / sec or more) and (width is 1.7 m or less)) is satisfied is 50% or more" by including the probability threshold in the search conditions indicated in the search query.

[0107] The search execution unit 2040 may be configured to automatically adjust the probability threshold based on the number of objects that satisfy the search criteria. In this case, for example, the search query specifies an upper limit on the number of objects to be extracted. In addition, an initial value and a step size are set for the probability threshold.

[0108] First, the search execution unit 2040 performs a search using search criteria to which an initial value of the probability threshold has been added. If the number of objects that satisfy the search criteria is equal to or less than the upper limit, information about those objects is extracted. On the other hand, if the number of objects that satisfy the search criteria exceeds the upper limit, the search execution unit 2040 increases the probability threshold and performs the search again. Specifically, if the initial value and step size are denoted as Ta and w, respectively, the probability threshold used in the i-th search is expressed as Ta+w*(i-1).

[0109] The search execution unit 2040 repeatedly executes searches until the number of objects satisfying the search criteria falls below an upper limit or the probability threshold exceeds an upper limit (e.g., 100%). When the number of objects satisfying the search criteria falls below the upper limit, the search execution unit 2040 extracts information about each object that satisfies the search criteria. On the other hand, when the probability threshold reaches an upper limit, for example, the search execution unit 2040 outputs a notification indicating that the number of objects that satisfy the search criteria exceeds the upper limit. Alternatively, when the probability threshold reaches an upper limit, for example, the search execution unit 2040 may extract information about each object that satisfied the search criteria in the last search.

[0110] Note that the values ​​of parameters such as the upper limit number of objects to be extracted, the initial value of the probability threshold, and the step width when changing the probability threshold may be determined in advance in the search system 2000.

[0111] <Result Output> The search execution unit 2040 extracts information about objects that satisfy the search conditions from the virtual space information 10. Then, the search execution unit 2040 generates search result information that indicates the extracted information.

[0112] The type of information extracted from the virtual space information 10 may be predetermined or may be indicated by a search query. In the latter case, for example, the search query indicates search conditions such as "(speed is 1.0 m / sec or more) and (width is 1.7 m or less)" and information to be extracted such as "identification information and type of object." In this case, the search result information indicates a pair of identification information and type of each object that satisfies the search conditions.

[0113] Here, it is assumed that the information specified as the extraction target is represented probabilistically in the virtual space information 10. In this case, for example, the search result information shows object information probabilistically. For example, it is assumed that the types of objects that satisfy the search criteria are shown as "humans: 86%, dogs: 7%, cats: 4%, cars: 1%, motorcycles: 1%, bicycles: 1%" in the object attribute information 40. In this case, the search result information shows, for example, information such as "humans: 86%, dogs: 7%, cats: 4%, cars: 1%, motorcycles: 1%, bicycles: 1%."

[0114] Alternatively, for example, the search result information may indicate a pair of the label with the highest probability and the probability corresponding to that label for information expressed probabilistically. For example, in the above example, the search result information indicates "human: 86%." Alternatively, for example, the search result information may indicate information about each label whose probability is equal to or greater than a predetermined threshold. For example, if the threshold is 5% in the above example, the search result information indicates "human: 86%, dog: 7%."

[0115] There are various methods for outputting search result information. For example, the search execution unit 2040 stores the search result information in an arbitrary storage unit. Alternatively, for example, the search execution unit 2040 displays the search result information on an arbitrary display device. Alternatively, for example, the search execution unit 2040 transmits the search result information to another device. For example, assume that a request including a search query is transmitted from a user's terminal to the search system 2000. In this case, the search system 2000 transmits a response including the search result information to the user's terminal.

[0116] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0117] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0118] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) A search system comprising: an acquisition unit that acquires search conditions in which at least one condition related to an object is expressed probabilistically; and a search execution unit that extracts information related to objects that satisfy the search conditions from virtual space information that includes information on one or more objects in a target virtual space, wherein the virtual space information probabilistically represents the position of each of the objects in the target virtual space. (Supplementary Note 2) The search system according to Supplementary Note 1, wherein the search conditions include a probability that a first partial condition related to the object and a second partial condition related to the object are simultaneously satisfied is equal to or greater than a first threshold, and the search execution unit calculates, for each of the objects shown in the virtual space information, a probability that the first partial condition and the second partial condition are simultaneously satisfied, and determines whether the calculated probability is equal to or greater than the first threshold. (Supplementary Note 3) The search system according to Supplementary Note 2, wherein the acquisition means further acquires a search query indicating, as a condition, that both the first partial condition and the second partial condition are satisfied, and information indicating the first threshold, and generates the search conditions based on the conditions indicated in the search query and the first threshold. (Supplementary Note 4) The search system according to Supplementary Note 3, wherein the acquisition means further acquires information indicating an upper limit of the number of the objects to be detected by the search execution means, and the search execution means detects the objects that satisfy the search conditions, and if the number of the detected objects is greater than the upper limit, detects the objects that satisfy the search conditions using the search conditions in which the first threshold is replaced with a second threshold that is greater than the first threshold. (Supplementary Note 5) The search system according to any one of Supplements 1 to 4, wherein the virtual space information indicates information about each of the objects for each of a plurality of time points, the search conditions include a time condition, and the search execution means detects the objects that satisfy the search conditions from information about each of the objects indicated in the virtual space information for which the time condition is satisfied.(Supplementary Note 6) The search system according to any one of Supplementary Notes 1 to 4, wherein the virtual space information includes state information relating to a state of each of the objects, attribute information relating to an attribute of each of the objects, or both the state information and the attribute information. (Supplementary Note 7) The search system according to any one of Supplementary Notes 1 to 4, comprising a generation unit that acquires sensing data generated by one or more sensors that sense objects in real space and generates the virtual space information using the acquired sensing data. (Supplementary Note 8) A search method executed by a computer, comprising: an acquisition step of acquiring search conditions in which at least one condition relating to an object is probabilistically expressed; and a search execution step of extracting information relating to objects that satisfy the search conditions from virtual space information including information of one or more objects in a target virtual space, wherein the virtual space information probabilistically expresses the position of each of the objects in the target virtual space. (Supplementary Note 9) The search method according to Supplementary Note 8, wherein the search conditions include a probability that a first partial condition related to the object and a second partial condition related to the object are simultaneously satisfied that is equal to or greater than a first threshold, and wherein the search execution step calculates, for each of the objects shown in the virtual space information, a probability that the first partial condition and the second partial condition are simultaneously satisfied, and determines whether the calculated probability is equal to or greater than the first threshold. (Supplementary Note 10) The search method according to Supplementary Note 9, wherein the acquisition step further acquires a search query indicating, as a condition, that both the first partial condition and the second partial condition are satisfied, and information indicating the first threshold, and generates the search conditions based on the condition indicated in the search query and the first threshold. (Supplementary Note 11) The search method described in Supplementary Note 10, wherein in the acquisition step, information indicating an upper limit number of the objects to be detected by the search execution step is further acquired; in the search execution step, the objects that satisfy the search conditions are detected; and if the number of the detected objects is greater than the upper limit number, the objects that satisfy the search conditions are detected using the search conditions in which the first threshold value is replaced with a second threshold value that is greater than the first threshold value.(Supplementary Note 12) The search method according to any one of Supplements 8 to 11, wherein the virtual space information indicates information about each of the objects for each of a plurality of time points, the search conditions include a time condition, and in the search execution step, an object that satisfies the search condition is detected from information about each of the objects indicated in the virtual space information for which the time condition is satisfied. (Supplementary Note 13) The search method according to any one of Supplements 8 to 11, wherein the virtual space information includes state information about a state of each of the objects, attribute information about attributes of each of the objects, or both the state information and the attribute information. (Supplementary Note 14) The search method according to any one of Supplements 8 to 11, comprising a generation step of acquiring sensing data generated by one or more sensors that sense an object in real space, and generating the virtual space information using the acquired sensing data. (Supplementary Note 15) A search device comprising: an acquisition means for acquiring search conditions in which at least one condition related to an object is expressed probabilistically; and a search execution means for extracting information related to objects that satisfy the search conditions from virtual space information including information on one or more objects in a target virtual space, wherein the virtual space information probabilistically represents the position of each of the objects in the target virtual space. (Supplementary Note 16) The search device according to Supplementary Note 15, wherein the search conditions include a probability that a first partial condition related to the object and a second partial condition related to the object are simultaneously satisfied is equal to or greater than a first threshold, and the search execution means calculates, for each of the objects shown in the virtual space information, a probability that the first partial condition and the second partial condition are simultaneously satisfied, and determines whether the calculated probability is equal to or greater than the first threshold. (Supplementary Note 17) The search device described in Supplementary Note 16, wherein the acquisition means further acquires a search query indicating as a condition that both the first partial condition and the second partial condition are satisfied, and information indicating the first threshold value, and generates the search conditions based on the condition indicated in the search query and the first threshold value.(Supplementary Note 18) The search device according to Supplementary Note 17, wherein the acquisition means further acquires information indicating an upper limit of the number of the objects to be detected by the search execution means, and the search execution means detects the objects that satisfy the search conditions, and if the number of the detected objects is greater than the upper limit, detects the objects that satisfy the search conditions using the search conditions in which the first threshold is replaced with a second threshold that is greater than the first threshold. (Supplementary Note 19) The search device according to any one of Supplements 15 to 18, wherein the virtual space information indicates information about each of the objects for each of a plurality of time points, and the search conditions include a time-related condition, and the search execution means detects the objects that satisfy the search conditions from information about each of the objects indicated in the virtual space information that satisfies the time-related condition. (Supplementary Note 20) The search device according to any one of Supplements 15 to 18, wherein the virtual space information includes state information about a state of each of the objects, attribute information about an attribute of each of the objects, or both the state information and the attribute information. (Supplementary Note 21) A search device according to any one of Supplementary Notes 15 to 18, comprising a generation means for acquiring sensing data generated by one or more sensors that sense objects in real space, and generating the virtual space information using the acquired sensing data.

[0119] REFERENCE SIGNS LIST 10 Virtual space information 20 Position information 30 Status information 31 Attitude 32 Speed ​​33 Orientation 40 Attribute information 41 Type 42 Shape 43 Size 44 Color 45 Brightness 50 Existence probability information 51 Object identifier 52 Existence probability 60 Graph 70 Graph 80 Graph 90 Graph 100 Graph 110 Graph 120 Real space 130 Target virtual space 200 Generation unit 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input / output interface 1120 Network interface 2000 Search system 2020 Acquisition unit 2040 Search execution unit 3000 Search device

Claims

1. A means for obtaining a search condition in which at least one condition relating to an object is probabilistically expressed, The system includes a search execution means for extracting information about objects that satisfy the search conditions from virtual space information that includes information about one or more objects in the target virtual space, The virtual space information is a search system that probabilistically represents the position of each object in the target virtual space.

2. The search condition includes the fact that the probability of the first partial condition and the second partial condition relating to the object being satisfied simultaneously is greater than or equal to a first threshold. The search execution means calculates the probability that the first partial condition and the second partial condition are simultaneously satisfied for each of the objects shown in the virtual space information, and determines whether the calculated probability is equal to or greater than the first threshold, according to claim 1.

3. The acquisition means is, Further obtaining a search query that specifies that both the first partial condition and the second partial condition must be satisfied, and information indicating the first threshold, The search system according to claim 2, which generates the search conditions based on the conditions shown in the search query and the first threshold.

4. The acquisition means further acquires information indicating the upper limit number of objects detected by the search execution means, The search execution means is The object that satisfies the aforementioned search conditions is detected, The search system according to claim 3, wherein if the number of detected objects is greater than the upper limit, the system uses the search conditions in which the first threshold is replaced with a second threshold greater than the first threshold to detect objects that satisfy the search conditions.

5. The virtual space information provides information about each of the objects for each of the multiple time points in time. The aforementioned search criteria include time-related conditions, The search execution means detects an object that satisfies the search condition from the information of each of the objects shown in the virtual space information, where the time condition is satisfied.

6. The search system according to any one of claims 1 to 4, wherein the virtual space information includes state information relating to the state of each object, attribute information relating to the attributes of each object, or both the state information and the attribute information.

7. A search system according to any one of claims 1 to 4, comprising a generation means for acquiring sensing data generated by one or more sensors that sense an object in real space, and for generating virtual space information using the acquired sensing data.

8. A retrieval step to obtain a search condition in which at least one condition concerning an object is probabilistically expressed, The system includes a search execution step of extracting information about an object that satisfies the search conditions from virtual space information that includes information about one or more objects in the target virtual space, The virtual space information is a computer-based search method that probabilistically represents the position of each object in the target virtual space.

9. The search condition includes the fact that the probability of the first partial condition and the second partial condition relating to the object being satisfied simultaneously is greater than or equal to a first threshold. The search method according to claim 8, wherein in the search execution step, the probability that the first partial condition and the second partial condition are simultaneously satisfied for each of the objects shown in the virtual space information is calculated, and it is determined whether the calculated probability is equal to or greater than the first threshold.

10. A means for obtaining a search condition in which at least one condition relating to an object is probabilistically expressed, The system includes a search execution means for extracting information about objects that satisfy the search conditions from virtual space information that includes information about one or more objects in the target virtual space, The virtual space information is a search device that probabilistically represents the position of each object in the target virtual space.