Information processing device, information processing method, and program

Abstract

To provide a technology for supporting appropriate collection of images associated with measurement positions not only outdoors but also indoors. [Solution] An information processing device according to one aspect of the present disclosure acquires a movement trajectory of an imaging device during the operation period of the imaging device, which is measured from measurement data of an inertial sensor arranged to observe the movement of an imaging device separately attached to a moving body, including the image capture period of a group of images captured by the imaging device, corrects the acquired movement trajectory so as to satisfy the constraints of movement conditions given in advance to the moving body, and outputs the corrected movement trajectory.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] There is a technology to collect images by attaching an imaging device to a moving body such as a vehicle or a person. For example, Non-Patent Document 1 describes a technology to collect images by attaching an imaging device to a moving body, capturing images with the imaging device while measuring the position with a GPS (Global Positioning System) sensor. Methods for collecting images in relation to location have been proposed. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] “How Street View Works and Where to Collect Images in the Future”, [online], [Retrieved June 16, 2025], Internet<URL:https: / / www.google.com / intl / ja / streetview / how-it-works / > Summary of the Invention [Problem to be solved by the invention]

[0004] According to the conventional method, it is possible to collect images associated with measurement positions. However, the inventors of the present invention have found the following problem with the conventional method. That is, a satellite positioning module such as a GPS sensor can be used outdoors, but is difficult to use indoors. Therefore, with the conventional method, it is difficult to collect images associated with measurement positions indoors.

[0005] In one aspect, the present disclosure has been made in view of the above circumstances, and one of the objects of the present disclosure is to provide a technology for supporting appropriate collection of images associated with measurement positions not only outdoors but also indoors. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the present disclosure employs the following configurations. Note that the following configurations can be combined as appropriate.

[0007] An information processing device according to one aspect of the present disclosure includes a control unit configured to acquire a movement trajectory of an imaging device attached separately to a moving object, the movement trajectory being measured from measurement data of an inertial sensor arranged to observe the movement of the imaging device during an operation period of the imaging device including an image capture period for a group of images captured by the imaging device, correct the acquired movement trajectory so as to satisfy constraints of movement conditions given in advance to the moving object, and output the corrected movement trajectory.

[0008] In this configuration, an inertial sensor is used to measure position. Using an inertial sensor, position can be measured not only outdoors but also indoors. However, position measurement using an inertial sensor is prone to errors. Therefore, this configuration aims to suppress errors in the measured position by correcting the movement trajectory (a set of measurement positions) so as to satisfy the constraints of the movement conditions. This is expected to provide measurement positions with reduced errors. Therefore, this configuration can support the appropriate collection of images associated with measurement positions not only outdoors but also indoors.

[0009] In the information processing device according to the above aspect, the constraint of the movement condition may include that the moving object is located in the same place at a plurality of times within the operation period. The method may include correcting the movement trajectory so that the measured positions at the respective times on the movement trajectory coincide with each other. With this configuration, it is possible to expect a reduction in calculation errors.

[0010] In the information processing device according to the above aspect, the moving object may be a robotic device that moves autonomously within a store. The same location may be a charging point for the robotic device. With this configuration, by using the travel condition that the robotic device passes through the charging point, it is possible to expect appropriate identification of the time when the robotic device is located at the same location. As a result, it is possible to expect appropriate correction of the travel trajectory.

[0011] In the information processing device according to the above aspect, the constraints on the movement conditions may include restricting the movement of the moving object within a predetermined movement range. Correcting the movement trajectory may include correcting the movement trajectory to fit the shape of the movement range. This configuration is expected to reduce errors caused by deviation from a coordinate system of real space.

[0012] In the information processing device according to the above aspect, the moving object may be a robotic device that moves autonomously within a store. The movement range may be defined within a floor of the store. With this configuration, by utilizing the movement condition that the robotic device moves within the store, it is possible to expect appropriate identification of the movement range limit. As a result, it is possible to expect appropriate correction of the movement trajectory.

[0013] In the information processing device according to the above aspect, the constraints on the movement conditions may further include internal constraints within the movement range. Correcting the movement trajectory may further include correcting the movement trajectory so as to satisfy the internal constraints. This configuration can further reduce calculation errors. As a result, it is expected that a movement trajectory with reduced errors can be obtained.

[0014] In the information processing device according to the above aspect, the internal constraint may include that the moving object does not pass through an obstacle range defined by the presence of an obstacle. Correcting the movement trajectory to satisfy the internal constraint may include correcting the movement trajectory to detour around the obstacle range. With this configuration, by using information on the range where the obstacle exists (obstacle range) as the internal constraint, it is possible to expect appropriate correction of the movement trajectory.

[0015] Note that the embodiments of the present disclosure may not be limited to the above-described information processing device. As another aspect of the information processing device according to each of the above aspects, one aspect of the present disclosure may be an information processing method that realizes all or part of each of the above configurations, a program, or a machine-readable storage medium storing such a program. Here, the machine-readable storage medium may be a non-transitory medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action. The non-transitory storage medium may include a storage medium (e.g., a CD, a DVD, a semiconductor memory), an auxiliary storage device of a computer, an external storage device connected to a computer, etc.

[0016] For example, an information processing method according to an aspect of the present disclosure may be executed by a computer, and may include acquiring a movement trajectory of an imaging device separately attached to a moving body, the movement trajectory being measured from measurement data of an inertial sensor arranged to observe the movement of the imaging device during an operation period of the imaging device including an image capture period of a group of images captured by the imaging device, correcting the acquired movement trajectory so as to satisfy constraints of movement conditions given in advance to the moving body, and outputting the corrected movement trajectory.

[0017] Furthermore, for example, a program according to an aspect of the present disclosure may be a program for causing a computer to execute an information processing method, the information processing method including: detecting a movement measured from measurement data of an inertial sensor arranged to observe the movement of an imaging device separately attached to a moving object; The movement trajectory may include acquiring the movement trajectory of the imaging device during the operation period of the imaging device, which includes the imaging period of the group of images captured by the imaging device, correcting the acquired movement trajectory so as to satisfy the constraints of the movement conditions given in advance to the moving body, and outputting the corrected movement trajectory. [Effects of the Invention]

[0018] According to one aspect of the present disclosure, it is possible to provide a technology for supporting appropriate collection of images associated with measurement positions not only outdoors but also indoors. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 schematically illustrates an example of a situation to which the present disclosure is applied. [Figure 2] FIG. 2 shows a schematic diagram of an example of a moving object. [Figure 3] FIG. 3 shows a schematic diagram of an example of a method for correcting a movement trajectory. [Figure 4] FIG. 4 shows a schematic diagram of an example of a method for correcting a movement trajectory. [Figure 5] FIG. 5 shows a schematic diagram of an example of a method for correcting a movement trajectory. [Figure 6] FIG. 6 is a diagram illustrating an example of another arrangement of information processing devices. [Figure 7] FIG. 7 is a schematic diagram showing an example of a scene in which a captured image is used. [Figure 8] FIG. 8 is a schematic diagram showing another example of a scene in which a captured image is used. [Figure 9] FIG. 9 shows an example of the provided data. [Figure 10] FIG. 10 is a diagram illustrating an example of a hardware configuration of an information processing device. [Figure 11] FIG. 11 is a diagram illustrating an example of a hardware configuration of a search device. [Figure 12] FIG. 12 is a diagram illustrating an example of the software configuration of the information processing device. [Figure 13] FIG. 13 is a diagram illustrating an example of the software configuration of the search device. [Figure 14] FIG. 14 is a flowchart illustrating an example of a processing procedure for correcting a movement trajectory by the information processing device. [Figure 15] FIG. 15 is a flowchart illustrating an example of a processing procedure for assigning the result of image analysis. [Figure 16] FIG. 16 is a flowchart illustrating an example of a processing procedure of the search device. [Figure 17] FIG. 17 shows the initial values ​​of the movement trajectory in the experimental example. [Figure 18] FIG. 18 shows the movement trajectory after correction by the first method in the experimental example. [Figure 19] FIG. 19 shows the movement trajectory after correction by the 2-1 method in the experimental example. [Figure 20] FIG. 20 shows the movement trajectory after correction by the 2-2 method in the experimental example. DETAILED DESCRIPTION OF THE INVENTION

[0020] An embodiment according to one aspect of the present disclosure will be described below with reference to the drawings. However, the embodiment described below is merely an example of the present disclosure in all respects. Various improvements or modifications may be made without departing from the scope of the present disclosure. In implementing the present disclosure, specific configurations according to the embodiment may be appropriately adopted. Note that while data appearing in the present embodiment is described in natural language, more specifically, it may be specified using pseudo-language, commands, parameters, machine language, electrical signals, etc. that can be recognized by a machine such as a computer.

[0021] §1 Application Examples FIG. 1 schematically shows an example of a situation to which the present disclosure is applied. In this embodiment, an imaging device 2 is separately attached to a moving body MB. An inertial sensor 3 is arranged to observe the movement of the imaging device 2. An information processing device 1 acquires a movement trajectory 300 of the imaging device 2 measured from measurement data 39 of the inertial sensor 3 during the operation period of the imaging device 2, which includes the image capture period of a group of images 200 captured by the imaging device 2. The group of images 200 may be composed of one or more images 20. The movement trajectory 300 may be composed of one or more measurement positions 30 (position measurement results). The information processing device 1 corrects the acquired movement trajectory 300 so as to satisfy a constraint 70 of a movement condition given in advance to the moving body MB. As a result, the information processing device 1 can obtain a corrected movement trajectory 325. The corrected movement trajectory 325 may be configured from one or more corrected measurement positions 32 (position-corrected measurement results). The information processing device 1 outputs the corrected movement trajectory 325.

[0022] In this embodiment, an inertial sensor 3 is used to measure the position of the imaging device 2 (moving body MB). The inertial sensor 3 can measure the position not only outdoors but also indoors. However, position measurement using the inertial sensor 3 is prone to errors. Therefore, in this embodiment, the movement trajectory 300 is corrected to satisfy the movement condition constraint 70, thereby attempting to suppress errors in the measurement position (measurement position 30). This makes it possible to expect the provision of a measurement position (measurement position 32) with reduced errors. Therefore, this embodiment can support the appropriate collection of an image (image 20) associated with the measurement position (measurement position 32) not only outdoors but also indoors.

[0023] [Inertial Sensor] The type of the inertial sensor 3 may be appropriately selected depending on the embodiment. For example, the inertial sensor 3 may be configured to measure inertial motion (translational motion and rotational motion) with six degrees of freedom by including an acceleration sensor and a gyro sensor (angular velocity sensor). The inertial sensor 3 may be configured to measure inertial motion with nine degrees of freedom by further including a geomagnetic sensor. The inertial sensor 3 may be an inertial measurement unit (IMU) or a motion sensor. The degrees of freedom of measurement by the inertial sensor 3 are not limited to this example, and may be set to other than six or nine degrees of freedom.

[0024] Each measured position 30 constituting the movement trajectory 300 may be obtained by analyzing measurement data 39 of the inertial sensor 3. As long as the measured position 30 can be derived, the method of analyzing the measurement data 39 is not particularly limited and may be appropriately selected depending on the embodiment. In one example, the analysis method may be a self-positioning method such as inertial navigation or pedestrian dead reckoning (PDR). Alternatively, an estimation method may be adopted. In another example, a trained machine learning model may be used to analyze the measurement data 39. The machine learning model is configured to have one or more calculation parameters that can be adjusted by machine learning. The one or more calculation parameters are used to calculate the desired inference (position estimation). The machine learning model may be configured, for example, by a neural network, a regression model, a decision tree model, a support vector machine, or other functional formula. The machine learning method may be appropriately selected depending on the machine learning model to be adopted (e.g., backpropagation). In the machine learning, the value of each calculation parameter of the machine learning model may be appropriately adjusted (optimized) using training samples so as to acquire the ability to derive position information from the measurement values ​​of the inertial sensor 3.

[0025] The position may be measured in two or three dimensions. Measurement conditions such as the position measurement interval, time resolution, and type of coordinate system are not particularly limited and may be determined appropriately depending on the embodiment. In one example, the orientation may be measured together with the position from the measurement data 39 of the inertial sensor 3. That is, the attitude (position and orientation) may be measured from the measurement data 39 of the inertial sensor 3, and the movement trajectory 300 may be expanded to be composed of one or more measured attitudes (attitude measurement results). The description of the position in this embodiment may be applied by replacing it with the attitude. Correcting the measured position 30 (movement trajectory 300) may be equivalent to correcting the measured attitude.

[0026] The movement trajectory 300 may be formed by continuously acquiring the measurement positions 30. The calculation process for deriving the measurement positions 30 from the measurement data 39 (generating the movement trajectory 300) may be executed on the information processing device 1, or may be executed on a computer other than the information processing device 1. In one example, the information processing device 1 may acquire the measurement data 39 directly from the inertial sensor 3, and derive the measurement positions 30 from the acquired measurement data 39 to acquire the movement trajectory 300. The information processing device 1 may acquire the measurement data 39 directly from the inertial sensor 3, and derive the measurement positions 30 from the acquired measurement data 39 via another computer, a network, a storage medium, etc. The movement trajectory 300 may be obtained by acquiring measurement data 39 and deriving the measured position 30 from the acquired measurement data 39. Alternatively, the movement trajectory 300 may be generated by another computer, and the information processing device 1 may acquire the movement trajectory 300 from another computer via another computer, a network, a storage medium, etc. If the inertial sensor 3 includes processor resources, the calculation process for deriving the measured position 30 from the measurement data 39 may be executed on the inertial sensor 3.

[0027] The movement trajectory 300 may be acquired by the information processing device 1 in real time. The movement trajectory 300 may be stored in any storage area such as the memory resources of the information processing device 1 or the memory resources of another computer. The movement trajectory 300 may be stored in any storage area in the form of a database. The movement trajectory 300 may be acquired by the information processing device 1 afterwards after being stored in any storage area.

[0028] (Inertial sensor placement) As long as the position of the imaging device 2 can be observed directly or indirectly, the location of the inertial sensor 3 is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the inertial sensor 3 may be built into the imaging device 2 or a device including the imaging device 2 (such as a terminal device UT). In another example, the inertial sensor 3 may be provided separately from the imaging device 2, or may be appropriately attached to the imaging device 2. In yet another example, the inertial sensor 3 may be attached to the moving body MB together with the imaging device 2. In this way, the inertial sensor 3 may be provided to indirectly measure the position of the imaging device 2 via the moving body MB.

[0029] (Continuous measurement) The inertial sensor 3 may continuously measure the position of the imaging device 2 regardless of whether the imaging device 2 is performing imaging. The movement trajectory 300 may be configured to include measurement positions 30 at one or more times, thereby showing the progress of movement of the moving body MB (imaging device 2) in chronological order. The movement trajectory 300 may be configured by one or more combinations of measurement positions 30 and measurement times. The measurement times may be obtained appropriately from a timer. The timer may be provided in any device. The timer may be included in, for example, the information processing device 1, another computer, etc.

[0030] Imaging by the imaging device 2 may be controlled arbitrarily. Imaging by the imaging device 2 may be performed continuously while the moving body MB is moving. Imaging by the imaging device 2 may be performed at all times. Imaging by the imaging device 2 may be performed in response to the satisfaction of a predetermined condition (i.e., may be performed only when the predetermined condition is satisfied). For example, the imaging device 2 may be controlled to perform imaging every time the moving body MB moves more than or equal to a predetermined amount. The captured image 20 may be stored in association with the imaging time. The imaging time may be obtained as appropriate from a timer. The timer may be included in, for example, the information processing device 1, another computer, the imaging device 2, etc. When imaging by the imaging device 2 is synchronized with the measurement by the inertial sensor 3, the imaging time of the imaging device 2 may be the same as the measurement time of the inertial sensor 3.

[0031] The image group 200 may be formed as appropriate from one or more captured images 20. The capturing of images by the imaging device 2 and the formation of the image group 200 may be executed on the information processing device 1, or may be executed on a computer other than the information processing device 1. In one example, the information processing device 1 may acquire the image group 200 by acquiring the images 20 directly from the imaging device 2. The information processing device 1 may acquire the image group 200 via another computer, a network, a storage medium, or the like. Alternatively, the image group 200 may be collected by another computer, and the information processing device 1 may not acquire the image group 200.

[0032] (operation period) The period during which the position is measured by the inertial sensor 3 may be defined as the operation period of the imaging device 2. The position measurement by the inertial sensor 3 may start from the start time of imaging by the imaging device 2 or any time before the start time of imaging. Furthermore, the position measurement by the inertial sensor 3 may end at the end time of imaging by the imaging device 2 or any time after the end time of imaging. As a result, the operation period of the imaging device 2 may include the imaging period of the image group 200. For example, position measurement by the inertial sensor 3 and imaging by the imaging device 2 may start in response to the establishment of a start trigger. Position measurement by the inertial sensor 3 and imaging by the imaging device 2 may end in response to the establishment of an end trigger. The start trigger and end trigger may be set arbitrarily. The start trigger may be, for example, the start of movement of the moving body MB, pressing of a physical or software switch, an instruction to execute an application, etc. The end trigger may be, for example, the end of movement of the moving body MB, pressing of a physical or software switch, an instruction to end an application, etc.

[0033] As a specific example, the information processing device 1 may detect the start of movement of the moving body MB based on the measurement data 39 of the inertial sensor 3. The criteria for detecting the start of movement may be set arbitrarily. For example, the start of movement of the moving body MB may be detected when the amount of change in position specified by the measurement data 39 exceeds a threshold. The threshold for detecting the start of movement may be set arbitrarily. Moving may include stopping in place and changing direction. Then, in response to detecting the start of movement of the moving body MB, the information processing device 1 may start recording the movement trajectory 300 using the inertial sensor 3 and capturing images using the imaging device 2. Furthermore, the information processing device 1 may detect the end of movement of the moving body MB based on the measurement data 39 of the inertial sensor 3. The criteria for detecting the end of movement may be set arbitrarily. For example, the stop of movement of the moving body MB may be detected when the amount of change in position specified by the measurement data 39 is less than a threshold. The threshold for detecting the stop of movement may be set arbitrarily. The end of the movement of the moving body MB may be detected in response to the stop of the movement of the moving body MB continuing for a predetermined period of time. The threshold value of the period for detecting the end of movement may be set arbitrarily. Then, in response to detecting the end of the movement of the moving body MB, the information processing device 1 may end the recording of the movement trajectory 300 by the inertial sensor 3 and the imaging by the imaging device 2. Note that the computer that controls the recording of the movement trajectory 300 by the inertial sensor 3 and the imaging by the imaging device 2 is not limited to the information processing device 1 and may be changed as appropriate depending on the embodiment. At least one of the recording of the movement trajectory 300 by the inertial sensor 3 and the imaging by the imaging device 2 may be controlled by a computer other than the information processing device 1.

[0034] The period during which the position is measured by the inertial sensor 3 is not limited to this example and may be changed as appropriate depending on the embodiment. The inertial sensor 3 may measure the position of the imaging device 2 during a period other than the operation period of the imaging device 2. Furthermore, some imaging by the imaging device 2 may be performed independently of the position measurement by the inertial sensor 3.

[0035] [Moving object] 2 is a schematic diagram showing an example of a moving object MB according to this embodiment. The moving object MB may be any object that moves. The type of the moving object MB may be appropriately selected depending on the embodiment. The object may include at least one of a living thing and a device (machine).

[0036] In one example, the mobile body MB may be any device MB1 configured to be movable. The device MB1 may include at least one of a robotic device MB11 that moves autonomously and a device MB12 that moves by direct or remote manual operation. The robotic device MB11 may include, for example, an automatically driven vehicle, an air vehicle that can fly automatically, or other robotic devices configured to be autonomously movable. The air vehicle may include a drone. The robotic device MB11 may include a robotic device used in a store (a robotic device that moves autonomously within a store). The robotic devices used in a store may include, for example, a cleaning robot, a food delivery robot, a transport robot, a guide robot, a security robot, etc.

[0037] Direct manual operation may include, for example, operating the device MB12 using an operation unit provided on the device MB12, physically interacting with the device MB12, etc. Physical interaction may include, for example, pushing or towing the device MB12. Remote manual operation may include, for example, remotely operating the device MB12 using a controller or a computer. The device MB12 may include, for example, a manually operated vehicle, a remotely operated aircraft, a cart that is pushed or driven, or other devices configured to be manually movable. The device MB12 may include a device used in a store (a device that is manually operated and moved within a store). The device used in a store may include, for example, a shopping cart. The shopping cart may include a smart cart, a cash register cart, etc. The store may include, for example, a commercial facility such as a supermarket, a convenience store, a drugstore, a restaurant, etc. The store may also include a commercial complex such as a shopping mall. According to an example of the present embodiment, when the device MB1 is operated as a mobile object MB, appropriate collection of images associated with measurement positions can be supported.

[0038] In addition, in one example, the moving object MB may be a living creature MB2. The type of living creature MB2 is not particularly limited and may be appropriately selected depending on the embodiment. The living creature MB2 may include at least one of a human being MB21 and another living creature MB22 other than the human being MB21. The other living creature MB22 may include, for example, a pet such as a dog or a cat. When the imaging device 2 is operated in a store, the living creature MB2 may be any living creature moving around in the store. For example, if the living creature MB2 is a human being MB21, the living creature MB2 (human being MB21) moving around in the store may be a store clerk, a patrol officer, a customer (client), etc. The patrol officer may include, for example, a security guard or a police officer. The store clerk may include not only a person employed by the store but also any person working in the store (such as a cleaner). According to one example of the present embodiment, when the imaging device 2 is attached to the living creature MB2 (at least one of a human being MB21 and another living creature MB22) to collect images 20, efficient acquisition of images 20 can be expected.

[0039] When multiple imaging devices 2 are operated, the type of moving body MB to which each imaging device 2 is attached may be selected arbitrarily. The types of moving body MBs to which each imaging device 2 is attached may be at least partially the same or different. In one example, multiple moving body MBs may be operated. The multiple moving body MBs may include one or more device MB1 and one or more living organism MB2. The number of imaging devices 2 attached to one moving body MB may be one, or two or more. The information processing device 1 may correct one or more movement trajectories 300 obtained for one moving body MB or imaging device 2. The information processing device 1 may correct multiple movement trajectories 300 obtained for multiple moving body MBs or imaging devices 2.

[0040] (Attached separately to the moving object) The method for separately attaching the imaging device 2 to the mobile body MB may be determined appropriately depending on the embodiment. For example, if the mobile body MB is device MB1, separately attaching the imaging device 2 to the mobile body MB may include attaching the imaging device 2 to the mobile body MB using a control system separate from device MB1 constituting the mobile body MB. Attaching the imaging device 2 to the mobile body MB using a separate control system may include externally attaching the imaging device 2 to the mobile body MB using any method, such as physically fixing the imaging device 2 to the housing of the mobile body MB. The attachment method may be a known method such as gluing, screwing, fitting, or using a fastener. The fastener may include a clamp, a peg, a clip, or the like. If the mobile body MB (device MB1) has a power source (such as a battery), it may be equipped with an external interface capable of receiving external power, such as a USB (Universal Serial Bus). In this case, attaching the imaging device 2 to the mobile body MB using a separate control system may include connecting the imaging device 2 to the external interface of the mobile body MB and operating the imaging device 2 while receiving power from the power source of the mobile body MB.

[0041] Also, for example, when the moving body MB is a living body MB2, it can be attached separately to the moving body MB. The term "accessory" may include being held by the mobile body MB, being attached to the clothing or equipment of the mobile body MB, etc. Clothing may include any object worn on the body. Clothing may include, for example, clothing, accessories, etc. Clothing may include pet clothing. Accessories may include hats (including helmets), belts, bracelets, etc. Accessories may include pet collars, harnesses, leashes, etc. Equipment may be any object that can be carried by the mobile body MB in any manner, such as being held or worn. Equipment may include, for example, a bag, a basket, etc. When the imaging device 2 is used in a store, the equipment of the mobile body MB may include any equipment that can be used in a store, such as a shopping basket, a shopping bag grip, etc. The method of attaching the imaging device 2 to clothing or equipment is not particularly limited and may be determined appropriately depending on the embodiment. Known methods may be used for the attachment method. For example, separately attaching the imaging device 2 to the mobile body MB may include attaching it to a pocket of the mobile body MB's clothing.

[0042] [Movement trajectory correction] (Correction method) The movement condition constraint 70 may be configured by some kind of restriction (condition) imposed when the moving object MB moves. As long as the movement trajectory 300 can be accurately corrected, the movement condition constraint 70 may be determined appropriately depending on the embodiment. In one example, the movement condition constraint 70 may be specified depending on the task of the moving object MB. The information processing device 1 may correct the movement trajectory 300 so as to satisfy the movement condition constraint 70 while suppressing changes in the characteristics (speed information, etc.) of the measurement positions 30. Correcting the movement trajectory 300 may be configured by correcting each measurement position 30 constituting the movement trajectory 300. Corrected measurement positions 32 may be obtained by correcting the measurement positions 30. As a result, the corrected movement trajectory 325 may be configured by one or more corrected measurement positions 32. Any optimization method may be used as a method for correcting the movement trajectory 300.

[0043] Errors in the position measurement results (measured position 30) by the inertial sensor 3 can be caused mainly by the following two factors. The first factor is the possibility of calculation errors, such as noise from the inertial sensor 3 and estimation errors of the measurement values. The second factor is that the measurement values ​​by the inertial sensor 3 are obtained as relative values, and therefore may deviate from absolute values ​​in the coordinate system of real space (for example, at least one of the direction and scale may deviate). In one example, in order to reduce errors caused by either of these two factors, the following configurations may be adopted for the movement condition constraints 70 and the correction method.

[0044] (1) First method (loop closure) FIG. 3 schematically illustrates an example of a first method for correcting a movement trajectory 300 according to this embodiment.

[0045] In one example, the movement condition constraint 70 may include the moving object MB being located at the same location at multiple times during the operation period of the imaging device 2. Being located at the same location at multiple times may include the moving object MB being located at the exact same location at each time and the moving object MB being located at the substantially same location at each time. The range of the substantially same location may be defined as appropriate depending on the embodiment.

[0046] Since the imaging device 2 is attached to the moving body MB, when the moving body MB is located at the same location, the imaging device 2 is also located at the same location at the relevant time. Therefore, correcting the movement trajectory 300 may include correcting the movement trajectory 300 so that the measurement positions 30 at each of the multiple times on the movement trajectory 300 coincide with each other. This correction process can obtain a corrected movement trajectory 320 (measurement positions 32). When only the correction process of the first method is performed or when the correction process of the first method is performed last, the corrected movement trajectory 325 may be the corrected movement trajectory 320.

[0047] The time at which the devices are co-located may be determined as appropriate depending on the embodiment. The time at which the mobile body MB is located may be determined according to the task of the mobile body MB. The time at which the mobile body MB is located at the same location may be determined based on the measurement data 39 of the inertial sensor 3. The time at which the mobile body MB is located at the same location may be given in advance. The same location may be a known location such as a charging location, or may be an unknown location.

[0048] In one example, the moving object MB may be a robotic device MB111 that moves autonomously within a store, among the autonomously moving robotic devices MB11. The same location may be a charging point CP for the robotic device MB111. The charging point CP may be a location of a charging station installed in the store. For example, the movement condition may be set so that the movement starts from the charging point CP and ends upon returning to the charging point CP. In this case, by detecting the start and end of the movement of the moving object MB using the detection method based on the measurement data 39 described above, it is possible to identify multiple times (start and end times of movement) when the moving object MB is located at the same location.

[0049] 3, a scene is assumed in which movement starts from a charging point CP and ends by returning to the charging point CP. Therefore, the position P1 (start position) at the start time of the movement and the position P2 (end position) at the end time are identified as the same location (charging point CP). In this case, as shown in FIG. 3, the movement trajectory 300 may be corrected so that the position P1 at the start time of the movement and the position P2 at the end time are the same.

[0050] A loop closure calculation method may be used as a calculation method for correcting the measurement positions 30 at a plurality of times so that they coincide with each other. In one example, when the position is expressed in two dimensions, the measurement value (measurement position 30) of the position at each time (1 to T) is expressed as "p t =(x t ,y t ) t=1, ,T”. If we assume that the position measurement value at each time is arbitrarily converted, the converted position measurement value can be expressed as “q a_t =F(p t ,θ t )". The function F may be defined arbitrarily. For example, the function F can be expressed as "F(p t ,θ t )=p t +θ t " In this case, θ t are parameters of the function F, and are unknown parameters to be optimized. The function F may also be defined as in Equation 1 below.

[0051]

number

[0052] According to one example of this embodiment, by performing correction using the first method, it is expected that calculation errors will be reduced. Also, in one example of this embodiment, the same location may be the charging point CP. By using the travel condition that the robot device MB111 passes through the charging point CP, it is expected that the time when the robot device MB111 is located at the same location will be appropriately identified. As a result, A suitable correction of the movement trajectory 300 can be expected.

[0053] (2-1) Method 2-1 (Map Matching) FIG. 4 schematically shows an example of the 2-1 method for correcting the movement trajectory 300 according to this embodiment.

[0054] In one example, the movement condition constraint 70 may include limiting the movement of the moving object MB within a predetermined movement range. Correcting the movement trajectory 300 may include correcting the movement trajectory 300 so that it fits the shape of the movement range. That is, the movement trajectory 300 may be corrected so that the shape of the actual movement range in the movement trajectory 300 fits the shape of a predetermined (assumed) movement range. This correction process can obtain a corrected movement trajectory 321 (measurement position 32). When only the correction process of the 2-1 method is performed or when the correction process of the 2-1 method is performed last, the corrected movement trajectory 325 may be the corrected movement trajectory 321.

[0055] The movement range may be a range within the environment in which the moving object MB is expected to pass. The movement range may be defined as appropriate depending on the embodiment. The movement range may be defined at least partially automatically based on information obtained from the environment, or may be defined manually.

[0056] In one example, the mobile body MB may be a robotic device MB111 that autonomously moves within a store. The movement range given as the constraint 70 may be defined within the floor FR of the store. For example, a floor map or floor guide of the store may be provided as appropriate. The floor map or floor guide may define a movable range within the floor FR (such as a range within which customers can pass) and a non-movable range (such as the range of product shelves). In this case, the movable range defined in the floor map or floor guide may be adopted as the movement range as is. The movement range may be automatically defined according to the task of the mobile body MB. For example, if the mobile body MB (robotic device MB111) is a cleaning robot, the cleaning range of the cleaning robot within the floor FR of the store may be defined in advance. The cleaning range may be defined by a known method. In this case, the given cleaning range may be adopted as the movement range as is. The movement range may also be manually defined within the floor FR of the store. The movement range may be defined as the entire floor FR of the store, or as a partial area within the floor FR. For example, if the area around the cash register is prone to congestion and the movement of the mobile object MB is restricted so that it does not enter this area around the cash register, the movement range may be defined as the area within the store floor FR excluding the area around the cash register.

[0057] In the example of FIG. 4, an outer edge RE1 and an inner edge RE2 are provided within a store floor FR, walls or shelves exist at the outer edge RE1 and the inner edge RE2, and a situation is assumed in which movement is possible between the outer edge RE1 and the inner edge RE2 as an aisle. In this situation, the space between the outer edge RE1 and the inner edge RE2 may be set as a movement range R1. Accordingly, the movement trajectory 300 may be corrected so that the shape of the actual movement range of the movement trajectory 300 matches the shape of the movement range R1. Note that, as exemplified in FIG. 4, as a result of the correction using method 2-1, it may be acceptable for part of the corrected movement trajectory 321 (measurement position 32) to deviate from the given movement range (movement range R1).

[0058] A map matching calculation method may be used as a calculation method for correcting the position to fit the shape of a given movement range. In one example, when the position is expressed in two dimensions, the measured value of the position (measured position 30) at each time (1 to T) is expressed as "p t =(x t ,y t ) t=1, ,T”. Within a given moving range (moving range R1), N points (p e_n ) may be randomly sampled. Each point is called "p e_n =(x e_n ,y e_n ) n=1,...,N. If we assume that the measured value is scaled, rotated, and translated, the measured value of the position after the transformation is "q b_t =s b (R b ·p t )+u b " can be expressed as: s b , R b and u b indicates the scalar value of the scale transformation, the rotation matrix, and the translation vector (2D vector). b , R b and u b is the unknown parameter to be optimized. For each point (p e_n ) and the transformed set of measurements (qb_t ) so that the distance between b , R b and u b The parameters of the above may be optimized. The distance between the sets may be expressed by a known distance index such as the Chamfer distance or the Jaccard index. The optimization method may include gradient descent, etc. A known method such as the above may be used. A known optimizer such as Adam may be used for the gradient descent calculation. This calculation allows the movement trajectory 300 to be corrected so as to fit the shape of the given movement range. Note that when the measurement position 30 is expressed in three dimensions, the above calculation method may be appropriately extended to three dimensions.

[0059] According to one example of this embodiment, by performing correction using method 2-1, it is expected that errors due to deviation from the coordinate system of the real space can be reduced. Also, in one example of this embodiment, the movement range may be defined within the floor FR of the store. In this way, by utilizing the movement condition that the robot device MB111 moves within the store, it is expected that the movement range limit can be appropriately specified. As a result, it is expected that the movement trajectory 300 can be appropriately corrected.

[0060] Note that the 2-1 method may be used together with or instead of the above-described first method. When both the first method and the 2-1 method are employed, either the first method or the 2-1 method may be executed first. When the correction process using the first method is executed first, the correction process using the 2-1 method may be executed on the corrected movement trajectory 320 (measured position 32) obtained by the first method. This is expected to result in a movement trajectory 321 (measured position 32) with further reduced error. Similarly, when the correction process using the 2-1 method is executed first, the correction process using the first method may be executed on the corrected movement trajectory 321 (measured position 32) obtained by the 2-1 method. This is expected to result in a movement trajectory 320 (measured position 32) with further reduced error.

[0061] (2-2) Method 2-2 (Particle filter) 5 schematically illustrates an example of the 2-2 method for correcting the movement trajectory 300 (321) according to this embodiment. In one example, when the correction by the 2-1 method is adopted, the correction by the 2-2 method may be adopted additionally. The correction process by the 2-2 method may be performed at any timing after the correction process by the 2-1 method is performed.

[0062] In one example, when the correction by the 2-2 method is adopted, the movement condition constraints 70 may further include an internal constraint 700 within the movement range. Correcting the movement trajectory 300 (321) may further include correcting the movement trajectory 321 so as to satisfy the internal constraint 700. This correction process can obtain a corrected movement trajectory 322 (measurement position 32). When the correction process by the 2-2 method is performed last, the corrected movement trajectory 325 may be the corrected movement trajectory 322.

[0063] The internal constraints 700 may be configured by some restrictions (conditions) imposed on the movement of the moving body MB within the movement range. The internal constraints 700 may be defined appropriately depending on the embodiment. The internal constraints 700 may be defined at least partially automatically based on information obtained from the environment, or may be defined manually.

[0064] In one example, the internal constraint 700 may include a requirement that the moving body MB not pass through an obstacle range defined by the presence of an obstacle. Correcting the movement trajectory 321 to satisfy the internal constraint 700 may be performed to bypass the obstacle range (i.e., the moving body MB not pass through the obstacle range). The method may include correcting the movement trajectory 321 to correct the obstacle. The obstacle may be any object that obstructs the passage of the moving body MB. The type of obstacle may be determined appropriately depending on the embodiment of the moving body MB, the environment, etc. For example, if the moving body MB is a robotic device (robot device MB111) that autonomously moves within a store, the obstacle may include a platform, a shelf (display shelf, etc.), a wall, etc. The obstacle range may be an area where the entry of the moving body MB is restricted due to the presence of an obstacle. The obstacle range may be given statically or dynamically. The obstacle range may be defined in advance or may be identified by observing the environment. For example, an obstacle present in the environment may be detected by observing the environment using a sensor such as an imaging device. The obstacle range may be identified appropriately from the location where the obstacle is detected. In one example, if the moving body MB is present within a store, the obstacle range may be set within the store.

[0065] 5, similar to FIG. 4, a situation is assumed in which walls or shelves exist at the outer edge RE1 and the inner edge RE2, and the space between the outer edge RE1 and the inner edge RE2 is set as the movement range R1. In this situation, the space outside the outer edge RE1 and the space inside the inner edge RE2 may be set as obstacle ranges (R2, R3). Accordingly, the movement trajectory 321 may be corrected to avoid the obstacle ranges (R2, R3).

[0066] The internal constraint 700 does not have to be limited to the above-described obstacle range constraint. The internal constraint 700 may include other constraints in addition to or instead of the obstacle range constraint. For example, the other constraints may include a no-entry range that prohibits the entry of a moving object MB. The no-entry range may be a range through which the moving object MB can pass but into which the moving object MB is prohibited. The no-entry range may be defined at least partially automatically based on information obtained from the environment, or may be defined manually. For example, an area near a cash register may be defined as a no-entry range because it is prone to congestion. The no-entry range may be treated in the same way as the above-described obstacle range. In other words, correcting the movement trajectory 321 to satisfy the internal constraint 700 may include correcting the movement trajectory 321 to detour around the no-entry range.

[0067] The obstacle range and the no-entry range are ranges that impose a restriction that the moving body MB does not pass through. Therefore, in another example, at least one of the obstacle range and the no-entry range may be included in the definition of the movement range by being excluded from the movement range rather than being given as the internal constraint 700. Furthermore, the internal constraint 700 does not have to be limited to constraints that impose a no-passage range, such as the obstacle range and the no-entry range, and may include constraints that impose restrictions (conditions) other than the no-passage range, such as a range where passage is unlikely or a range where passage is likely.

[0068] A particle filter calculation method may be used as a calculation method for correcting the movement trajectory 321 so as to satisfy the internal constraint 700. In one example, when the position is expressed in two dimensions, the measured value (measured position 32) of the position at each time (1 to T) on the movement trajectory 321 is expressed as "p c_t =(x c_t ,y c_t ) t=1, ,T”. The amount of movement at each time on this movement trajectory 321 can be expressed as “v c_t =p c_t+1 -p c_t ". The measurement position of interest is p c_t Assuming that, p c_t +v c_t A plurality of candidate points of movement may be sampled from the surroundings of the target point. Among the sampled candidate points, (A) the candidate point with the movement amount v c_t and (B) satisfying the internal constraint 700. The extracted candidate point is used as the next measurement position p c_t+1 In one example, satisfying the internal constraints 700 may include not entering an obstacle area (not colliding with an obstacle). c_t+1 Similarly, by sampling multiple candidate points and extracting candidate points that satisfy the above conditions (A) and (B), the next measurement position p c_t+2By repeating this calculation from the start time to the end time of the movement, the movement trajectory 321 can be corrected so as to satisfy the internal constraint 700 (bypass the obstacle area). Note that when the measurement position 32 is expressed in three dimensions, the above calculation method may be appropriately extended to three dimensions.

[0069] According to one example of this embodiment, after reducing errors due to deviation from the coordinate system of the real space by correction using method 2-1, correction using method 2-2 can be performed to further reduce calculation errors. As a result, it is expected that a movement trajectory 322 with reduced errors can be obtained. Furthermore, in one example of this embodiment, the internal constraint 700 may include a requirement that the moving object MB not pass through an obstacle range. By using information on the range where the obstacle exists (obstacle range) as the internal constraint 700, it is expected that the movement trajectory 321 can be appropriately corrected.

[0070] Note that the 2-1 method and the 2-2 method may be used together with or instead of the above-described 1st method. When all of the 1st method, the 2-1 method, and the 2-2 method are employed, the correction process by the 1st method may be performed before the correction process by the 2-1 method, between the correction processes by the 2-1 method and the 2-2 method, or after the correction process by the 2-2 method. When the correction process by the 1st method is performed after the correction process by the 2-1 method, and the correction process by the 2-2 method is performed after the correction process by the 1st method, the correction process by the 2-2 method may be performed on the corrected movement trajectory 320 (measured positions 32) obtained by the 1st method. This is expected to result in a movement trajectory 322 (measured positions 32) with further reduced error. Similarly, when the correction process by the first method is performed after the correction process by the 2-2 method, the correction process by the first method may be performed on the corrected movement trajectory 322 (measured positions 32) obtained by the correction process by the 2-2 method. This is expected to result in a movement trajectory 320 (measured positions 32) with further reduced error.

[0071] (Execution timing) The movement trajectory 300 may be corrected afterward. In one example, when a correctable amount of history of the measurement positions 30 has been accumulated, the information processing device 1 may perform a correction process on the measurement positions 30 (movement trajectory 300) accumulated up to that point in time. The information processing device 1 may also perform a correction process on the movement trajectory 300 that is temporarily stored. The information processing device 1 may also perform a correction process on the movement trajectory 300 after the movement trajectory 300 has been registered in a database or the like.

[0072] The information processing device 1 may periodically perform a correction process on the movement trajectory 300. For example, a movement unit may be given. The movement unit may be arbitrarily defined. The movement unit may be defined according to the task of the moving body MB, such as one journey, a period for performing a task, etc. For example, if the moving body MB starts moving from a specific point and ends its movement by returning to the specific point, one journey may be defined as the period from starting movement from the specific point to returning to the specific point. The specific point may be arbitrarily given. For example, if the moving body MB is an autonomously moving robotic device, the specific point may be a point originating from the robotic device, such as a charging point (charging point CP) for the robotic device. Furthermore, the movement unit may be defined in time units, such as a specific movement period. The movement unit may be defined by a movement period, such as a time period (morning, noon, night, etc.), a day, a day of the week, etc. When this movement unit is given, the information processing device 1 may perform a correction process on the movement trajectory 300 for each movement unit. As a specific example, if a unit of navigation is defined, such as starting movement from a specific point and returning to the specific point, the information processing device 1 may perform a correction process on the movement trajectory 300 for each navigation.

[0073] In this embodiment, since the inertial sensor 3 is used to measure the position, the measured position 30 (position measurement result) is obtained as a relative value from the initial position (initial value). The measured position 30 at each time is derived from the difference with the position at a past time. In other words, the measured position 30 at each time is derived based on the measured position 30 at a past time. In one example, a correction process is performed on the measured positions 30 (movement trajectory 300) accumulated up to a certain time (hereinafter also referred to as "correction time"). After performing the above, the inertial sensor 3 may continue to measure the position. The measured positions 30 after the correction time newly obtained by the inertial sensor 3 may be accumulated following the corrected measured positions 32 up to the correction time. In other words, the measured positions 30 after the correction time may be derived based on the corrected measured positions 32 up to the correction time. While the inertial sensor 3 continues to measure the position, there may temporarily be a time when at least a part of the measured positions (movement trajectories) collected by the inertial sensor 3 is composed of the corrected measured positions 32 (movement trajectory 325), and the rest of the collected measurement results are composed of the measured positions 30 (movement trajectory 300).

[0074] [Arrangement form] The information processing device 1, the imaging device 2, and the inertial sensor 3 may be arranged arbitrarily as long as the conditions that the imaging device 2 is separately attached to the moving body MB and the inertial sensor 3 is arranged so as to be able to measure the position of the imaging device 2 (moving body MB) are satisfied. The information processing device 1, the imaging device 2, and the inertial sensor 3 may be provided integrally, or at least partially separately. When the image group 200 and the movement trajectory 300 are directly acquired, the information processing device 1 may be connected to the imaging device 2 and the inertial sensor 3. In this case, the information processing device 1, the imaging device 2, and the inertial sensor 3 may constitute an imaging system. When the movement trajectory 300 is indirectly acquired via another computer or the like, the information processing device 1 does not need to be connected to the inertial sensor 3. When the image group 200 is not acquired, or when the image group 200 is indirectly acquired via another computer or the like, the information processing device 1 does not need to be connected to the imaging device 2.

[0075] In one example, as shown in FIG. 1, the information processing device 1 may constitute a terminal device UT together with an imaging device 2 and an inertial sensor 3. That is, the information processing device 1, the imaging device 2, and the inertial sensor 3 may be integrally arranged as components of the terminal device UT. The terminal device UT may be, for example, a mobile phone (smartphone, etc.), a tablet terminal, a notebook PC (Personal Computer), a dedicated device, etc. In one example of this embodiment, in a situation where the terminal device UT is used, This can assist in appropriate collection of images associated with the measurement positions. However, the arrangement of the information processing device 1 is not limited to this example.

[0076] FIG. 6 schematically illustrates an example of another arrangement of the information processing device 1 according to this embodiment. In another example, the imaging device 2 and the inertial sensor 3 may be integrally provided, and the information processing device 1 may be separately provided. For example, the terminal device UT1 may include the imaging device 2, the inertial sensor 3, and the controller 40. That is, similar to the example of FIG. 1, the imaging device 2 and the inertial sensor 3 may be provided as components of the terminal device UT1. The terminal device UT1 may be similar to the terminal device UT. The controller 40 may be configured with one or more computers. The controller 40 may be similar to the information processing device 1 of the terminal device UT. However, unlike the example of FIG. 1, the information processing device 1 may be provided separately from the terminal device UT1. The controller 40 may not operate as the information processing device 1, but may operate as a communication device that performs wired or wireless data communication with the information processing device 1. The communication standard is not particularly limited and may be selected appropriately depending on the embodiment. The information processing device 1 may be directly or indirectly connected to the controller 40, and may acquire the movement trajectory 300 via the controller 40. The derivation of the measurement position 30 (the formation of the movement trajectory 300) may be executed on the controller 40, on the information processing device 1, or on another computer. In one example, the information processing device 1 may acquire the image group 200 via the controller 40. In the example of FIG. 6, the information processing device 1 may be a terminal device separate from the terminal device UT1, a general-purpose PC, a server device, a dedicated device, or the like.

[0077] In yet another example, at least one of the image capturing device 2 and the inertial sensor 3 may be provided separately from the terminal device (UT, UT1). When the image capturing device 2 and the inertial sensor 3 are provided separately from the terminal device (UT, UT1) and configured to be able to communicate with each other via wire or wirelessly, the controller 40 may be omitted. The information processing device 1, the image capturing device 2, and the inertial sensor 3 are each The imaging device 2 may be provided separately. The imaging device 2 may be configured by a sensor that generates an image or image-like data through observation. The type of imaging device 2 may be selected arbitrarily. The imaging device 2 may include, for example, a general RGB camera, a half-sphere camera, a celestial sphere camera, etc.

[0078] [output] The corrected movement trajectory 325 may be output as appropriate depending on the embodiment. In one example, the information processing device 1 may output the corrected movement trajectory 325 to at least one of an output device and a storage area. The output device may be, for example, an output device of the information processing device 1 (output device 15 described below) or an output device of another computer. Outputting to a storage area may mean storing the corrected movement trajectory 325 in the storage area. The storage area to which the corrected movement trajectory 325 is output may be, for example, a memory resource of the information processing device 1, a memory resource of another computer, an external storage device, etc. The external storage device may include a data server such as a NAS (Network Attached Storage). The storage area for storing the corrected movement trajectory 325 may be changed as appropriate before the corrected movement trajectory 325 is used, such as by temporarily storing the corrected movement trajectory 325 in the memory resource of the information processing device 1 and then transferring it from the information processing device 1 to an external storage device. Similarly, the storage area for storing the image group 200 may also be changed as appropriate before the image group 200 is used. In one example, the image group 200 and the corrected movement trajectory 325 may be transferred as appropriate via a network, a storage medium, etc.

[0079] In one example, when acquiring the image group 200 as shown in FIG. 1 , outputting the corrected trajectory 325 may include associating and saving the image group 200 and the corrected trajectory 325. The image group 200 may be acquired at any timing before associating with the corrected trajectory 325. The corrected trajectory 325 may be associated with the image group 200 in any manner. In one example, the corrected trajectory 325 may be directly associated with the image group 200. In another example, associating the corrected trajectory 325 with the image group 200 may include associating each image 20 included in the image group 200 with a measurement position 32 on the corrected trajectory 325 at the time of capturing each image 20. Note that the computer that associates the image group 200 and the corrected trajectory 325 is not limited to the information processing device 1. The association of the image group 200 and the corrected trajectory 325 may be performed by a computer other than the information processing device 1. For example, if the image group 200 is collected by another computer, the information processing device 1 does not need to acquire the image group 200. In this case, the information processing device 1 may output the corrected movement trajectory 325 to the other computer, thereby causing the other computer to associate the image group 200 with the corrected movement trajectory 325.

[0080] [Image usage scenarios] One or more images 20 (image group 200) captured by the imaging device 2 may be stored as appropriate. The one or more stored images 20 (image group 200) may be used arbitrarily.

[0081] (1st usage scenario) FIG. 7 schematically shows an example of a usage scenario of images 20 (image group 200) captured by the imaging device 2 according to this embodiment. In one example, a storage device MR may be provided for storing the images 20 (image group 200). When the information processing device 1, the imaging device 2, the inertial sensor 3, and the storage device MR are connected, an imaging system may be constructed by the information processing device 1, the imaging device 2, the inertial sensor 3, and the storage device MR. The storage device MR may store the images 20 (image group 200) captured by the imaging device 2 in association with a corrected measurement position 32 (movement trajectory 325) of the imaging device 2 at the time of capturing the image 20. Each measurement position 32 may indicate the capturing position of the corresponding image 20. The storage device MR may further store the capturing time TT of each image 20. Each image 20 may be further associated with the capturing time TT of each image 20. The image time TT may be omitted.

[0082] In one example, each combination of image 20, measurement position 32, and time TT may be stored as a database in storage device MR. A search device 5 may be provided to search this database in response to a request from user U1. When an imaging system is constructed from information processing device 1 and storage device MR, search device 5 may be provided as a component within the imaging system, or may be provided as a component external to the imaging system. In one example, as shown in FIG. 7, search device 5 may be configured by a computer other than information processing device 1. In another example, information processing device 1 may operate as search device 5. That is, search device 5 may be configured by information processing device 1.

[0083] The database may be constructed by any computer. In one example, at least one of the information processing device 1, the search device 5, and the other information processing device 45 may be involved in constructing the database. The other information processing device 45 may be a computer other than the information processing device 1 and the search device 5. The other information processing device 45 may be composed of one or more arbitrary computers. The storage device MR may be arranged arbitrarily. In one example, the storage device MR may be composed of at least one of the memory resources of the information processing device 1, the memory resources of the search device 5, the memory resources of the other information processing device 45, and an external storage device.

[0084] The search device 5 may be one or more computers configured to search a database (storage device MR) in response to a request from a user U1. The user U1 may be any person. In one example, the search device 5 may receive a query 60 from a terminal device T1 of the user U1. The terminal device T1 may be connected to the search device 5 via a network or directly. The query 60 may include at least one of a target image, a target location, and a target time to be searched. The user U1 may operate the terminal device T1 to appropriately specify at least one of the target image, the target location, and the target time. The target image may consist of one image or multiple images. The target location may be specified by a single point or a range. The target time may be specified by a single time or a range (such as a time period, a date, or a day of the week).

[0085] The search device 5 may extract elements matching the query 60 from the database (storage device MR) by searching the database. As a result, the search device 5 can obtain a search result 65 composed of the extracted elements. Whether or not the elements match the query 60 may be determined appropriately. The extracted elements may be composed of at least one of an image 20, a measurement position 32, and an image capture time TT. For example, if the query 60 includes a target image, the search device 5 may compare each image 20 stored in the storage device MR with the target image, and based on the comparison result, extract an image 20 that matches the target image from among the images 20 stored in the storage device MR. In one example, the search device 5 may further extract an image 20 similar to the target image. The criterion for similarity may be defined arbitrarily, for example, that the degree of match between the images exceeds a threshold. Similarly, when the query 60 includes a target position, the search device 5 may compare each measurement position 32 stored in the storage device MR with the target position, and extract measurement positions 32 that match the target position from among the measurement positions 32 stored in the storage device MR based on the comparison result. In one example, the search device 5 may further extract measurement positions 32 that belong to a neighborhood range of the target position. The neighborhood range may be defined arbitrarily, for example, by determining whether the deviation from the target position is less than a threshold. Furthermore, when the query 60 includes a target time, the search device 5 may compare each imaging time TT stored in the storage device MR with the target time, and extract imaging times TT that match or belong to the target time from among the imaging times TT stored in the storage device MR based on the comparison result.

[0086] In one example, when extracting a search element from the database, the search device 5 may further extract other elements associated with the search element. When a target position is specified as the query 60, the search result 65 may include at least one of the image 20 and the measurement position 32 associated with the extracted image 20, in addition to the image 20 extracted as matching the target image. When a target position is specified as the query 60, the search result 65 may include at least one of the image 20 and the image capture time TT associated with the extracted measurement position 32, in addition to the measurement position 32 extracted as matching the target position. When a target time is specified as the query 60, the search result 65 may include at least one of the image 20 and the measurement position 32 associated with the extracted image capture time TT, in addition to the image capture time TT extracted as matching the target time. When there is no element matching the query 60, the search result 65 may be empty.

[0087] Then, the search device 5 may return the obtained search results 65 to the terminal device T1. The terminal device T1 may receive the search results 65 from the search device 5 and output the received search results 65 as appropriate. This allows the user U1 to obtain the search results 65 in response to the request (query 60). The search results 65 may be used for any purpose. Note that if the imaging time TT is omitted, the search by target time may be omitted.

[0088] As a specific example, a database (storage device MR) and a search device 5 may be provided in a scene where a mobile object MB is an entity moving within a store and an image capturing device 2 collects images 20 (image group 200) of the inside of the store. In this scene, a user U1 may be a customer, a store clerk, etc. The mobile object MB, which is an entity moving within the store, may be, for example, a robot device (robot device MB111) that moves autonomously within the store, a device that moves manually within the store, a living thing that moves within the store, etc.

[0089] For example, the user U1 may request a search for an image 20 showing a target object (such as a product, a sign, or an advertisement) that may be present in a store by specifying the target image 20 as a query 60. If the target object is a product, the target image may be a product image of the product. In one example, if the target object is a product, the target image (product image) showing the target object may be provided from coupon information for the product. In response to receiving the query 60 including the target image showing the target object from the terminal device T1, the search device 5 may search for an image 20 that matches the query 60 (i.e., an image 20 showing the target object). The search device 5 may extract the image 20 showing the target object from a database (storage device MR) and further extract a measurement location 32 associated with the extracted image 20. The search device 5 may return the extracted image 20 and the measurement location 32 to the terminal device T1 as a search result 65. The terminal device T1 may appropriately output the measurement location 32 obtained as the search result 65. This allows the user U1 to confirm the position of the target object. In one example, the terminal device T1 may perform route guidance from the current position of the user U1 to the position of the target object. The current position of the user U1 may be obtained appropriately by the terminal device T1 using a known method. The route guidance may be performed using a known method. The measured position 32 may be used as the position of the target object as is. That is, the terminal device T1 may perform route guidance to the measured position 32. Alternatively, the position of the target object may be estimated appropriately from the measured position 32, the position where the target object appears in the image 20, the size of the target object, etc. A known image processing method may be used to estimate the position of the target object. The estimation of the position of the target object may be performed by any computer. The terminal device T1 may perform route guidance to the estimated position.

[0090] Furthermore, for example, the user U1 may request a search for a specified position or a specified range by specifying an arbitrary position or a range of positions within a store as a target position. In response to receiving a query 60 including the specified position or the specified range as a target position from the terminal device T1, the search device 5 may search for a measurement position 32 that matches the target position. The search device 5 may extract the matching measurement positions 32 and further extract the images 20 associated with the extracted measurement positions 32. The search device 5 may return the extracted measurement positions 32 and images 20 to the terminal device T1 as search results 65. The terminal device T1 may appropriately output the images 20 obtained as the search results 65. This allows the user U1 to confirm the scene at the specified position or within the specified range via the output images 20. In one example, the terminal device T1 may perform an information search using the obtained images 20. For example, the terminal device T1 may perform a web search for information on objects (products, signs, advertisements, etc.) appearing in the images 20 by providing the obtained images 20 to a search engine. The terminal device T1 may provide the obtained images 20 to the search engine as they are, or may extract an area in which the object appears from the obtained images 20 and provide a partial image of the extracted area to the search engine. A known search engine may be used as the search engine. The terminal device T1 may appropriately output search results obtained from the search engine. This allows user U1 to obtain information about the object appearing in image 20. The information to be searched on the web may be selected as appropriate depending on the embodiment. In one example, the object may be a product, and terminal device T1 may obtain any information related to the product appearing in image 20 as a result of the web search, such as coupon information, review information, price information, etc.

[0091] Furthermore, for example, the user U1 may request a search for a specified time by specifying an arbitrary time as the target time. In response to receiving a query 60 including the specified time as the target time from the terminal device T1, the search device 5 may search for an image capture time TT that matches the target time. The search device 5 may extract an image capture time TT that matches the target time and further extract an image 20 associated with the extracted image capture time TT. The search device 5 may return the extracted image capture time TT and the image 20 to the terminal device T1 as a search result 65. This allows the user U1 to check the image 20 obtained at the specified time. In one example, the search device 5 may further extract a measurement position 32 associated with the extracted image capture time TT. This allows the search result 65 to further include the measurement position 32. This allows the user U1 to check the image 20 as well as the image capture position of the image 20.

[0092] According to one example of this embodiment, by associating the measurement position 32 with the image 20 and storing it in the storage device MR, it is possible to provide information based on a combination of the measurement position 32 and the image 20. For example, as shown in the example of FIG. 7 above, a search system (search device 5) can be provided for accessing from one of the image 20 and the measurement position 32 to the other. Note that the form of searching the database is not limited to the above example and may be changed as appropriate depending on the embodiment. In another example, the search device 5 may be directly operated to search the database (storage device MR) without going through the terminal device T1. Furthermore, the terminal device T1 may be configured to directly access the storage device MR without going through the search device 5 to search the database (storage device MR).

[0093] (Second usage scenario) FIG. 8 schematically shows another example of a usage scenario of the image 20 captured by the imaging device 2 according to this embodiment. In one example, a memory device MR may be provided, similar to FIG. 7 . The memory device MR may store each image 20, the measurement position 32 at the time of capturing each image 20, and the capturing time TT in association with each other. The capturing time TT may be omitted. In addition, the memory device MR may further store a result 25 of image analysis for each image 20. Each image 20 may be further associated with the result 25 of image analysis for each image 20. That is, the memory device MR may store the image 20 in association with the result 25 of image analysis for each image 20. Except for the point related to the result 25 of image analysis, the configuration of FIG. 8 may be the same as the configuration of FIG. 7 .

[0094] The image analysis of the image 20 may be performed by any computer at any timing after the image 20 is acquired. In this case, the information processing device 1 may perform image analysis on the image 20 at any timing after acquiring the image 20. The information processing device 1 may perform image analysis each time an image 20 is acquired, or may perform image analysis on each of multiple images 20 collectively after acquiring multiple images 20. The information processing device 1 may perform image analysis using any method. The information processing device 1 may perform image analysis using a known method, such as a general image analysis method (edge ​​extraction, pattern matching, etc.) or a method using a trained machine learning model. This allows the information processing device 1 to acquire an image analysis result 25. The information processing device 1 may store the acquired image analysis result 25 in association with the image 20. Note that the computer that performs the image analysis is not limited to the information processing device 1 and may be changed as appropriate depending on the embodiment. In another example, the search device 5 or another information processing device 45 may perform image analysis on the image 20 and store the image analysis result 25 in association with the image 20. The image analysis result 25 may be stored in the storage device MR immediately, or may be stored in the storage device MR via another storage area.

[0095] Image analysis may detect any feature appearing in image 20, and the results 25 of the image analysis may include any information about the detected feature.

[0096] In one example, the image analysis may include detection of an object OB appearing in the image 20. As a result, the image analysis result 25 may include a detection result of the object OB. If multiple objects OB appear in the image 20, a detection result may be obtained for each object OB. The object OB is an example of a feature appearing in the image 20. The type of object OB is not particularly limited and may be appropriately selected depending on the embodiment. The object OB may include, for example, products / merchandise (food, beverages, daily necessities, clothing, electrical appliances, etc.), tools (including household goods such as desks, chairs, shelves, etc.), equipment (kitchen equipment, refrigeration equipment, cash registers, etc.), architectural structures (entrances, doors, windows, pillars, etc.), monuments, artworks (statues, paintings, etc.), parts of these, etc. The object OB may also be referred to as a landmark.

[0097] In one example, detecting the object OB may include extracting the range (such as a bounding box) in which the object OB appears. Thus, the detection result of the object OB (image analysis result 25) may include the extraction result of the range in which the object OB appears (such as range information, a partial image, etc.). Also, in one example, detecting the object OB may include analyzing the attributes of the object OB, such as classifying the object OB, identifying the object OB, estimating the position of the object OB, and analyzing text information attached to the object OB. Thus, the detection result of the object OB may include the analysis result of the attribute information of the object OB. The attribute information of the object OB may include, for example, a category, identification information (such as a name or an identifier), an estimated position, text information attached to the object OB, etc. The attribute information of the object OB may include any information about the object OB obtained from reference information such as the web. For example, the attribute information of the object OB may include information about the components, elements, and other characteristics of the object OB. In one example, if the object OB is absent in the image 20, the detection result of the object OB may be configured to indicate that the object OB is not present. Alternatively, if the object OB is absent in the image 20, the association of the detection result of the object OB may be omitted.

[0098] The object OB to be detected may be selected as appropriate depending on the embodiment such as the operating scene. In one example, the moving object MB may be an entity moving within the store, and the object OB may be a product OB1 sold within the store. As a result, the detection result of the object OB may include the detection result of the product OB1. In one example, the detection result of the product OB1 may include attribute information of the product OB1. The attribute information of the product OB1 may include, for example, the category of the product OB1, identification information (product name, identifier, etc.), estimated location, attached character information, other product information, etc. Note that when the moving object MB is an entity moving within the store, the object to be detected may be The target object OB does not have to be limited to the product OB1. The object OB may include an object other than the product OB1 present in the store, such as a signboard or advertisement, together with or instead of the product OB1.

[0099] In one example, the image analysis may include detection of an event EV captured in the image 20, in addition to or instead of detecting the object OB. Thus, the image analysis result 25 may include a detection result of the event EV. When multiple event EVs are captured in the image 20, a detection result may be obtained for each event EV. An event EV is an example of a feature captured in the image 20. The event EV may be defined appropriately depending on the embodiment, such as the purpose of detection. In one example, the event EV may be any phenomenon related to an object captured in the image 20, such as the behavior, situation, or appearance of the object. The detection result of the event EV may include an analysis result of a phenomenon related to the object. The analysis result of the phenomenon may include attribute information such as the category of the event EV, the estimated position of the object (estimated position of the event EV), and the content of the phenomenon. The analysis result of the phenomenon (the content of the phenomenon, etc.) may be expressed in any data format, such as text or numerical values. The object for which the event EV is detected may include the object OB, or may include an object other than the object OB. The object for which the event EV is detected may be selected appropriately depending on the embodiment. The number of objects involved in an event EV (event) may be one or more.

[0100] The event EV to be detected may be selected as appropriate depending on the embodiment of the operation scene, etc. In one example, the mobile object MB may be an entity that moves within a store, and the event EV may include at least one of a customer event EV1 related to customers in the store and a product event EV2 related to products sold in the store.

[0101] The customer event EV1 may include any phenomenon related to customers. For example, the customer event EV1 may include whether or not a specific object is being viewed, the number of customers present, etc. The specific object may be, for example, a product, a sign, an advertisement, etc. Whether or not a specific object is being viewed is an example of customer behavior. The number of customers present is an example of a customer's situation or appearance. The product event EV2 may include any phenomenon related to products. For example, the product event EV2 may include the number of remaining products, whether or not a product is out of stock, the arrangement of products on a display shelf, etc. The number of remaining products, whether or not a product is out of stock, and the arrangement of products on a display shelf are examples of a product situation or appearance. The number of remaining products or whether or not a product is out of stock may be used as an indicator for replenishing products. The arrangement of products on a display shelf may be used as an indicator for moving products forward (for example, if products are only present at the back of the display shelf, a notification may be sent to perform the moving product forward operation).

[0102] It should be noted that when the mobile object MB is an entity moving within a store, the event EV to be detected need not be limited to the customer event EV1 and the product event EV2, and may be changed as appropriate depending on the embodiment. The event EV may include any event that may occur within the store. In another example, the event EV may include at least one of a store clerk event related to a store clerk in the store, a sales promotion event related to a product promotion held within the store, and an equipment event related to equipment present within the store, together with or instead of at least one of the customer event EV1 and the product event EV2. The store clerk event may include any phenomenon related to the store clerk. The store clerk event may include, for example, the work status of the store clerk, the category of work performed by the store clerk, etc. The work of the store clerk may include product promotion activities such as attracting customers. The sales promotion event may include any phenomenon related to product promotion. Product promotion may include, for example, the installation of a special shelf, the placement of advertisements (such as POP advertisements: Point of Purchase Advertising), attracting customers by the store clerk, etc. The promotion event may include, for example, the status of the promotion place, etc. The promotion event may include, for example, the behavior of customers at the promotion place (whether or not they are looking at the promotional product, etc.), the status of the promotional product (number of remaining products, etc.), The facility event may include any phenomenon related to the facility. The facility event may include, for example, the number of items remaining in the shopping cart, the number of items remaining in the shopping cart, the availability of lockers, etc. The facility event may include, for example, the facility usage status, such as the number of items remaining in the shopping cart, the number of items remaining in the shopping cart, the availability of lockers, etc.

[0103] Similar to the example of FIG. 7, the search device 5 may receive a query 60 from the terminal device T1. The search device 5 may search a database (storage device MR) to extract elements that match the query 60 from the database. As a result, the search device 5 may obtain a search result 65 that is made up of elements that match the query 60. The search device 5 may return the obtained search result 65 to the terminal device T1. Through this series of information processing, the search device 5 can provide information similar to that shown in FIG. 7.

[0104] 8, when at least one of a target image, a target position, and a target time is specified as the query 60, the search device 5 may further extract the image analysis result 25 associated with the extracted search element. As a result, the search result 65 may further include the image analysis result 25 associated with the extracted search element in addition to the extracted search element (at least one of the image 20, the measurement position 32, and the imaging time TT).

[0105] Furthermore, the query 60 may include search target information for image analysis, together with or instead of at least one of the target image, target location, and target time. The target information may be specified as appropriate depending on the embodiment. The target information may be configured to specify, for example, the category of the object / event to be searched, the name of the object (product name, etc.), text information attached to the object, the content of the phenomenon, and other attributes. The target information may be expressed in any data format, for example, text, numerical values, etc. When the query 60 includes the target information, the search device 5 may compare the image analysis results 25 for each image 20 stored in the storage device MR with the target information, and based on the comparison results, extract image analysis results 25 that match the target information from the image analysis results 25 stored in the storage device MR. Whether the image analysis results match the target information may be determined as appropriate based on the degree of text match, the degree of satisfaction of numerical conditions, etc. As a result, the search results 65 may include the image analysis results 25 extracted as matching the target information. When the image analysis result 25 includes multiple items, the search result 65 may consist of only information on the item specified as the target information, or may be configured to include information on items other than the item specified as the target information. For example, in a situation where the image analysis includes detection of an object OB, the object OB includes a product OB1, and the image analysis result 25 includes the product name and the estimated location of the product OB1, the query 60 may be specified to include the product name to be searched. In contrast, the search result 65 may consist of only the product name that matches the specified product name as the extracted image analysis result 25, or may be configured to further include the estimated location of the product OB1.

[0106] In one example, the search device 5 may extract the image analysis result 25 and further extract at least one of the image 20, the measurement position 32, and the image capture time TT associated with the extracted image analysis result 25. As a result, the search result 65 may include, in addition to the extracted image analysis result 25, at least one of the image 20, the measurement position 32, and the image capture time TT associated with the extracted image analysis result 25. Note that if the image analysis result 25 includes an extraction result (partial image) of the range in which the object OB appears, in the above information search, the extraction result of the range in which the object OB appears may be matched with the target image together with or instead of the image 20. If the image analysis result 25 includes the estimated position of the object OB, in the above information search, the estimated position of the object OB may be matched with the target position together with or instead of the measurement position 32.

[0107] As a specific example, assuming the above-described scene in a store, user U1 may request a search for a target product by specifying the product name as a query 60. In one example, the target product name (the name of the target product) may be provided from coupon information for the target product. In response to receiving the query 60 specifying the product name from terminal device T1, search device 5 may search for image analysis results 25 that match the query 60 (specified product name). The search device 5 may extract the image analysis results 25 that match the query 60 and further extract the images 20 and measurement positions 32 associated with the extracted image analysis results 25. The search device 5 may return the extracted images 20, measurement positions 32, and image analysis results 25 to terminal device T1 as search results 65. The terminal device T1 may appropriately output the images 20 obtained as search results 65. This allows user U1 to confirm the scene of the specified target product. If the image analysis result 25 includes an extraction result of the area in which the product OB1 is captured, the terminal device T1 may output the extraction result of the area in which the target product is captured together with or instead of the image 20. In one example, the terminal device T1 may perform a web search for information about the target product by providing the extraction result of the area in which the target product is captured to a search engine. The terminal device T1 may also appropriately output the measured position 32 obtained as the search result 65. This allows the user U1 to confirm the location of the target product. In one example, the terminal device T1 may perform route guidance from the current location of the user U1 to the location of the target product. The measured position 32 obtained as the search result 65 may be used as the location of the target product. Alternatively, if the image analysis result 25 includes an estimated location of the product OB1, the estimated location of the target product included in the image analysis result 25 may be used as the location of the target product. Note that the attribute information used to search for the target product is not limited to the product name. Attribute information other than the product name may also be used as the search query 60. The search device 5 may replace the product name with other attribute information and execute the above search process. For example, the user U1 may specify character information attached to the target object as the query 60. This allows the user U1 to obtain a search result 65 related to the target object attached with the specified character information.The target object may be not only the above-mentioned product but also an object other than a product, such as a signboard or advertisement. In one example, the terminal device T1 may perform route guidance to the position of an object other than a product (target object). According to one example of the present embodiment, as long as the image 20 is obtained, even if the position of the target object has been changed, the changed position of the target object can be tracked.

[0108] Furthermore, similar to the search based on the attribute information of the target product, the user U1 may specify the attribute information (category, content, etc.) of the target event as the target information (query 60). In response, the search device 5 may provide search results 65 related to the specified target event. In one example, if the search results 65 include at least one of the measured location 32 and the estimated location of the target event, the terminal device T1 may perform route guidance to the location of the target event. According to one example of the present embodiment, as long as the image 20 is obtained, even if the location of the target event is changed, the location of the changed target event can be tracked.

[0109] Further, for example, similar to the specific example of FIG. 7 , the user U1 may specify a target image (product image) showing a target product as the query 60. In response, the search device 5 may search for an image 20 matching the query 60 and provide the search result 65 to the terminal device T1. In addition, in the example of FIG. 8 , the image analysis result 25 may include an extraction result (partial image) of the area showing the product OB1. The search device 5 may search for the image analysis result 25 matching the query 60 by comparing the target image specified as the query 60 with the extraction result of the product OB1. In this search, similar to the search for the image 20, the search result 65 related to the target product can be provided to the user U1. In one example, if the search result 65 includes at least one of the measurement location 32 and the estimated location of the target product, the terminal device T1 may perform route guidance to the location of the target product.

[0110] Also, for example, similar to the specific example of FIG. 7 above, user U1 may specify an arbitrary position or range of positions within a store as a target position (query 60). In response, search device 5 may provide search results 65 related to the specified position or range. Additionally, in the example of FIG. 8 , search results 65 may include image analysis results 25 associated with the extracted measurement position 32. Image analysis results 25 may include extraction results (partial images) of the range in which product OB1 appears. In one example, terminal device T1 may perform a web search for information on the target product by providing the extraction results of the range in which the target product appears to a search engine.

[0111] According to one example of the present embodiment, by associating the measurement position 32 and the image analysis result 25 with the image 20 and storing them in the storage device MR, it is possible to provide information based on a combination of the measurement position 32, the image analysis result 25, and the image 20. For example, as shown in the example of FIG. 8 above, it is possible to provide a search system (search device 5) for accessing other elements from at least one of the image 20, the measurement position 32, and the image analysis result 25. According to one example of the present embodiment, the image analysis includes detection of an object OB appearing in the image 20, thereby providing information about the object OB. According to one example of the present embodiment, the object OB includes a product OB1, thereby providing information about the product OB1. According to one example of the present embodiment, the image analysis includes detection of an event EV appearing in the image 20, thereby providing information about the event EV. According to one example of the present embodiment, the event EV includes a customer event EV1, thereby providing information about the customer event EV1. Furthermore, according to one example of the present embodiment, the event EV includes a product event EV2, thereby providing information about the product event EV2. Note that, as with FIG. 7 , the form of searching the database is not limited to the above example and may be modified as appropriate depending on the embodiment. In another example, the search device 5 may be operated directly to search the database without going through the terminal device T1. Also, the terminal device T1 may access the storage device MR directly to search the database without going through the search device 5.

[0112] (others) The form of information provision does not have to be limited to the above search example, and may be changed as appropriate depending on the embodiment. That is, the system for providing information does not have to be limited to the above search system. At least a portion of the accumulated data (image 20, measurement position 32, image analysis result 25, and image capture time TT) may be utilized for any purpose other than search. The search device 5 may execute predetermined information processing utilizing the accumulated data together with or instead of the above search processing.

[0113] (1) Provided data FIG. 9 schematically illustrates an example of provided data according to this embodiment. In one example, the search device 5 may output at least a portion of the accumulated data as is. The search device 5 may generate provided data by processing at least a portion of the accumulated data and output the generated provided data. In FIG. 9, as an example of processing, a situation is assumed in which a point PT related to an image capture is plotted on a map MP of the environment. The map MP may be obtained as appropriate. For example, when a mobile object MB is operated within a store, the map MP may be obtained from floor information such as a floor map or floor guide. The point PT to be plotted may be the acquisition point of the image 20 (measurement position 32), or may be a point obtained as a result 25 of image analysis (such as the estimated position of an object OB or the estimated position of an event EV). The map MP including each plotted point PT may be output as an example of provided data.

[0114] When the measurement position 32 is used as the point PT, the point where the image 20 was acquired (the imaged range) can be viewed from above based on the output map MP. When the estimated position of the object OB is used as the point PT, at least one of the position of each object OB and the degree of detection of the object OB in each area can be confirmed based on the output map MP. In one example, if a mobile object MB is operated in a store and an object OB is a product, a product map (a map showing the location of the product) can be generated by plotting the estimated positions of the product on the map MP. Even if the location of the product is changed, a product map showing the new product location can be obtained by using the image analysis results 25 of the images 20 collected after the change. This makes it possible to track changes to the product. Furthermore, if the estimated position of the event EV is used as the point PT, the occurrence position of each event EV can be confirmed based on the output map MP. In one example, if the event EV includes a customer event EV1, the estimated position of the event EV may be the estimated position of the detected customer. By plotting this estimated position of the customer as the point PT, it is possible to confirm the degree of customer presence in each area of ​​the map MP. For example, it is possible to confirm areas with a large number of customers, areas with a small number of customers, etc.

[0115] In one example, the search device 5 may plot data belonging to a target movement unit on the map MP. For example, if the movement unit is defined as one voyage, the search device 5 may plot data obtained in one voyage on the map MP. This makes it possible to check environmental information obtained during one voyage (such as the point where the image 20 was acquired, the position of the object OB, and the position of the event EV).

[0116] (2) Real-time analysis In another example, the accumulated data may be used to analyze the environment in real time. For example, when image analysis including detection of an event EV is performed, the search device 5 may refer to the acquired image analysis result 25 in real time and detect the occurrence of a specific event. In response to detecting the occurrence of the specific event, the search device 5 may output a notification regarding the occurrence of the specific event.

[0117] As a specific example, if the image analysis result 25 includes a detection result of product event EV2, the search device 5 may detect the occurrence of a specific product-related event by referring to the obtained detection result of product event EV2. The specific product-related event to be detected may include, for example, a low remaining quantity of the product, a product being out of stock, or a product being present only at the back of a display shelf. The remaining quantity of the product may be evaluated appropriately based on a predetermined criterion. For example, the search device 5 may evaluate that the remaining quantity of the product is low if the remaining quantity of the product is equal to or less than a threshold value. The arrangement of the product on the display shelf may also be evaluated appropriately based on a predetermined criterion. For example, the arrangement of the product on the display shelf may be expressed by the distance from the front edge of the display shelf to the product. If there is no product for which this distance is equal to or less than a threshold value (i.e., there is no product within a predetermined range extending from the front edge of the product shelf toward the back of the product shelf), the search device 5 may evaluate that the product is present only at the back of the display shelf. The search device 5 may evaluate that a product is present only at the back of the display shelf even when the number of products present within the specified range of the product shelf is small. Each threshold value may be set arbitrarily. The search device 5 may appropriately output a notification regarding the detected specific event in response to detecting the occurrence of a specific product-related event. For example, when the search device 5 detects that the remaining number of products is low or that a product is out of stock, the search device 5 may output a notification to the store clerk instructing them to replenish the product on the display shelf. The search device 5 may also output a notification to the store clerk instructing them to purchase the product. When the search device 5 detects that a product is present only at the back of the display shelf, the search device 5 may output a notification to the store clerk instructing them to move the product forward.

[0118] Furthermore, if the image analysis result 25 includes a facility event detection result, the search device 5 may identify the facility usage status by referring to the acquired facility event detection result. The search device 5 may output the identified facility usage status as appropriate. For example, the search device 5 may output the facility usage status, such as the number of items remaining in a shopping basket, the number of items remaining in a shopping cart, and the availability of lockers. The status may be output.

[0119] (3) Post-mortem analysis In another example, the accumulated data may be used to analyze the environment after the fact. For example, if the image analysis result 25 includes a detection result of an event EV, the search device 5 may analyze the characteristics of the event EV occurring in the environment based on the accumulated image analysis result 25. The search device 5 may output the analysis result as appropriate.

[0120] As a specific example, if the event EV includes a customer event EV1, the search device 5 may analyze customer behavior from the image analysis result 25 (detection result of customer event EV1). Customer behavior is an example of a feature of an event EV. For example, the search device 5 may identify a location where a large number of customers are present by referring to the detection result of customer event EV1. The number of customers present may be appropriately evaluated based on a predetermined criterion. For example, the search device 5 may evaluate a location where a large number of customers are present if the number of customers exceeds or is equal to or greater than a threshold. The search device 5 may output the identified location where a large number of customers are present. This allows the location where a large number of customers are present to be identified. The search device 5 may also output an image 20 of the identified location. The reason for the large number of customers can be analyzed based on the output image. The location where a large number of customers are present may be the measurement location 32 associated with the detection result of customer event EV1 evaluated as having a large number of customers. The image 20 of the identified location may be the image 20 associated with the detection result of customer event EV1 evaluated as having a large number of customers. Furthermore, for example, the search device 5 may refer to the detection result of the customer event EV1 to determine whether or not a customer is looking at a specific object (such as a product, a sign, or an advertisement). The search device 5 may output the result of determining whether or not a specific object is being looked at. Locations where there are many customers and whether or not a specific object is being looked at are examples of analysis results of customer behavior. The analysis results of customer behavior may be used for any purpose, such as store design (layout of display shelves, placement of products / signs / advertisements, etc.).

[0121] Furthermore, if the event EV includes at least one of a customer event EV1, a product event EV2, a store clerk event, and a promotional event, the search device 5 may collect the detection results of the promotional features by referring to the image analysis results 25 (detection results of the event EV). The promotional features may include, for example, the status of the promotional activities by the store clerk, the status of the promotional location, the behavior of customers at the promotional location, the status of the promotional products, etc. The search device 5 may output the collected detection results of the promotional features. The promotional measures can be evaluated based on the output detection results of the promotional features. This makes it possible to consider improvements to the promotional measures.

[0122] Also, assume a situation in which a mobile MB is operated within a store, a target product is sold at multiple locations, and the mobile MB moves through each location. In this situation, the search device 5 may extract images 20 showing each location by searching a database using information about the target product as a query 60. The information about the target product may be, for example, the product name, product image, product attribute information, etc. The search device 5 may output each extracted image 20. By comparing the output images 20, it is possible to analyze the differences between the locations selling the target product. For example, it is possible to identify a location with a large number of customers from among multiple locations selling the target product.

[0123] Each of the above analyses may be performed for each target period (time period, date, day of the week, etc.). This allows for analyzing differences in the environment for each period. For example, by analyzing the customer behavior for each time period (morning, noon, and night), it is possible to analyze differences in customer behavior for each time period.

[0124] In one example, the search device 5 provides a list of detection results of the events EV and prompts including analysis instructions to the large-scale generative model, thereby identifying the characteristics of the events EV occurring in the environment. The analysis may be performed using a large-scale generative model. The large-scale generative model may include, for example, a large-scale language model (LLM), a large-scale visual language model (VLM), a large-scale speech model, etc. The search device 5 may output the analysis results obtained from the large-scale generative model as appropriate.

[0125] (4) Data linkage In addition, in one example, the accumulated data may be utilized in conjunction with other data. For example, when the imaging device 2 is operated in a store, the accumulated data may be utilized in conjunction with the purchase data of the target store. Accordingly, the search device 5 may refer to either the accumulated data or the purchase data, and vice versa. The purchase data may be stored in a known POS (Point Of Sale) The purchasing data may be collected by a known method and stored in any storage area, so that the purchasing data may be referenced as needed.

[0126] As a specific example, the search device 5 may identify a time period during which sales of a target product are good by analyzing purchase data. The degree of sales may be evaluated appropriately based on predetermined criteria. For example, the search device 5 may evaluate the target product as selling well if the total sales volume of the target product during a target period exceeds a first threshold or is equal to or greater than the first threshold. The search device 5 may evaluate the target product as selling poorly if the total sales volume of the target product during a target period is equal to or less than a second threshold. The second threshold may be the same as the first threshold or may be smaller than the first threshold. The first threshold and the second threshold may each be set arbitrarily. The search device 5 may extract images 20 depicting the target product during the specified time period by searching a database using information related to the specified time period and the target product as a query 60. The search device 5 may output the extracted images 20. Based on the output images 20, the status of the time period during which sales of the target product are good can be analyzed.

[0127] The search device 5 may analyze the purchase data to identify days on which sales of the target product are good and days on which sales are bad. The search device 5 may search a database using information about the identified day and the target product as a query 60, thereby extracting images 20 that depict the target product on the identified day. The search device 5 may output the extracted images 20. By comparing the images 20 from days on which sales are good and the images 20 from days on which sales are bad, the reason for fluctuations in sales of the target product can be analyzed.

[0128] The search device 5 may analyze the purchase data to identify days or time periods when sales of the target product are poor. The search device 5 may search a database using information about the identified day or time period and the target product as a query 60, and extract at least one of an image 20 depicting the target product on the identified day or time period and a result of image analysis 25. The search device 5 may output at least one of the extracted image 20 and the result of image analysis 25. For example, based on the fact that the remaining quantity of the target product depicted in the image 20 on a day when sales are poor is low, the remaining quantity of the target product analyzed as product event EV2 is low, or the target product is detected as being out of stock as product event EV2, it can be analyzed that the reason for the decline in sales of the target product is due to inappropriate replenishment of the target product.

[0129] The search device 5 may detect characteristic periods within the store by analyzing at least one of the stored images 20 and the image analysis results 25. For example, the search device 5 may identify a time (image capture time TT) when there are many customers from at least one of the stored images 20 and the image analysis results 25 (customer event EV1, etc.). The number of customers may be evaluated appropriately based on a predetermined criterion. For example, the search device 5 may evaluate that there are many customers when the number of customers captured in the images 20 or the number of customers detected as the image analysis results 25 exceeds or is equal to or greater than a threshold. The threshold may be set arbitrarily. The search device 5 may extract purchase data for a specific period (time period, date, day of the week, etc.) including the identified time from the stored purchase data. The search device 5 may then identify a specific time period with a large number of customers based on the extracted purchase data. The characteristics of purchasing activity during a certain period (for example, whether a particular product is selling well) may be analyzed.

[0130] The search device 5 may identify the time (image capture time TT) when the remaining quantity of the target product is low or when the target product is out of stock by analyzing at least one of the accumulated images 20 and the image analysis results 25. The remaining quantity of the target product may be evaluated appropriately based on predetermined criteria. For example, the search device 5 may evaluate that the remaining quantity of the target product is low if the remaining quantity of the target product is equal to or less than a threshold value. The threshold value may be set arbitrarily. The search device 5 may extract purchase data for a specific period including the identified time from the accumulated purchase data. The search device 5 may verify, based on the extracted purchase data, any adverse effects caused by the target product not being replenished. For example, if it is verified from the purchase data that the target product is selling well during the specific period, the search device 5 may evaluate that the remaining quantity of the target product is low or the target product is out of stock simply because the target product is selling well (i.e., no adverse effects have occurred), and output the evaluation result. On the other hand, if it is verified from the purchase data that sales of the target product are poor during a specific period, the search device 5 may evaluate that an adverse effect is occurring due to the target product not being replenished, and output the evaluation result. In one example, if it is evaluated that an adverse effect is occurring, the search device 5 may generate a stocking schedule for the target product so as to make it less likely to cause an adverse effect, for example, by increasing the amount of stocking or the number of stockings, and output the generated schedule.

[0131] (5) Supplementary information The destination of the various pieces of information (1) to (4) output by the search device 5 may be selected appropriately depending on the embodiment. The output destination may be, for example, the memory resources of the search device 5, an output device of the search device 5 (output device 55 described later), another computer, an external storage device, etc. Furthermore, the entity that executes the information processing related to the provision of each piece of information does not have to be limited to the search device 5. That is, at least a part of the information processing by the search device 5 may be executed on a computer other than the search device 5. The other computer may be, for example, the information processing device 1, another information processing device 45, or terminal device T1.

[0132] §2 Configuration example [Hardware configuration] (Information processing device) 10 is a diagram illustrating an example of a hardware configuration of the information processing device 1 according to this embodiment. In one example, the information processing device 1 may be configured as a computer to which a control unit 11, a storage unit 12, an external interface 13, an input device 14, and an output device 15 are electrically connected.

[0133] The control unit 11 is configured to execute information processing based on programs and various data. For example, the control unit 11 includes a hardware processor such as a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory). The control unit 11 (CPU) is an example of a processor resource of the information processing device 1.

[0134] The storage unit 12 is configured to hold any data. For example, the storage unit 12 may include a hard disk drive, a solid state drive, a semiconductor memory, etc. The storage unit 12, RAM, and ROM are examples of memory resources of the information processing device 1. In one example, the storage unit 12 may store various types of information such as a program 81.

[0135] The program 81 is a program for causing the information processing device 1 to execute information processing (FIG. 14 described later) related to the correction of the movement trajectory 300. The program 81 includes a series of instructions for the information processing. In one example, the program 81 may further include instructions for information processing other than information processing related to the correction of the movement trajectory 300, such as information processing related to image analysis (FIG. 15 described later). In one example, at least a part of the storage device MR may be configured by the storage unit 12. In this case, at least a part of the accumulated data (image 20, measurement position 32, imaging time TT, and image analysis result 25) may be stored in the storage unit 12.

[0136] In one example, the program 81 may be stored in a storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to store various information (such as the stored program) by electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The storage unit 12 and the storage medium 91 are examples of non-transitory storage media. The information processing device 1 may acquire the program 81 from the storage medium 91. The storage medium 91 may be a disk-type storage medium (such as a CD or DVD) or a non-disk type storage medium such as a semiconductor memory (such as a flash memory). Any drive device may be used to read the information stored in the storage medium 91. The type of drive device may be selected depending on the storage medium 91. The drive device may be connected to the information processing device 1 in any manner. The storage medium 91 may include an external storage device. At least a portion of the accumulated data (the image 20, the measurement position 32, the imaging time TT, and the image analysis result 25) may be stored in the storage medium 91.

[0137] The external interface 13 is configured to connect to an external device via a wired or wireless connection. The external interface 13 may include, for example, a USB (Universal Serial Bus) port, a dedicated port, a communication port (communication module), etc. The type and number of external interfaces 13 may be determined appropriately depending on the embodiment. When the external interface 13 includes a communication port (communication module), the standard of the communication network may be selected arbitrarily. The communication standard may be selected appropriately from, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, etc. In one example, the information processing device 1 may be connected to the imaging device 2 and the inertial sensor 3 via the external interface 13. In one example, when at least a portion of the storage device MR is configured as a storage area other than the memory resources of the information processing device 1, the information processing device 1 may be connected to the storage device MR via the external interface 13.

[0138] The input device 14 is configured to accept input of information. The input device 14 may be configured, for example, by an imaging device, a microphone, a mouse, a keyboard, a touch panel, an operator, etc. The imaging device of the input device 14 may be the same as or different from that of the imaging device 2. The output device 15 is configured to output information. The output device 15 may be configured, for example, by a display, a speaker, etc. The information processing device 1 may be operated using the input device 14 and the output device 15. The input device 14 and the output device 15 may be directly connected to the information processing device 1 or indirectly connected via the external interface 13. The input device 14 and the output device 15 may be at least partially integrated with a touch panel display, etc.

[0139] It should be noted that, with regard to the specific hardware configuration of the information processing device 1, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processors may be a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), a GP The external interface 13, the input device 14, and the output device 15 may be configured by a graphics processing unit (U), an application specific integrated circuit (ASIC), etc. At least one of the above may be omitted. The program 81 may be stored in an external storage device such as a NAS. An external storage device is also an example of a non-transitory storage medium. The information processing device 1 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. The information processing device 1 may be a computer designed specifically for the service provided, as well as a general-purpose server device, a general-purpose PC, a notebook PC, a terminal device, etc. In one example, the information processing device 1 includes an imaging device 2 and an inertial By including the sensor 3, it may be configured as a terminal device UT.

[0140] (Search device) 11 is a diagram illustrating an example of a hardware configuration of the search device 5 according to this embodiment. The search device 5 according to this embodiment may be configured as a computer in which a control unit 51, a storage unit 52, an external interface 53, an input device 54, and an output device 55 are electrically connected.

[0141] The control unit 51 to the output device 55 and the storage medium 95 of the search device 5 may be configured similarly to the control unit 11 to the output device 15 and the storage medium 91 of the information processing device 1, respectively. The control unit 51 (CPU) is an example of a processor resource of the search device 5. The storage unit 52 (and RAM, ROM) is an example of a memory resource of the search device 5. In this embodiment, the storage unit 52 stores various information such as a program 85.

[0142] The program 85 is a program for causing the search device 5 to execute information processing related to database search (see FIG. 16 described below). The program 85 includes a series of instructions for the information processing. In one example, the program 85 may further include instructions for information processing other than information processing related to database search, such as information processing related to image analysis (see FIG. 15 described below). In one example, at least a portion of the storage device MR may be configured by the storage unit 52. In this case, at least a portion of the accumulated data (image 20, measurement position 32, imaging time TT, and image analysis result 25) may be stored in the storage unit 52.

[0143] In one example, the program 85 may be stored in a storage medium 95 instead of or together with the storage unit 52. The search device 5 may acquire the program 85 from the storage medium 95. The program 85 may be saved in an external storage device. At least a portion of the accumulated data (images 20, measurement positions 32, imaging times TT, and image analysis results 25) may be stored in the storage medium 95.

[0144] The retrieval device 5 may perform data communication with other computers (such as terminal device T1) via an external interface 53 (communication module). In one example, when at least a part of the storage device MR is configured with a storage area other than the memory resources of the retrieval device 5, the retrieval device 5 may be connected to the storage device MR via the external interface 53. The retrieval device 5 may be operated using an input device 54 and an output device 55.

[0145] Note that, with regard to the specific hardware configuration of the search device 5, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 51 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA, DSP, GPU, ASIC, etc. At least one of the external interface 53, the input device 54, and the output device 55 may be omitted. The search device 5 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. The search device 5 may be a computer designed specifically for the service provided, as well as a general-purpose server device, a general-purpose PC, a laptop PC, a terminal device, etc.

[0146] [Software configuration] (Information processing device) 12 is a diagram illustrating an example of the software configuration of the information processing device 1 according to this embodiment. The control unit 11 of the information processing device 1 executes instructions included in the program 81 stored in the storage unit 12 using a CPU. As a result, the information processing device 1 operates as a computer including an acquisition unit 111, a correction processing unit 112, and an output processing unit 113 as software modules. That is, in one example, each software module of the information processing device 1 is a control unit 11 (CPU).

[0147] The acquisition unit 111 is configured to acquire a movement trajectory 300 of the imaging device 2 during an operation period of the imaging device 2, which period includes an image capture period for a group of images 200 captured by the imaging device 2, measured from measurement data 39 of an inertial sensor 3 that is separately attached to the moving body MB and is arranged to observe the movement of the imaging device 2. The correction processing unit 112 is configured to correct the acquired movement trajectory 300 so as to satisfy a constraint 70 of a movement condition given in advance to the moving body MB. The output processing unit 113 is configured to output a corrected movement trajectory 325.

[0148] In one example, when the information processing device 1 is configured to perform image analysis on the image 20, the software configuration of the information processing device 1 may further include an image analysis unit 114. The image analysis unit 114 may be configured to perform image analysis on the image 20 and store the obtained image analysis result 25 in association with the image 20.

[0149] (Search device) 13 schematically shows an example of the software configuration of the search device 5 according to this embodiment. The control unit 51 of the search device 5 executes instructions included in the program 85 stored in the storage unit 52 using the CPU. This causes the search device 5 to operate as a computer having a reception unit 511, a search unit 512, and a response unit 513 as software modules. That is, in one example, similar to the information processing device 1, each software module of the search device 5 may also be realized by the control unit 51 (CPU).

[0150] The receiving unit 511 is configured to receive a query 60 from the terminal device T1. The searching unit 512 is configured to search a database (storage device MR) to extract elements that match the query 60 from the database. The replying unit 513 is configured to reply to the terminal device T1 with the obtained search result 65.

[0151] In one example, when the search device 5 performs image analysis on the image 20, the software configuration of the search device 5 may further include an image analysis unit 514. The image analysis unit 514 may be configured similarly to the image analysis unit 114.

[0152] (others) In the example of the present embodiment, each software module of the information processing device 1 and the search device 5 is implemented by a general-purpose CPU. However, the method of implementing each of the above modules is not limited to this example and may be changed as appropriate depending on the embodiment. Some or all of the above software modules may be implemented by one or more dedicated processors or chipsets. Each of the above modules may be implemented as a hardware module. With regard to the software configurations of the information processing device 1 and the search device 5, modules may be omitted, replaced, or added as appropriate depending on the embodiment. For example, in the software configuration of the information processing device 1, the image analysis unit 114 may be omitted. In the software configuration of the search device 5, the image analysis unit 514 may be omitted.

[0153] §3 Example of operation [Movement trajectory correction] FIG. 14 is a flowchart showing an example of a processing procedure for correcting the movement trajectory 300 by the information processing device 1 according to this embodiment. The following processing procedure is an example of an information processing method (correction method) executed by a computer. The following processing procedure is merely an example, and each step may be changed as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0154] (Step S101) In step S101, the control unit 11 operates as the acquisition unit 111 to acquire the movement trajectory 300 of the imaging device 2 during the operation period of the imaging device 2.

[0155] In one example, the control unit 11 may acquire measurement data 39 directly from the inertial sensor 3 or indirectly via another computer or the like, and derive each measurement position 30 from the acquired measurement data 39, thereby acquiring the movement trajectory 300. In another example, the movement trajectory 300 may be formed by another computer, and the control unit 11 may acquire the movement trajectory 300 directly or indirectly from the other computer. Upon acquiring the movement trajectory 300, the control unit 11 proceeds to the next step S102.

[0156] (Step S102) In step S102, the control unit 11 operates as the correction processing unit 112 to correct the acquired movement trajectory 300 so as to satisfy the constraints 70 of the movement conditions given in advance to the moving object MB. As a result of executing this correction processing, the control unit 11 can obtain a corrected movement trajectory 325.

[0157] In one example, the movement condition constraint 70 may include that the moving object MB is located at the same location at multiple times during the operation period of the imaging device 2. The control unit 11 may correct the movement trajectories (300, 321, 322) so that the measurement positions (measurement position 30, measurement position 32) at multiple times on the movement trajectories (300, 321, 322) coincide with each other. This allows a corrected movement trajectory 320 to be obtained. In one example, the moving object MB may be a robotic device MB111 that moves autonomously within a store. The same location may be a charging point CP for the robotic device MB111.

[0158] In one example, the movement condition constraint 70 may include limiting the movement of the moving object MB within a predetermined movement range. The control unit 11 may correct the movement trajectory (300, 320) so that it fits the shape of the movement range. This allows a corrected movement trajectory 321 to be obtained. In one example, the moving object MB may be a robotic device MB111 that moves autonomously within a store. The movement range may be defined within a floor FR of the store.

[0159] In one example, the movement condition constraint 70 may further include an internal constraint 700 within the movement range. The control unit 11 may correct the movement trajectory (320, 321) so as to satisfy the internal constraint 700. This allows a corrected movement trajectory 322 to be obtained. In one example, the internal constraint 700 may include a requirement that the moving object MB not pass through an obstacle range defined by the presence of an obstacle. The control unit 11 may correct the movement trajectory (320, 321) so as to bypass the obstacle range. In one example, any of the corrected movement trajectories (320, 321, 322) may be acquired as a corrected movement trajectory 325. When the correction process is completed, the control unit 11 proceeds to the next step S103.

[0160] (Step S103) In step S103, the control unit 11 operates as the output processing unit 113 to output the corrected movement trajectory 325.

[0161] The output destination of the movement trajectory 325 may be selected appropriately depending on the embodiment. In one example, the control unit 11 may output the corrected movement trajectory 325 to at least one of an arbitrary output device and a storage area. For example, the control unit 11 may output the corrected movement trajectory 325 to at least one of the output device 15, RAM, the storage unit 12, the storage medium 91, another computer, and an external storage device. In one example, the control unit 11 may output the corrected movement trajectory 325 before or after the processing of step S103. During the processing of step S103, the control unit 11 may acquire the image group 200 captured by the imaging device 2. As an output process of the corrected movement trajectory 325, the control unit 11 may store the image group 200 and the corrected movement trajectory 325 in association with each other. In another example, the control unit 11 may output the corrected movement trajectory 325 to another computer that collects the image group 200. In this way, the control unit 11 may cause the other computer to associate the image group 200 with the corrected movement trajectory 325.

[0162] In one example, when a storage device MR is provided, the image group 200 (images 20) and the corrected movement trajectory 325 (measurement positions 32) associated with each other may be stored in the storage device MR as appropriate. For example, the image group 200 and the corrected movement trajectory 325 may be stored simultaneously in the storage device MR. After the image group 200 is stored in the storage device MR, the stored movement trajectory 325 may be stored later in the storage device MR. Also, after the movement trajectory 300 before correction is temporarily registered, the corrected movement trajectory 325 may be stored in place of the movement trajectory 300 before correction. When the output of the corrected movement trajectory 325 is completed, the control unit 11 ends the processing procedure related to the correction of the movement trajectory 300 according to this operation example.

[0163] (Features) In this embodiment, an inertial sensor 3 is used to measure the position of the imaging device 2 (moving body MB). The inertial sensor 3 can measure the position not only outdoors but also indoors. However, position measurement using the inertial sensor 3 is prone to errors. Therefore, in this embodiment, the processing in step S102 corrects the movement trajectory 300 so as to satisfy the movement condition constraint 70, thereby attempting to suppress errors in the measurement position (measurement position 30). As a result, it is expected that the processing in step S103 will provide a measurement position (measurement position 32) with reduced errors. Therefore, this embodiment can support the appropriate collection of an image (image 20) associated with the measurement position (measurement position 32) not only outdoors but also indoors.

[0164] [Image analysis] Fig. 15 is a flowchart showing an example of a processing procedure for image analysis according to this embodiment. The processing procedure in Fig. 15 is merely an example, and each step may be changed as much as possible. Furthermore, steps in the processing procedure in Fig. 15 may be omitted, replaced, or added as appropriate depending on the embodiment. Execution of the processing procedure in Fig. 15 may be started at any timing after image 20 is obtained.

[0165] (Step S301) In step S301, the control unit 11 operates as the image analysis unit 114 to perform image analysis on the image 20. As a result, the control unit 11 can obtain the result 25 of the image analysis.

[0166] In one example, the control unit 11 may perform image analysis to detect an object OB appearing in the image 20. In one example, the moving object MB may be an entity that moves within a store, and the object OB may be a product OB1 sold in the store.

[0167] In one example, the control unit 11 may perform image analysis to detect events EV captured in the image 20, together with or instead of detecting the object OB. In one example, the moving object MB may be an entity that moves within the store, and the events EV may include at least one of a customer event EV1 related to a customer in the store and a product event EV2 related to a product sold in the store.

[0168] The image 20 may be acquired as appropriate before executing the image analysis. The control unit 11 may acquire the image 20 directly from the imaging device 2 or indirectly via another computer or the like. When the image analysis is completed, the control unit 11 advances the process to the next step S302.

[0169] (Step S302) In step S302, the control unit 11 operates as the image analysis unit 114 to associate the acquired image analysis result 25 with the image 20 and store it.

[0170] The destination for saving the image 20 and the image analysis result 25 may be selected appropriately depending on the embodiment. In one example, when a storage device MR is provided, the control unit 11 may save the image 20 and the image analysis result 25 in the storage device MR. In the storage device MR, the image 20 may be associated with the measurement position 32 and the image analysis result 25. In one example, the image analysis result 25 may be saved in the storage device MR simultaneously with the image 20. In another example, after the image 20 is saved in the storage device MR, the image analysis result 25 may be saved in the storage device MR later. When saving of the image analysis result 25 is completed, the control unit 11 ends the processing procedure related to the image analysis according to this operation example.

[0171] Note that the entity that executes the processes of steps S301 and S302 does not have to be limited to the control unit 11 (information processing device 1). In another example, the control unit 51 of the search device 5 may operate as the image analysis unit 514 to execute the processes of steps S301 and S302. In yet another example, another information processing device 45 may execute the processes of steps S301 and S302.

[0172] [Search process] Fig. 16 is a flowchart showing an example of a processing procedure for searching a database by the search device 5 according to this embodiment. The processing procedure in Fig. 16 is merely an example, and each step may be changed as much as possible. Furthermore, steps in the processing procedure in Fig. 16 may be omitted, replaced, or added as appropriate depending on the embodiment.

[0173] (Step S501) In step S501, the control unit 51 operates as the reception unit 511 and receives the query 60 from the terminal device T1.

[0174] In one example, the storage device MR may store the image 20, the measurement position 32, and the imaging time TT in association with each other. The control unit 51 may accept a query 60 including at least one of a search target image, a target position, and a target time. In one example, the storage device MR may store the image 20, the measurement position 32, the image analysis result 25, and the imaging time TT in association with each other. The control unit 51 may accept the query 60 including at least one of a search target image, a target position, target information, and a target time. Upon accepting the query 60, the control unit 51 proceeds to the next step S502.

[0175] (Step S502) In step S502, control unit 51 operates as search unit 512 to search the database (storage device MR). Through this search, control unit 51 extracts elements that match query 60 from the database. This allows control unit 51 to obtain search result 65 made up of the extracted elements. Upon obtaining search result 65, control unit 51 proceeds to the next step S503.

[0176] (Step S503) In step S503, the control unit 51 operates as the response unit 513 to respond (reply) to the terminal device T1 with the obtained search result 65. After returning the search result 65 to the terminal device T1, the control unit 51 ends the processing procedure related to the search according to this operation example.

[0177] §4 Variations Although the embodiments of the present disclosure have been described in detail above, the above description is merely an example of the present disclosure in every respect. The processes and means described in the present disclosure can be freely combined and implemented as long as no technical contradiction occurs. Furthermore, various improvements or modifications may be made to the above embodiments as appropriate.

[0178] §5 Experimental Examples The following experiment was conducted to verify the effect of the correction process on the movement trajectory obtained by the inertial sensor, although the present disclosure is not limited to the following experimental example.

[0179] First, a rectangular frame-shaped passage was generated on a computer. Points were set at the four corners of the generated passage: top left, top right, top right, and bottom right. The ground truth of the movement trajectory was set as a circular route that starts from the top left point, passes through the top right, top right, and bottom right points in that order, and returns to the top left point. Additionally, assuming self-position estimation using an inertial sensor, a position slightly away from each of the four corner points was set as the destination value, and an arc-like movement trajectory was derived for the portion from the top left point to the top right point, and a linear movement trajectory was derived for the remaining portion. The movement trajectory obtained by connecting these was then rotated and scaled by appropriate values ​​to generate the initial inaccurate movement trajectory. The start and end points of the movement were the same. Under the condition that the initial value of the movement trajectory is corrected by the first method, a provisionally corrected movement trajectory (LC Output) was obtained. Next, a rectangular frame-shaped passage was set in the movement range. Then, the correction of the 2-1 method is applied to the movement trajectory (LC Output) to perform further correction. The movement trajectory (MM Output) was obtained. Then, the inside and outside of the passage were set as obstacle ranges. By applying the correction of the 2-2 method to the movement trajectory (MM Output), the final movement The trajectory (PF Output) was obtained.

[0180] FIG. 17 shows the initial inaccurate trajectory in the second experimental example. FIG. 18 shows the LC Output and the Ground Truth of the movement trajectory after correction by the first method in the second experimental example. FIG. 19 shows the LC Output and the Ground Truth of the movement trajectory after correction by the first method in the second experimental example. 20 shows the movement trajectory (MM Output) and the true value of the movement trajectory (Ground Truth) after correction by the 2-1 method in the second experimental example. FIG. 21 shows the movement trajectory (PF Output) and the true value of the movement trajectory (Ground Truth) after correction by the 2-2 method in the second experimental example.

[0181] Between the initial inaccurate trajectory and the ground truth The average error was 8.71 m, and the maximum error was 12.40 m. In contrast, the average error between the corrected movement trajectory (LC Output) and the true value (Ground Truth) for the first method was 7. The average error was 57m, with a maximum error of 11.04m. This result shows that the correction by Method 1 can improve the accuracy of the movement trajectory. In addition, the average error between the movement trajectory (MM Output) and the true value (Ground Truth) after correction by Method 2-1 was 0.63m. The maximum error was 2.48 m. From this result, it was found that the accuracy of the movement trajectory can be improved even by the correction of the 2-1 method. Furthermore, the average error between the movement trajectory (PF Output) and the true value (Ground Truth) after the correction of the 2-2 method was 1.43 m. The maximum error was 1.83 m. In other words, although the average error worsened with the correction using Method 2-2, the maximum error was reduced. This result shows that the correction using Method 2-2 can also be expected to improve the accuracy of the movement trajectory. The above results show that by correcting the movement trajectory obtained by the inertial sensor so that it satisfies the constraints of the movement conditions, it is possible to ensure the accuracy of the movement conditions. [Explanation of symbols]

[0182] 1...information processing device, 2...imaging device, 3...inertial sensor, 11...control unit, 12...storage unit, 200...image group, 39...measurement data, 300...movement trajectory, 325...corrected movement trajectory, 70…constraint, MB…mobile object

Claims

1. a movement trajectory of an imaging device attached separately to a moving body, the movement trajectory being measured from measurement data of an inertial sensor arranged to observe the movement of the imaging device, the movement trajectory of the imaging device during an operation period of the imaging device including an image capture period of a group of images captured by the imaging device; Correcting the acquired movement trajectory so as to satisfy the constraints of the movement conditions given in advance to the moving body; and outputting the corrected movement trajectory; A control unit configured as follows: the movement condition constraint includes the moving object being located at the same location at multiple times during the operation period; correcting the movement trajectory includes correcting the movement trajectory so that measured positions at the respective times on the movement trajectory coincide with each other; the mobile object is a robot device that moves autonomously within the store, the same location is a charging point for the robotic device; Information processing device.

2. A movement trajectory measured from measurement data of an inertial sensor arranged to observe the movement of an imaging device separately attached to a moving body, the movement trajectory of the imaging device during an operation period of the imaging device including an image capture period of a group of images captured by the imaging device, Correcting the acquired movement trajectory so as to satisfy the constraints of the movement conditions given in advance to the moving body; and outputting the corrected movement trajectory; A control unit configured as follows: the restriction on the movement condition includes limiting the movement of the moving object to a predetermined movement range; correcting the movement trajectory includes correcting the movement trajectory to fit the shape of the movement range; Information processing device.

3. the mobile object is a robot device that moves autonomously within the store, The movement range is defined within the floor of the store. The information processing device according to claim 2 .

4. The constraints of the movement conditions further include an internal constraint within the movement range; correcting the movement trajectory further includes correcting the movement trajectory to satisfy the internal constraint.

4. The information processing device according to claim 2 or 3.

5. the internal constraint includes that the moving body does not pass through an obstacle range defined by the presence of an obstacle; correcting the movement trajectory to satisfy the internal constraint includes correcting the movement trajectory to bypass the obstacle area. The information processing device according to claim 4 .

6. 1. A computer-implemented information processing method, comprising: a movement trajectory of an imaging device attached separately to a moving body, the movement trajectory being measured from measurement data of an inertial sensor arranged to observe the movement of the imaging device, the movement trajectory of the imaging device during an operation period of the imaging device including an image capture period of a group of images captured by the imaging device; Correcting the acquired movement trajectory so as to satisfy the constraints of the movement conditions given in advance to the moving body; and outputting the corrected movement trajectory; This includes: the movement condition constraint includes the moving object being located at the same location at multiple times during the operation period; correcting the movement trajectory includes correcting the movement trajectory so that measured positions at the respective times on the movement trajectory coincide with each other; the mobile object is a robot device that moves autonomously within the store, the same location is a charging point for the robotic device; Information processing methods.

7. An information processing method executed by a computer, comprising: a movement trajectory of an imaging device attached separately to a moving body, the movement trajectory being measured from measurement data of an inertial sensor arranged to observe the movement of the imaging device, the movement trajectory of the imaging device during an operation period of the imaging device including an image capture period of a group of images captured by the imaging device; Correcting the acquired movement trajectory so as to satisfy the constraints of the movement conditions given in advance to the moving body; and outputting the corrected movement trajectory; This includes: the restriction on the movement condition includes limiting the movement of the moving object to a predetermined movement range; correcting the movement trajectory includes correcting the movement trajectory to fit the shape of the movement range; Information processing methods.

8. The mobile object is a robotic device that moves autonomously within the store, The movement range is defined within the floor of the store. The information processing method according to claim 7.

9. The constraints of the movement conditions further include internal constraints within the movement range, Correcting the trajectory includes correcting the trajectory so as to satisfy the internal constraint. further comprising:

9. The information processing method according to claim 7 or 8.

10. The internal constraint includes that the moving body does not pass through an obstacle range defined by the presence of an obstacle, correcting the movement trajectory to satisfy the internal constraint includes correcting the movement trajectory to bypass the obstacle area. The information processing method according to claim 9.

11. A program for causing a computer to execute an information processing method, The information processing method includes: a movement trajectory of an imaging device attached separately to a moving body, the movement trajectory being measured from measurement data of an inertial sensor arranged to observe the movement of the imaging device, the movement trajectory of the imaging device during an operation period of the imaging device including an image capture period of a group of images captured by the imaging device; Correcting the acquired movement trajectory so as to satisfy the constraints of the movement conditions given in advance to the moving body; and outputting the corrected movement trajectory; This includes: the movement condition constraint includes the moving object being located at the same location at multiple times during the operation period; correcting the movement trajectory includes correcting the movement trajectory so that measured positions at the respective times on the movement trajectory coincide with each other; the mobile object is a robot device that moves autonomously within the store, the same location is a charging point for the robotic device; program.

12. A program for causing a computer to execute an information processing method, comprising: The information processing method includes: a movement trajectory of an imaging device attached separately to a moving body, the movement trajectory being measured from measurement data of an inertial sensor arranged to observe the movement of the imaging device, the movement trajectory of the imaging device during an operation period of the imaging device including an image capture period of a group of images captured by the imaging device; Correcting the acquired movement trajectory so as to satisfy the constraints of the movement conditions given in advance to the moving body; and outputting the corrected movement trajectory; This includes: the restriction on the movement condition includes limiting the movement of the moving object to a predetermined movement range; correcting the movement trajectory includes correcting the movement trajectory to fit the shape of the movement range; program.

13. The mobile object is a robotic device that moves autonomously within the store, The movement range is defined within the floor of the store. The program according to claim 12.

14. The constraints of the movement conditions further include internal constraints within the movement range, correcting the movement trajectory further includes correcting the movement trajectory to satisfy the internal constraint. The program according to claim 12 or 13.

15. The internal constraint includes that the moving body does not pass through an obstacle range defined by the presence of an obstacle, correcting the movement trajectory to satisfy the internal constraint includes correcting the movement trajectory to bypass the obstacle area. The program according to claim 14.

Images