Imaging system, information processing method and program

Abstract

To provide a technique capable of efficiently acquiring images within the movement range of a moving body. [Solution] According to one aspect of the present disclosure, an imaging system includes an imaging device separately attached to a moving object, a sensor for measuring the attitude of the imaging device, and a control device. The control device stores map information that records the imaged range. The control device acquires the measurement results of the attitude of the imaging device by the sensor, and determines, based on the map information, whether the target range toward which the imaging device is facing, identified from the acquired attitude measurement results, has been imaged. If it is determined that the target range has not been imaged, the control device executes an imaging process using the imaging device, and if it is determined that the target range has been imaged, the control device omits execution of the imaging process using the imaging device.

Description

[Technical Field]

[0001] The present disclosure relates to an imaging system, an information processing method, and a program. [Background technology]

[0002] Various methods have been proposed for efficiently acquiring images of new fields of view using an imaging device mounted on a mobile object. For example, Non-Patent Document 1 proposes a method for collecting images using a mobile robot equipped with a camera for 3D reconstruction or map construction of a scene. Specifically, the proposed method calculates the gain in the amount of information in the field of view at each candidate imaging point, determines the next imaging point from among the candidates so that the calculated gain is maximized, and moves the mobile robot to the determined imaging point. Furthermore, Non-Patent Document 2, for example, proposes a method for moving a robot to create a map of an unknown location in an environment. Specifically, the proposed method models uncertainty about the map and uses the resulting model to have the robot search in a direction that minimizes uncertainty. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Stefan Isler, et al. “An Information Gain Formulation for Active Volumetric 3D Reconstruction”, [online], [Retrieved June 16, 2025], Internet<URL:https: / / rpg.ifi.uzh.ch / docs / ICRA16_Isler.pdf> [Non-patent document 2] Ayoung Kim, et al. “Active Visual SLAM for Robotic Area Coverage: Theory and Experiment”, [online], [Retrieved June 16, 2020], Internet<URL:https: / / robots.engin.umich.edu / publications / akim-2015a.pdf> Summary of the Invention [Problem to be solved by the invention]

[0004] According to conventional methods, images from new viewpoints can be efficiently collected by moving a moving object according to a predetermined standard. However, the present inventors have discovered the following problem with conventional methods. Specifically, if the movement of a moving object can be controlled, images can be efficiently acquired by moving the moving object to an unknown viewpoint. On the other hand, when an imaging device is separately attached to a moving object, the moving object is not necessarily controllable. Examples of cases in which an imaging device is separately attached to a moving object include retrofitting an imaging device to a cleaning robot or attaching an imaging device to a working person. When the movement of a moving object is difficult to control, efficient image collection by moving the moving object to an unknown viewpoint, as in conventional methods, is difficult. If the imaging device is operated constantly and continuously captured while the moving object is moving in order to prevent missing images from unknown viewpoints, the imaging device consumes a lot of power.

[0005] In one aspect, the present disclosure has been made in view of the above circumstances, and one of the objects of the present disclosure is to provide a technology that can efficiently acquire images within the movement range of a moving object. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the present disclosure employs the following configurations. Note that the following configurations can be combined as appropriate.

[0007] An imaging system according to one aspect of the present disclosure includes an imaging device separately attached to a moving object, a sensor for measuring the attitude of the imaging device, and a control device. The control device includes a storage unit for storing map information that records the imaged range, and a control unit. The control unit is configured to The image capturing device is configured to acquire a measurement result of the orientation of the image capturing device, determine whether or not an image of a target range toward which the image capturing device is facing, which is identified from the acquired orientation measurement result, has been captured based on map information stored in a storage unit, and, if it is determined that the target range has not been captured, execute an image capturing process by the image capturing device, and, if it is determined that the target range has been captured, omit execution of the image capturing process by the image capturing device. The image capturing process includes starting the image capturing device, controlling the image capturing device to capture an image of the target range, stopping the start of the image capturing device after capturing the image of the target range, and updating the map information to indicate that the target range has been captured.

[0008] In this configuration, if the target range identified by the sensor has not yet been imaged, the image capture process is executed, whereas if the target range has already been imaged, the image capture process is omitted. This makes it possible to prevent overlapping image capture processes from being executed in the imaged range. Therefore, with this configuration, the number of image capture processes can be reduced compared to when the image capture device is constantly capturing images, and efficient image acquisition can be expected.

[0009] In the imaging system according to the above aspect, the moving object may be an autonomously moving robotic device. With this configuration, efficient image acquisition can be expected in situations where the autonomously moving robotic device is operated.

[0010] In the imaging system according to the above aspect, the moving object may be a device that is moved by direct or remote manual operation. With this configuration, efficient image acquisition can be expected in situations where a manually operated device is used.

[0011] In the imaging system according to the above aspect, the moving object may be a living thing. With this configuration, when the imaging device is attached to the living thing to collect images, efficient image acquisition can be expected.

[0012] In the imaging system according to the above aspect, the living thing may be a human. With this configuration, efficient image acquisition can be expected in situations where the imaging device is attached to a human and images are collected.

[0013] In the imaging system according to the above aspect, the sensor may be an inertial sensor. The inertial sensor can observe the posture not only outdoors but also indoors. Therefore, with this configuration, the imaging system can be operated not only outdoors but also indoors.

[0014] The imaging system according to the above aspect may be configured by a terminal device that integrally includes an imaging device, a sensor, and a control device. With this configuration, efficient image acquisition can be expected when using a terminal device such as a smartphone as an imaging system.

[0015] The imaging system according to the above aspect may further include a storage device that stores images captured by the imaging device in association with measurement results of the orientation of the imaging device at the time of capturing the images. According to this configuration, by storing the measurement results of the orientation of the imaging device at the time of capturing the images in association with the images in the storage device, it is possible to provide information based on a combination of the orientation measurement results and the images. For example, it is possible to provide a system that allows access from one of the images and the orientation measurement results to the other.

[0016] In the imaging system according to the above aspect, the posture measurement result associated with the image may be obtained by correcting the posture measurement result by the sensor so as to satisfy constraints of a movement condition given in advance to the moving object. With this configuration, it is expected that the accuracy of the posture measurement result associated with the image can be ensured.

[0017] In the imaging system according to the above aspect, the storage device may store the images in association with the results of image analysis of the images. According to this configuration, by storing the posture measurement results and the image analysis results in association with the images in the storage device, it is possible to provide information based on a combination of the posture measurement results, the image analysis results, and the images. For example, it is possible to provide a system for accessing other elements from at least one of the images, the posture measurement results, and the image analysis results.

[0018] In the imaging system according to the above aspect, the image analysis may include detecting an object appearing in the image. With this configuration, it is possible to provide information about the object.

[0019] In the imaging system according to the above aspect, the moving body may be an entity that moves within a store, and the object may include a product sold within the store. With this configuration, it is possible to provide information about the product.

[0020] In the imaging system according to the above aspect, the image analysis may include detection of an event captured in the image. With this configuration, it is possible to provide information about the event.

[0021] In the imaging system according to the above aspect, the moving object may be an entity that moves within a store, and the event may include a customer event related to a customer within the store. With this configuration, it is possible to provide information about the customer event.

[0022] In the imaging system according to the above aspect, the moving object may be an entity that moves within a store, and the event may include a product event related to a product sold within the store. With this configuration, it is possible to provide information about the product event.

[0023] The embodiments of the present disclosure are not limited to the imaging system described above. As another aspect of the imaging system according to each of the above aspects, one aspect of the present disclosure may be a control device (information processing device) for configuring the imaging system according to any of the above aspects. Furthermore, one aspect of the present disclosure may be an information processing method (control method) for realizing all or part of each of the above configurations, a program, or a machine-readable storage medium storing such a program, such as a computer. Here, the machine-readable storage medium may be a non-transitory medium that stores information, such as a program, by electrical, magnetic, optical, mechanical, or chemical action. The non-transitory storage medium may include a storage medium (e.g., a CD, a DVD, a semiconductor memory), an auxiliary storage device of a computer, an external storage device connected to a computer, etc.

[0024] For example, an information processing method according to an aspect of the present disclosure may be executed by a computer. The computer may be connected to an imaging device separately attached to a moving object and a sensor for measuring the attitude of the imaging device, and may store map information recording an imaged range. The information processing method may include acquiring a measurement result of the attitude of the imaging device by the sensor, determining, based on the stored map information, whether an image has been captured of a target range toward which the imaging device is facing, identified from the acquired attitude measurement result, if it is determined that the target range has not been captured, performing an imaging process by the imaging device, and omitting the execution of the imaging process by the imaging device if it is determined that the target range has been captured. The imaging process may include starting the imaging device, controlling the imaging device to capture the target range, stopping the activation of the imaging device after capturing the target range, and updating the map information to indicate that the target range has been captured.

[0025] Furthermore, for example, a program according to an aspect of the present disclosure may be a program for causing a computer to execute an information processing method. The information processing method may include: connecting an imaging device to a target object and a sensor for measuring the orientation of the imaging device; and storing map information recording an imaged range. The information processing method may include acquiring a measurement result of the orientation of the imaging device by the sensor; determining, based on the stored map information, whether an image has been captured of a target range toward which the imaging device is facing, identified from the acquired orientation measurement result; executing an imaging process by the imaging device if it is determined that the target range has not been captured; and omitting execution of the imaging process by the imaging device if it is determined that the target range has been captured. The imaging process may include starting the imaging device, controlling the imaging device to capture the target range, stopping the activation of the imaging device after capturing the target range, and updating the map information to indicate that the target range has been captured. [Effects of the Invention]

[0026] According to one aspect of the present disclosure, it is possible to provide a technology that can efficiently acquire images within the movement range of a moving object. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 schematically illustrates an example of a situation to which the present disclosure is applied. [Figure 2] FIG. 2 shows a schematic diagram of an example of a moving object. [Figure 3] FIG. 3 is a schematic diagram showing an example of another configuration of the imaging system. [Figure 4] FIG. 4 shows an example of a scene in which it is determined whether or not the target range has been imaged. [Figure 5] FIG. 5 is a schematic diagram showing an example of a scene in which a captured image is used. [Figure 6] FIG. 6 is a schematic diagram showing another example of a scene in which a captured image is used. [Figure 7] FIG. 7 shows an example of the provided data. [Figure 8] FIG. 8 is a diagram showing an example of a scene in which an image and an imaging attitude are associated with each other. [Figure 9] FIG. 9 shows a schematic diagram of an example of a method for correcting the posture measurement results. [Figure 10] FIG. 10 is a diagram showing an example of a method for correcting the posture measurement results. [Figure 11] FIG. 11 shows a schematic diagram of an example of a method for correcting the posture measurement results. [Figure 12] FIG. 12 is a diagram illustrating an example of a hardware configuration of the control device. [Figure 13] FIG. 13 is a diagram illustrating an example of a hardware configuration of an information processing device. [Figure 14] FIG. 14 is a diagram illustrating an example of the software configuration of the control device. [Figure 15] FIG. 15 is a diagram illustrating an example of the software configuration of the information processing device. [Figure 16] FIG. 16 is a flowchart illustrating an example of a processing procedure regarding control of the imaging device by the control device. [Figure 17] FIG. 17 is a flowchart showing an example of the imaging process. [Figure 18] FIG. 18 is a flowchart illustrating an example of a processing procedure for correcting the posture measurement result. [Figure 19] FIG. 19 is a flowchart illustrating an example of a processing procedure for assigning the result of image analysis. [Figure 20] FIG. 20 is a flowchart illustrating an example of a processing procedure of the information processing device. [Figure 21] FIG. 21 shows the results of the first experimental example. [Figure 22] FIG. 22 shows the initial values ​​of the movement trajectory in the second experimental example. [Figure 23] FIG. 23 shows the movement trajectory after correction by the first method in the second experimental example. [Figure 24] FIG. 24 shows the movement trajectory after correction by the 2-1 method in the second experimental example. [Figure 25] FIG. 25 shows the movement trajectory after correction by the 2-2 method in the second experimental example. DETAILED DESCRIPTION OF THE INVENTION

[0028] Hereinafter, an embodiment according to one aspect of the present disclosure will be described with reference to the drawings. However, the embodiment described below is merely an example of the present disclosure in all respects. Various improvements or modifications may be made without departing from the scope of the present disclosure. In implementing the present disclosure, a specific configuration according to the embodiment may be appropriately adopted. Note that, although the data appearing in this embodiment is described in natural language, more specifically, it is also possible to use a language that can be recognized by a machine such as a computer. It is specified by various pseudo-languages, commands, parameters, machine language, electrical signals, etc.

[0029] §1 Application Examples FIG. 1 schematically illustrates an example of a situation to which the present disclosure is applied. The imaging system SY according to this embodiment includes a control device 1, an imaging device 2, and a sensor 3. The imaging device 2 is separately attached to a moving body MB. The sensor 3 is a measurement device for measuring the attitude of the imaging device 2. The control device 1 holds map information 125 that records the imaged range. The control device 1 may be one or more computers configured to control the operation of the imaging device 2.

[0030] The control device 1 according to this embodiment acquires a measurement result 30 of the attitude of the imaging device 2 obtained by the sensor 3. The control device 1 determines, based on the stored map information 125, whether or not an image of a target range 35 toward which the imaging device 2 is facing, which is identified from the acquired attitude measurement result 30, has already been captured. If it is determined that an image of the target range 35 has not already been captured, the control device 1 executes an imaging process by the imaging device 2. On the other hand, if it is determined that an image of the target range 35 has already been captured, the control device 1 omits the execution of the imaging process by the imaging device 2.

[0031] The imaging process includes starting the imaging device 2, controlling the imaging device 2 to capture an image of the target range 35, stopping the imaging device 2 after capturing an image of the target range 35, and updating the map information 125 to indicate that the target range 35 has been captured. By performing this imaging process, an image 20 of the target range 35 or its vicinity can be obtained. The number of images 20 acquired in one imaging process is not particularly limited and may be determined appropriately depending on the embodiment. The number of images 20 acquired in one imaging process may be one, or two or more.

[0032] In this embodiment, the control device 1 controls the operation of the imaging device 2 to execute an imaging process if the target range 35 identified by the sensor 3 has not yet been imaged, but controls the operation of the imaging device 2 to omit the execution of the imaging process if the target range 35 has already been imaged. This makes it possible to prevent overlapping execution of imaging processes in an imaged range. Therefore, according to this embodiment, the number of times imaging processes are executed can be reduced compared to when the imaging device 2 is constantly capturing images, and therefore it is possible to expect efficient acquisition of the image 20.

[0033] [Moving object] 2 is a schematic diagram showing an example of a moving object MB according to this embodiment. The moving object MB may be any object that moves. The type of the moving object MB may be appropriately selected depending on the embodiment. The object may include at least one of a living thing and a device (machine).

[0034] In one example, the mobile body MB may be any device MB1 configured to be movable. The device MB1 may include at least one of a robotic device MB11 that moves autonomously and a device MB12 that moves by direct or remote manual operation. The robotic device MB11 may include, for example, an automatically driven vehicle, an air vehicle that can fly automatically, or other robotic devices configured to be autonomously movable. The air vehicle may include a drone. The robotic device MB11 may include a robotic device used in a store (a robotic device that moves autonomously within a store). The robotic devices used in a store may include, for example, a cleaning robot, a food delivery robot, a transport robot, a guide robot, a security robot, etc.

[0035] Direct manual operation may include, for example, operating the movement of device MB12 using an operation unit provided on device MB12, physically acting on device MB12, etc. Physical acting may include, for example, pushing, towing, etc. Remote manual operation may include, for example, remotely operating the movement of device MB12 using a controller or computer. Device MB12 may be, for example, a manually operated vehicle, a remotely controlled flying object, a manually pushed or driven object, etc. The device MB12 may include a cart that is manually operated, or other device configured to be movable by manual operation. The device MB12 may include a device used in a store (a device that is manually moved within a store). The device used in a store may include, for example, a shopping cart. The shopping cart may include a smart cart, a cash register cart, etc. The store may include, for example, a commercial facility such as a supermarket, a convenience store, a drugstore, or a restaurant. The store may include a commercial complex such as a shopping mall. According to one example of the present embodiment, efficient acquisition of images 20 can be expected in situations where the device MB1 (at least one of the autonomously moving robot device MB11 and the manually moved device MB12) is operated as the mobile body MB.

[0036] In addition, in one example, the moving object MB may be a living creature MB2. The type of living creature MB2 is not particularly limited and may be appropriately selected depending on the embodiment. The living creature MB2 may include at least one of a human being MB21 and another living creature MB22 other than the human being MB21. The other living creature MB22 may include, for example, a pet such as a dog or a cat. When the imaging device 2 is operated in a store, the living creature MB2 may be any living creature moving around in the store. For example, if the living creature MB2 is a human being MB21, the living creature MB2 (human being MB21) moving around in the store may be a store clerk, a patrol officer, a customer (client), etc. The patrol officer may include, for example, a security guard or a police officer. The store clerk may include not only a person employed by the store but also any person working in the store (such as a cleaner). According to one example of the present embodiment, when the imaging device 2 is attached to the living creature MB2 (at least one of a human being MB21 and another living creature MB22) to collect images 20, efficient acquisition of images 20 can be expected.

[0037] When multiple imaging systems SY are operated, the type of moving body MB to which each imaging device 2 is attached may be selected arbitrarily. The types of moving body MB to which each imaging device 2 is attached may be at least partially the same or different. In one example, multiple moving bodies MB may be operated. The multiple moving bodies MB may include one or more devices MB1 and one or more living organisms MB2. The imaged range (map information 125) may be shared or independent between each moving body MB. The number of imaging devices 2 attached to one moving body MB may be one, or two or more. The number of imaging devices 2 whose operation is controlled by one control device 1 may be one, or two or more.

[0038] (Attached separately to the moving object) The method for separately attaching the imaging device 2 to the mobile body MB may be determined appropriately depending on the embodiment. For example, if the mobile body MB is device MB1, separately attaching the imaging device 2 to the mobile body MB may include attaching the imaging device 2 to the mobile body MB using a control system separate from device MB1 constituting the mobile body MB. Attaching the imaging device 2 to the mobile body MB using a separate control system may include externally attaching the imaging device 2 to the mobile body MB using any method, such as physically fixing the imaging device 2 to the housing of the mobile body MB. The attachment method may be a known method such as gluing, screwing, fitting, or using a fastener. The fastener may include a clamp, a peg, a clip, or the like. If the mobile body MB (device MB1) has a power source (such as a battery), it may be equipped with an external interface capable of receiving external power, such as a USB (Universal Serial Bus). In this case, attaching the imaging device 2 to the mobile body MB using a separate control system may include connecting the imaging device 2 to the external interface of the mobile body MB and operating the imaging device 2 while receiving power from the power source of the mobile body MB.

[0039] Also, for example, if the moving body MB is a living being MB2, separately attaching it to the moving body MB may include being held by the moving body MB, being attached to clothing or equipment of the moving body MB, etc. Clothing may include any object worn on the body. Clothing may include, for example, clothes, accessories, etc. Clothing may include pet clothing. Accessories may include hats (including helmets), belts, bracelets, etc. Accessories may include pet collars, harnesses, leads, etc. Equipment may be carried on the moving body MB in any way, such as by being held or worn. It may be any object. The equipment may include, for example, a bag, a basket, etc. When the imaging device 2 is operated in a store, the equipment of the mobile body MB may include, for example, any equipment that can be used in a store, such as a shopping basket, a shopping bag grip, etc. The method of attaching the imaging device 2 to clothing or equipment is not particularly limited and may be determined appropriately depending on the embodiment. Known methods may be used for the attachment method. For example, separately attaching the imaging device 2 to the mobile body MB may include attaching it to a pocket of the clothing of the mobile body MB.

[0040] [System Configuration] The configuration of the imaging system SY is not particularly limited and may be determined appropriately depending on the embodiment as long as it includes the control device 1, the imaging device 2, and the sensor 3. The components of the control device 1, the imaging device 2, and the sensor 3 may be provided integrally, or at least partially separately.

[0041] In one example, as shown in FIG. 1, the imaging system SY may be configured by a terminal device UT that integrally includes a control device 1, an imaging device 2, and a sensor 3. The type of the terminal device UT may be appropriately selected depending on the embodiment. The terminal device UT may be, for example, a mobile phone (smartphone, etc.), a tablet terminal, a notebook PC (Personal Computer), a dedicated device, etc. According to one example of the present embodiment, when the terminal device UT is used as the imaging system SY, it is possible to expect efficient acquisition of the image 20. However, the configuration of the imaging system SY is not limited to this example and may be changed as appropriate depending on the embodiment.

[0042] FIG. 3 schematically illustrates another example of the configuration of the imaging system SY according to this embodiment. In another example, the imaging device 2 and the sensor 3 may be integrated, and the control device 1 may be provided separately. For example, the terminal device UT1 may include the imaging device 2, the sensor 3, and the controller 40. That is, similar to the example of FIG. 1, the imaging device 2 and the sensor 3 of the imaging system SY may be provided in the terminal device UT1. The terminal device UT1 may be similar to the terminal device UT. The controller 40 may be composed of one or more computers. The controller 40 may be similar to the control device 1 of the terminal device UT. However, unlike the example of FIG. 1, the control device 1 of the imaging system SY may be provided separately from the terminal device UT1. The controller 40 may not operate as the control device 1, but may operate as a communication device that performs wired or wireless data communication with the control device 1. The communication standard is not particularly limited and may be selected appropriately depending on the embodiment. The control device 1 may be directly or indirectly connected to the controller 40, and may acquire the measurement results 30 from the sensor 3 via the controller 40, and may control the operation of the imaging device 2. In this case, the control device 1 may be a terminal device separate from the terminal device UT1, a general-purpose PC, a server device, a dedicated device, or the like.

[0043] In yet another example, at least one of the imaging device 2 and the sensor 3 may be installed separately from the terminal device (UT, UT1). When the imaging device 2 and the sensor 3 are installed separately from the terminal device (UT, UT1) and configured to be able to communicate with each other via wired or wireless communication, the controller 40 may be omitted. The control device 1, the imaging device 2, and the sensor 3 may each be installed separately. In this case, the control device 1 may be appropriately connected to the imaging device 2 and the sensor 3 via wired or wireless communication. The imaging device 2 may be configured by a sensor that generates an image or data similar to an image through observation. The type of the imaging device 2 may be selected arbitrarily. The imaging device 2 may include, for example, a general RGB camera, a hemispherical camera, a celestial sphere camera, etc.

[0044] [Start / Stop imaging device] Starting up the imaging device 2 may include any operation that puts the imaging device 2 into a state where it can capture images. Stopping the activation of the imaging device 2 may include any operation that puts it into a power-saving state compared to a state where it can capture images. For example, starting up the imaging device 2 may be turning on the power of the imaging device 2 or turning on the imaging operation of the imaging device 2. Turning on the imaging device 2 may be, for example, starting an imaging application, entering an imaging execution mode, etc. On the other hand, stopping the activation of the imaging device 2 may be turning off the power of the imaging device 2 or turning off the imaging operation of the imaging device 2. Turning off the imaging operation may include entering a standby or sleep mode.

[0045] [posture] In one example, the posture may include a position and an orientation. The posture may be measured in two or three dimensions. Measurement conditions such as a posture measurement interval, a time resolution, and a type of coordinate system may not be particularly limited and may be determined appropriately depending on the embodiment.

[0046] The measurement data of the sensor 3 may directly indicate the posture measurement result 30. That is, the posture measurement result 30 may be obtained directly from the measurement data of the sensor 3. Alternatively, the posture measurement result 30 may be obtained by applying a predetermined arithmetic processing (analysis processing) to the measurement data of the sensor 3. That is, the posture measurement result 30 may be obtained indirectly from the measurement data of the sensor 3. The arithmetic processing for obtaining the posture measurement result 30 from the measurement data may be executed on the control device 1 or on a computer other than the control device 1. The control device 1 may obtain the posture measurement result 30 from the sensor 3 via another computer. If the sensor 3 includes processor resources, the arithmetic processing for obtaining the posture measurement result 30 from the measurement data may be executed on the sensor 3.

[0047] In one example, the sensor 3 may be configured to be able to measure the posture in higher dimensions relative to the posture measurement result 30. For example, the sensor 3 may be configured to be able to measure a three-dimensional posture, while the two-dimensional measurement result 30 may be obtained from the measurement data of the sensor 3. That is, the measurement data of the sensor 3 may be used with a reduced dimension. In another example, the measurement data of the sensor 3 may be used as is without reducing the dimension.

[0048] [Sensor] (sensor type) The sensor 3 may be configured with one or more sensor devices that observe the state related to the attitude of the imaging device 2. As long as the attitude of the imaging device 2 can be observed, the type of the sensor 3 is not particularly limited and may be appropriately selected depending on the embodiment.

[0049] (1) Inertial sensor As shown in FIG. 1, in one example, the sensor 3 may be an inertial sensor 3S. The type of the inertial sensor 3S may be appropriately selected depending on the embodiment. For example, the inertial sensor 3S may be configured to measure inertial motion (translational motion and rotational motion) with six degrees of freedom by including an acceleration sensor and a gyro sensor (angular velocity sensor). The inertial sensor 3S may be configured to measure inertial motion with nine degrees of freedom by further including a geomagnetic sensor. The inertial sensor 3S may be an inertial measurement unit (IMU) or a motion sensor. The degrees of freedom of measurement by the inertial sensor 3S are not limited to this example, and may be set to other than six or nine degrees of freedom.

[0050] When the inertial sensor 3S is used as the sensor 3, the attitude measurement result 30 may be obtained by analyzing the measurement data of the inertial sensor 3S. As long as the attitude can be derived, the method of analyzing the measurement data is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the analysis method may employ a self-position estimation method such as inertial navigation or pedestrian dead reckoning (PDR). Measuring the attitude is useful for estimating the self-position. In another example, a trained machine learning model may be used to analyze the measurement data. The machine learning model may be configured to have one or more operational parameters that can be adjusted by machine learning. The one or more operational parameters may be adjusted to achieve a desired result. The machine learning model is used to calculate inference (posture estimation) based on the posture information. The machine learning model may be configured, for example, by a neural network, a regression model, a decision tree model, a support vector machine, or other functional formulas. The machine learning method may be appropriately selected depending on the machine learning model to be adopted (for example, backpropagation). In the machine learning, the values ​​of each calculation parameter of the machine learning model may be appropriately adjusted (optimized) using training samples so as to acquire the ability to derive posture information from the measurement values ​​of the inertial sensor 3S.

[0051] The inertial sensor 3S can observe the attitude not only outdoors but also indoors. Therefore, according to one example of this embodiment, by using the inertial sensor 3S as the sensor 3, the imaging system SY can be operated not only outdoors but also indoors. Note that in measurements using the inertial sensor 3S, the attitude measurement result 30 is obtained as a relative value from the initial attitude. The attitude measurement result 30 may be used as a relative value as is, or may be used after being adjusted to an absolute value in real space by any method. For example, when a specific attitude is passed through at a specific time point, such as when starting movement from a specific point, the attitude measurement result 30 may be adjusted to an absolute value in real space by aligning the measurement value at the specific time point with the value of the specific attitude.

[0052] (2) Other In another example, the sensor 3 may include a positioning module. The positioning module may include, for example, a GPS (Global Positioning System) sensor, a GNSS (Global Navigation Satellite System) sensor, etc. When the attitude includes a position and an orientation, the attitude measurement result may be The position measurement results of the sensors 30 may be obtained by a positioning module. The measurement data of the positioning module may be used as the position measurement results as is. The orientation measurement results may be obtained as appropriate by a sensor other than the positioning module. That is, the sensor 3 may further include another sensor for measuring orientation. The other sensor may be, for example, an inertial sensor, a geomagnetic sensor, etc.

[0053] In yet another example, the sensor 3 may include a communication module and be configured to measure its position using wireless communication with wireless access points (beacons, base stations, etc.). The wireless communication standard may be selected arbitrarily. The measurement data of the communication module may be configured to indicate the strength of signals received from each wireless access point. When the attitude includes position and orientation, the position measurement result among the attitude measurement results 30 may be derived as appropriate from the measurement data of the communication module. For example, the distance to each wireless access point may be estimated from the signal strength indicated by the measurement data. The relationship between signal strength and distance may be given as appropriate. The position measurement result may be derived from the estimated distance to each wireless access point using a known method (such as triangulation). When the communication method of the communication module is capable of measuring the angle of arrival (AoA), the orientation measurement result may also be derived as appropriate from the measurement data of the communication module. Alternatively, as in the case of the positioning module described above, the sensor 3 may further include another sensor for measuring orientation. The orientation measurement result may be obtained by another sensor.

[0054] (Sensor placement) As long as the state related to the attitude of the imaging device 2 can be observed directly or indirectly, the location of the sensor 3 is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the sensor 3 may be built into the imaging device 2 or a device including the imaging device 2 (such as a terminal device UT). In another example, the sensor 3 may be provided separately from the imaging device 2, or may be attached to the imaging device 2 as appropriate. In yet another example, the sensor 3 may be attached to the moving body MB together with the imaging device 2. In this way, the sensor 3 may be arranged to indirectly measure the attitude of the imaging device 2 via the moving body MB.

[0055] (Continuous measurement) The sensor 3 may continuously measure the orientation of the imaging device 2 regardless of whether the imaging device 2 is performing imaging. Continuously measuring the orientation may include continuously estimating the self-orientation (self-position). Continuously obtaining the orientation measurement result 30 may include obtaining a movement trajectory. The movement trajectory may be configured to include orientation (position) measurement values ​​at one or more times (points of time) to chronologically indicate the progress of the movement of the moving body MB (imaging device 2). For example, the movement trajectory may be configured by one or more combinations of orientation measurement values ​​and measurement times. The orientation measurement value may be a measurement result 30 or a corrected measurement result 32 described below. When processing is performed in real time, the time at which it is determined whether the target range 35 has been imaged may be the current time, and the target range 35 may be identified from the measurement result 30 at the current time among the measurement results 30 that constitute the movement trajectory. Accordingly, the target range 35 may be the range to which the imaging device 2 is currently facing.

[0056] An image group may be obtained by capturing images using the imaging device 2. The image group may be composed of one or more images 20. The acquired image group and movement trajectory (posture measurement results 30) may be associated and stored. The image group and movement trajectory may be stored in any storage area. The storage area may be, for example, the memory resource of the control device 1, the memory resource of another computer, an external storage device, etc. The external storage device may include a data server such as a NAS (Network Attached Storage). The storage area for storing the image group and movement trajectory may be changed as appropriate before at least one of the image group and movement trajectory is used, such as by temporarily storing the image group in the memory resource of the control device 1 and then transferring it from the control device 1 to an external storage device. The image group and movement trajectory may be transferred via a network, a storage medium, etc. The movement trajectory may be associated with the image group in any manner. In one example, the movement trajectory may be directly associated with the image group. Additionally, associating the image group and the movement trajectory may include associating a measurement value of the image capture time of each image 20 included in the image group and each image 20 on the movement trajectory. The associated posture measurement value may be a measurement result 30 or a corrected measurement result 32 described below. The image capture time may be obtained from a timer. The timer may be provided in any device. The timer may be included in, for example, the control device 1 or the image capture device 2.

[0057] The period during which the attitude is measured by the sensor 3 may be set as appropriate depending on the embodiment. In one example, the period during which the attitude is measured by the sensor 3 may be defined as the operation period of the imaging device 2. The attitude measurement by the sensor 3 may start from the start time of imaging by the imaging device 2 or any time before the start time of imaging. Furthermore, the attitude measurement by the sensor 3 may end at the end time of imaging by the imaging device 2 or any time after the end time of imaging. As a result, the operation period of the imaging device 2 may include the period during which the image group is captured. For example, the attitude measurement by the sensor 3 and the imaging by the imaging device 2 may start in response to the establishment of a start trigger. The attitude measurement by the sensor 3 and the imaging by the imaging device 2 may end in response to the establishment of an end trigger. The start trigger and the end trigger may be set arbitrarily. The start trigger may be, for example, the start of movement of the moving body MB, the pressing of a physical or software switch, an instruction to execute an application, etc. The end trigger may be, for example, the end of movement of the moving body MB, the pressing of a physical or software switch, an instruction to end an application, etc.

[0058] As a specific example, when an inertial sensor 3S is used as the sensor 3, the control device 1 may detect the start of movement of the moving body MB based on the measurement data of the inertial sensor 3S. The criteria for detecting the start of movement may be set arbitrarily. For example, the start of movement of the moving body MB may be detected in response to the amount of change in attitude specified by the measurement data exceeding a threshold. The threshold for detecting the start of movement may be set arbitrarily. Moving may include stopping in place and changing direction. Then, in response to detecting the start of movement of the moving body MB, the control device 1 may start recording the measurement results 30 of the attitude (movement trajectory) by the inertial sensor 3S. Furthermore, the control device 1 may detect the end of movement of the moving body MB based on the measurement data of the inertial sensor 3S. The criteria for detecting the end of movement may be set arbitrarily. For example, The stop of movement of the moving body MB may be detected in response to the amount of change in the attitude specified by the above measurement being less than a threshold. The threshold for detecting the stop of movement may be set arbitrarily. The end of movement of the moving body MB may be detected in response to the stop of movement of the moving body MB continuing for a predetermined period of time. The threshold for the period for detecting the end of movement may be set arbitrarily. Then, in response to detecting the end of movement of the moving body MB, the control device 1 may stop recording the attitude measurement result 30 (movement trajectory) by the inertial sensor 3S.

[0059] The measurement period of the sensor 3 is not limited to this example and may be changed as appropriate depending on the embodiment. The sensor 3 may measure the attitude of the imaging device 2 during a period other than the operation period of the imaging device 2. Furthermore, if it does not matter whether imaging has already been performed, imaging by the imaging device 2 may be performed independently of the measurement of the attitude by the sensor 3.

[0060] [Map Information] The imaging range of the imaging device 2 can be specified depending on the posture of the imaging device 2 at the time of imaging. Therefore, as long as at least one of the posture of the imaged device 2 and the imaging range due to that posture can be specified, the configuration of the map information 125 is not particularly limited and may be determined appropriately depending on the embodiment.

[0061] As shown in FIG. 1 , in one example, the map information 125 may be configured to include a posture measurement history at the time of imaging. The posture measurement history may be a list of posture measurement values. The posture measurement value at the time of imaging may also be referred to as the imaging posture (imaged posture). That is, a record of the imaged range (posture measurement history) may be shown as a list of past imaging postures, and the map information 125 may be defined as information indicating the history of the imaging posture. At least some of the posture measurement values ​​(imaged postures) may be measurement results 30 or corrected measurement results 32, which will be described later. Accordingly, updating the map information 125 to indicate that the target range 35 has been imaged may be configured by adding the posture measurement results 30 at the time of imaging to the map information 125.

[0062] Note that the configuration of the map information 125 is not limited to this example and may be modified as appropriate depending on the embodiment. In one example, the map information 125 may further include the time of each image capture (image capture time). The posture measurement value at each image capture time may be recorded in association with each image capture time. The unit of image capture time may be defined arbitrarily. The image capture time may include, for example, year / month / day / hour / minute / second. In one example, history with a long elapsed time (i.e., old history), such as when the elapsed time from the image capture time to the update time exceeds a threshold, may be deleted from the map information 125. The update time may be the time at which the map information 125 is updated, and the threshold used as a criterion for determining whether the elapsed time is long may be set arbitrarily.

[0063] Furthermore, the map information 125 may be configured to include a history of the imaging range together with or instead of the history of the imaging orientation. The imaging range may be appropriately identified from the imaging orientation and the imaging characteristics (angle of view, focal length, etc.) of the imaging device 2. The imaging orientation and imaging range may each be expressed directly or indirectly using a degree of probability, etc. For example, the map information 125 may be configured to indicate the degree of whether or not imaging has been completed, such as a heat map. The degree of whether or not imaging has been completed may be calculated using any method from the history of the imaging orientation. In one example, the degree of whether or not imaging has been completed may be obtained by expressing a set of past imaging orientations as a probability distribution using a known method (e.g., Non-Patent Document 1).

[0064] Furthermore, the map information 125 may further include any information other than information about the already captured range (imaging attitude and imaging range). In one example, the map information 125 may further include environmental information about the real environment in which the moving body MB exists. The environmental information may include, for example, constraint information indicating constraints on movement conditions such as a predetermined movement range and internal constraints within the movement range. Moving body If the MB is an entity that moves within the store (such as a robotic device used within the store, a manually operated device used within the store, or a living thing that moves within the store), the map information 125 may include floor information related to floors within the store, such as a floor map, a floor guide, etc. The constraint information may be obtained from the floor information.

[0065] In one example, the initial data of the map information 125 may be generated appropriately at any time before the first imaging process is performed or when the first imaging process is performed. Furthermore, the map information 125 may be provided for each predetermined unit of movement. The unit of movement may be defined arbitrarily. The unit of movement may be defined according to the task of the mobile body MB, such as one trip, a period for performing a task, etc. For example, if the mobile body MB starts moving from a specific point and ends its movement by returning to the specific point, one trip may be defined as the period from starting movement from the specific point to returning to the specific point. The specific point may be defined arbitrarily. For example, if the mobile body MB is an autonomously moving robotic device, the specific point may be a point originating from the robotic device, such as a charging point for the robotic device. Furthermore, the unit of movement may be defined in terms of time, such as a specific period of movement. The unit of movement may be defined by a period of movement, such as a time period (morning, noon, night, etc.), a day, a day of the week, etc. For example, if a specified unit of movement is defined as one voyage, the history of the imaging attitude added in the Nth voyage in the map information 125 may be reset before or when the N+1th voyage begins (N is any natural number).

[0066] When multiple mobile MBs are used, the map information 125 may be shared by at least some combination of the multiple mobile MBs, or may be stored independently for each mobile MB. A known sharing method may be used to share the map information 125.

[0067] (Whether the target area has already been photographed) 4 schematically shows an example of a scene in which it is determined whether or not the target range 35 according to this embodiment has been imaged. Whether or not the target range 35 has been imaged may be determined as appropriate using map information 125. In one example, the control device 1 may determine whether or not the target range 35 has been imaged according to a predetermined criterion. The predetermined criterion may be set as appropriate so that the probability of executing the image capture process increases as the unimaged range becomes more likely to be captured.

[0068] In one example, the predetermined criterion may be defined to determine whether or not the target range 35 has been imaged depending on the discrepancy between the posture measurement result 30 and the imaging posture included in the map information 125. For example, the control device 1 may determine that the target range 35 has not been imaged depending on whether the discrepancy between the posture measurement result 30 and the imaging posture included in the map information 125 exceeds a threshold or is equal to or greater than a threshold. The control device 1 may determine that the target range 35 has been imaged depending on whether the discrepancy between the posture measurement result 30 and the imaging posture included in the map information 125 is equal to or less than a threshold. The threshold used as the criterion for determining whether or not the target range 35 has been imaged may be set arbitrarily.

[0069] When the posture includes a position and an orientation, the deviations of the position and the orientation may be evaluated separately or collectively. For example, when the deviations of the position and the orientation are evaluated separately, the control device 1 may determine that the target range 35 has not been imaged if the deviation of at least one of the position and the orientation exceeds a threshold or is equal to or greater than a threshold. The control device 1 may determine that the target range 35 has been imaged if the deviations of both the position and the orientation are equal to or less than a threshold. Furthermore, for example, the deviations of the position and the orientation may be expressed collectively by any index such as distance. A known distance such as Euclidean distance may be used as the distance for evaluating the deviation. The control device 1 may calculate the distance between the posture measurement result 30 and the imaged posture included in the map information 125. The control device 1 may determine that the target range 35 has not been imaged if the calculated distance exceeds a threshold or is equal to or greater than a threshold. The control device 1 may determine that the target range 35 has not been imaged if the calculated distance is equal to or less than a threshold. It may be determined that the target area 35 has already been imaged.

[0070] In another example, the predetermined criterion may be defined to determine whether the target range 35 has been imaged based on the degree to which the imaging range (target range 35) in the measured orientation (measurement result 30) has not yet been imaged. For example, the map information 125 may be configured to represent a set of past imaging orientations as a probability distribution using a known method (e.g., Non-Patent Document 1). Alternatively, the control device 1 may calculate the probability distribution of imaging orientations from a list of past imaging orientations included in the map information 125. The control device 1 may calculate the amount of information of the imaging range (target range 35) in the measured orientation (measurement result 30) relative to the probability distribution of imaging orientations obtained from the map information 125 using the method of Non-Patent Document 1. The higher the degree to which the target range 35 has not yet been imaged, the larger the calculated amount of information. Therefore, the control device 1 may determine that the target range 35 has not yet been imaged based on whether the calculated amount of information exceeds a threshold or is equal to or greater than a threshold. The control device 1 may determine that the target range 35 has already been imaged based on whether the calculated amount of information is equal to or less than a threshold.

[0071] The predetermined criteria are not limited to these examples and may be changed as appropriate depending on the embodiment. When determining whether the target range 35 has been imaged, the control device 1 may refer to all pieces of information (such as the image-capturing attitude) included in the map information 125, or may refer to only the most recent k pieces of information (k is an arbitrary natural number). The range of information to be referenced may be defined as appropriate depending on the embodiment.

[0072] The predetermined criteria may further include conditions other than the above-described condition regarding whether an image has been captured. In one example, the predetermined criteria may further include a time condition. For example, the control device 1 may determine whether an image of the target range 35 has been captured within a predetermined period from the current time by deleting history records with a long elapsed time in the map information 125, or by referencing only history records within a predetermined period from the current time. The current time may be the time at which it is determined whether an image of the target range 35 has been captured. Furthermore, for example, the map information 125 may be generated for each predetermined unit of movement, or by referencing only history records corresponding to the predetermined unit of movement, or by other such method, the control device 1 may determine whether an image of the target range 35 has been captured within the predetermined unit of movement. As described above, the unit of movement may be defined arbitrarily. The unit of movement may be defined according to the task of the moving body MB, or may be defined by a time unit (such as a movement period). For example, the control device 1 may determine whether an image of the target range 35 has been captured within the Nth navigation. Furthermore, for example, the control device 1 may determine whether or not the target range 35 has been imaged within a specific movement period (time period, date, day of the week, etc.). The movement period of the target to be determined may be selected arbitrarily.

[0073] [Image usage scenarios] One or more images 20 (image group) captured by the imaging device 2 may be stored as appropriate. The one or more stored images 20 (image group) may be used as desired.

[0074] (1st usage scenario) FIG. 5 schematically shows an example of a usage scenario of an image 20 captured by the imaging device 2 according to this embodiment. In one example, the imaging system SY may further include a storage device MR. The storage device MR may store the image 20 captured by the imaging device 2 in association with a measurement result 39 (imaging attitude) of the attitude of the imaging device 2 at the time of capturing the image 20. The measurement result 39 may be the measurement result 30 or a corrected measurement result 32, which will be described later. The storage device MR may further store the imaging time TT of each image 20. Each image 20 may be further associated with the imaging time TT of each image 20. The imaging time TT may be omitted.

[0075] In one example, each combination of the image 20, the measurement result 39, and the time TT may be stored as a database in the storage device MR. To search the base, an information processing device 5 may be provided. The information processing device 5 may be provided as a component within the imaging system SY, or may be provided as a component external to the imaging system SY. In one example, the information processing device 5 may be configured by a computer other than the control device 1. In another example, the control device 1 may operate as the information processing device 5. That is, the information processing device 5 may be configured by the control device 1.

[0076] The database may be constructed by any computer. In one example, at least one of the control device 1, the information processing device 5, and the other information processing device 45 may be involved in constructing the database. The other information processing device 45 may be a computer other than the control device 1 and the information processing device 5. The other information processing device 45 may be composed of one or more arbitrary computers. The storage device MR may be arranged arbitrarily. In one example, the storage device MR may be composed of at least one of the memory resources of the control device 1, the memory resources of the information processing device 5, the memory resources of the other information processing device 45, and an external storage device.

[0077] The information processing device 5 may be one or more computers configured to search a database (storage device MR) in response to a request from a user U1. The user U1 may be any person. In one example, the information processing device 5 may receive a query 60 from a terminal device T1 of the user U1. The terminal device T1 may be connected to the information processing device 5 via a network or directly. The query 60 may include at least one of a target image, a target posture, and a target time for the search. The user U1 may operate the terminal device T1 to appropriately specify at least one of the target image, the target posture, and the target time. The target image may consist of one image or multiple images. The target posture may be specified by a single point or a range. The target posture may be configured to specify only a target position. The target time may be specified by a single time or a range (such as a time period, a day, or a day of the week).

[0078] The information processing device 5 may search a database (storage device MR) to extract elements matching the query 60 from the database. As a result, the information processing device 5 can obtain a search result 65 composed of the extracted elements. Whether or not the elements match the query 60 may be determined appropriately. The extracted elements may be composed of at least one of an image 20, a posture measurement result 39, and an image capture time TT. For example, if the query 60 includes a target image, the information processing device 5 may compare each image 20 stored in the storage device MR with the target image and, based on the comparison result, extract an image 20 that matches the target image from among the images 20 stored in the storage device MR. In one example, the information processing device 5 may further extract an image 20 similar to the target image. The criterion for similarity may be defined arbitrarily, for example, that the degree of match between the images exceeds a threshold. Similarly, when the query 60 includes a target posture, the information processing device 5 may compare the measurement results 39 of each posture stored in the storage device MR with the target posture, and extract measurement results 39 that match the target posture from the measurement results 39 of each posture stored in the storage device MR based on the comparison result. In one example, the information processing device 5 may further extract measurement results 39 that belong to a neighborhood range of the target posture. The neighborhood range may be defined arbitrarily, for example, by a deviation from the target posture that is less than a threshold. Furthermore, when the query 60 includes a target time, the information processing device 5 may compare each imaging time TT stored in the storage device MR with the target time, and extract imaging times TT that match or belong to the target time from each imaging time TT stored in the storage device MR based on the comparison result.

[0079] In one example, when extracting a search element from the database, the information processing device 5 may further extract other elements associated with the search element. For example, when a target image is specified as the query 60, the search result 65 may include at least one of the posture measurement result 39 and the image capture time TT associated with the extracted image 20, in addition to the image 20 extracted as matching the target image. When a target posture is specified as the query 60, the search result The search result 65 may include at least one of the image 20 and the image capture time TT associated with the extracted measurement result 39, in addition to the posture measurement result 39 extracted as matching the posture of interest. Furthermore, when a target time is specified as the query 60, the search result 65 may include at least one of the image 20 and the posture measurement result 39 associated with the extracted image capture time TT, in addition to the image capture time TT extracted as matching the target time. If there is no element matching the query 60, the search result 65 may be empty.

[0080] The information processing device 5 may then return the obtained search results 65 to the terminal device T1. The terminal device T1 may receive the search results 65 from the information processing device 5 and output the received search results 65 as appropriate. This allows the user U1 to obtain the search results 65 in response to the request (query 60). The search results 65 may be used for any purpose. Note that if the imaging time TT is omitted, the search by target time may be omitted.

[0081] As a specific example, a database (storage device MR) and an information processing device 5 may be provided in a scene where a mobile object MB is an entity moving within a store and an image 20 of the store is collected by an imaging device 2. In this scene, a user U1 may be a customer, a store clerk, etc. The mobile object MB, which is an entity moving within the store, may be, for example, a robot device that moves autonomously within the store, a device that moves manually within the store, a living thing that moves within the store, etc.

[0082] For example, the user U1 may request a search for images 20 containing a target object (such as a product, a sign, or an advertisement) that may be present in a store by specifying the target image 20 as a query 60. If the target object is a product, the target image may be a product image of the product. In one example, if the target object is a product, the target image (product image) containing the target object may be provided from coupon information for the product. In response to receiving the query 60 including the target image containing the target object from the terminal device T1, the information processing device 5 may search for images 20 that match the query 60 (i.e., images 20 containing the target object). The information processing device 5 may extract the images 20 containing the target object from a database (storage device MR) and further extract posture measurement results 39 associated with the extracted images 20. The information processing device 5 may return the extracted images 20 and posture measurement results 39 to the terminal device T1 as search results 65. The terminal device T1 may appropriately output the posture measurement results 39 obtained as the search results 65. This allows the user U1 to confirm the position of the target object. In one example, the terminal device T1 may perform route guidance from the current position of the user U1 to the position of the target object. The current position of the user U1 may be appropriately obtained by the terminal device T1 using a known method. The route guidance may be performed using a known method. The orientation measurement result 39 may be used as the position of the target object. That is, the terminal device T1 may perform route guidance to the position indicated by the measurement result 39. If the orientation includes a position and an orientation, the terminal device T1 may further provide guidance on the orientation indicated by the measurement result 39 upon reaching the position indicated by the measurement result 39. Alternatively, the position of the target object may be appropriately estimated from the orientation measurement result 39, the position of the target object in the image 20, the size of the target object, and the like. A known image processing method may be used to estimate the position of the target object. The estimation of the position of the target object may be performed by any computer. The terminal device T1 may perform route guidance to the estimated position.

[0083] Furthermore, for example, the user U1 may request a search for a specified position or a specified range by specifying an arbitrary position or a range of positions within a store as a target posture. In response to receiving a query 60 including the specified position or the specified range as a target posture from the terminal device T1, the information processing device 5 may search for measurement results 39 of postures that match the target posture. The information processing device 5 may extract the measurement results 39 of postures that match the target posture, and may further extract an image 20 associated with the extracted measurement results 39. The information processing device 5 may then search for the measurement results 39 of the extracted postures. The terminal device T1 may return the measurement result 39 and the image 20 to the terminal device T1 as a search result 65. The terminal device T1 may appropriately output the image 20 obtained as the search result 65. This allows the user U1 to confirm the scene at the specified position or within the specified range via the output image 20. In one example, the terminal device T1 may perform an information search using the obtained image 20. For example, the terminal device T1 may provide the obtained image 20 to a search engine to perform a web search for information on an object (such as a product, a sign, or an advertisement) appearing in the image 20. The terminal device T1 may provide the obtained image 20 to the search engine as is, or may extract an area in which the object appears from the obtained image 20 and provide a partial image of the extracted area to the search engine. A known search engine may be used as the search engine. The terminal device T1 may appropriately output search results obtained from the search engine. This allows the user U1 to obtain information on the object appearing in the image 20. The information to be searched on the web may be selected appropriately depending on the embodiment. In one example, the object may be a product, and the terminal device T1 may obtain any information about the product shown in the image 20 as a result of a web search, such as coupon information, review information, price information, etc.

[0084] Furthermore, for example, the user U1 may request a search for a specified time by specifying an arbitrary time as the target time. In response to receiving a query 60 including the specified time as the target time from the terminal device T1, the information processing device 5 may search for an image capture time TT that matches the target time. The information processing device 5 may extract an image capture time TT that matches the target time and further extract an image 20 associated with the extracted image capture time TT. The information processing device 5 may return the extracted image capture time TT and the image 20 to the terminal device T1 as a search result 65. This allows the user U1 to check the image 20 obtained at the specified time. In one example, the information processing device 5 may further extract a posture measurement result 39 associated with the extracted image capture time TT. This allows the search result 65 to further include the posture measurement result 39. This allows the user U1 to further check the image 20 as well as the image 20 and the posture at which the image 20 was captured.

[0085] According to one example of this embodiment, by storing posture measurement results 39 in association with images 20 in the storage device MR, it is possible to provide information based on a combination of the posture measurement results 39 and the images 20. For example, as shown in the example of FIG. 5 above, a search system (information processing device 5) for accessing one of an image and a posture from the other can be provided. Note that the form of searching the database is not limited to the above example and may be modified as appropriate depending on the embodiment. In another example, the information processing device 5 may execute a search of the database (storage device MR) by being directly operated without going through the terminal device T1. Furthermore, the terminal device T1 may execute a search of the database (storage device MR) by directly accessing the storage device MR without going through the information processing device 5.

[0086] (Second usage scenario) FIG. 6 schematically illustrates another example of a usage scenario for images 20 captured by the imaging device 2 according to this embodiment. In one example, similar to FIG. 5, the imaging system SY may further include a storage device MR. The storage device MR may store each image 20, a measurement result 39 of the posture of the imaging device 2 at the time of capturing each image 20, and an imaging time TT in association with each other. The imaging time TT may be omitted. In addition, the storage device MR may further store a result 25 of image analysis for each image 20. Each image 20 may be further associated with the result 25 of image analysis for each image 20. That is, the storage device MR may store the image 20 in association with the result 25 of image analysis for each image 20. Except for the point related to the result 25 of image analysis, the configuration of FIG. 6 may be the same as the configuration of FIG. 5.

[0087] The image analysis of the image 20 may be performed by any computer at any time after the image 20 is acquired. The control device 1 may perform image analysis on the image 20 at any timing after the image 20 is acquired. The control device 1 may perform image analysis each time an image 20 is acquired, or may perform image analysis on each image 20 collectively after acquiring multiple images 20. The control device 1 may perform image analysis using any method. The control device 1 may perform image analysis using a known method, such as a general image analysis method (edge ​​extraction, pattern matching, etc.) or a method using a trained machine learning model. This allows the control device 1 to acquire an image analysis result 25. The control device 1 may store the acquired image analysis result 25 in association with the image 20. Note that the computer that performs the image analysis is not limited to the control device 1 and may be changed as appropriate depending on the embodiment. In another example, the information processing device 5 or another information processing device 45 may perform image analysis on the image 20 and store the image analysis result 25 in association with the image 20. The image analysis result 25 may be stored in the storage device MR immediately, or may be stored in the storage device MR via another storage area.

[0088] Image analysis may detect any feature appearing in image 20, and the results 25 of the image analysis may include any information about the detected feature.

[0089] In one example, the image analysis may include detection of an object OB appearing in the image 20. As a result, the image analysis result 25 may include a detection result of the object OB. If multiple objects OB appear in the image 20, a detection result may be obtained for each object OB. The object OB is an example of a feature appearing in the image 20. The type of object OB is not particularly limited and may be appropriately selected depending on the embodiment. The object OB may include, for example, products / merchandise (food, beverages, daily necessities, clothing, electrical appliances, etc.), tools (including household goods such as desks, chairs, shelves, etc.), equipment (kitchen equipment, refrigeration equipment, cash registers, etc.), architectural structures (entrances, doors, windows, pillars, etc.), monuments, artworks (statues, paintings, etc.), parts of these, etc. The object OB may also be referred to as a landmark.

[0090] In one example, detecting the object OB may include extracting the range (such as a bounding box) in which the object OB appears. Thus, the detection result of the object OB (image analysis result 25) may include the extraction result of the range in which the object OB appears (such as range information, a partial image, etc.). Also, in one example, detecting the object OB may include analyzing the attributes of the object OB, such as classifying the object OB, identifying the object OB, estimating the position of the object OB, and analyzing text information attached to the object OB. Thus, the detection result of the object OB may include the analysis result of the attribute information of the object OB. The attribute information of the object OB may include, for example, a category, identification information (such as a name or an identifier), an estimated position, text information attached to the object OB, etc. The attribute information of the object OB may include any information about the object OB obtained from reference information such as the web. For example, the attribute information of the object OB may include information about the components, elements, and other characteristics of the object OB. In one example, if the object OB is absent in the image 20, the detection result of the object OB may be configured to indicate that the object OB is not present. Alternatively, if the object OB is absent in the image 20, the association of the detection result of the object OB may be omitted.

[0091] The object OB to be detected may be selected as appropriate depending on the embodiment, such as the operating scene. In one example, the moving object MB may be an entity moving within the store, and the object OB may be a product OB1 sold within the store. As a result, the detection result of the object OB may include the detection result of the product OB1. In one example, the detection result of the product OB1 may include attribute information of the product OB1. The attribute information of the product OB1 may include, for example, the category of the product OB1, identification information (product name, identifier, etc.), estimated location, attached character information, other product information, etc. Note that when the moving object MB is an entity moving within the store, the object OB to be detected does not have to be limited to the product OB1. The object OB is In addition to or instead of the product OB1, the image may include an object other than the product OB1 present in the store, such as a signboard or advertisement.

[0092] In one example, the image analysis may include detection of an event EV captured in the image 20, in addition to or instead of detecting the object OB. Thus, the image analysis result 25 may include a detection result of the event EV. When multiple event EVs are captured in the image 20, a detection result may be obtained for each event EV. An event EV is an example of a feature captured in the image 20. The event EV may be defined appropriately depending on the embodiment, such as the purpose of detection. In one example, the event EV may be any phenomenon related to an object captured in the image 20, such as the behavior, situation, or appearance of the object. The detection result of the event EV may include an analysis result of a phenomenon related to the object. The analysis result of the phenomenon may include attribute information such as the category of the event EV, the estimated position of the object (estimated position of the event EV), and the content of the phenomenon. The analysis result of the phenomenon (the content of the phenomenon, etc.) may be expressed in any data format, such as text or numerical values. The object for which the event EV is detected may include the object OB, or may include an object other than the object OB. The object for which the event EV is detected may be selected appropriately depending on the embodiment. The number of objects involved in an event EV (event) may be one or more.

[0093] The event EV to be detected may be selected as appropriate depending on the embodiment of the operation scene, etc. In one example, the mobile object MB may be an entity that moves within a store, and the event EV may include at least one of a customer event EV1 related to customers in the store and a product event EV2 related to products sold in the store.

[0094] The customer event EV1 may include any phenomenon related to customers. For example, the customer event EV1 may include whether or not a specific object is being viewed, the number of customers present, etc. The specific object may be, for example, a product, a sign, an advertisement, etc. Whether or not a specific object is being viewed is an example of customer behavior. The number of customers present is an example of a customer's situation or appearance. The product event EV2 may include any phenomenon related to products. For example, the product event EV2 may include the number of remaining products, whether or not a product is out of stock, the arrangement of products on a display shelf, etc. The number of remaining products, whether or not a product is out of stock, and the arrangement of products on a display shelf are examples of a product situation or appearance. The number of remaining products or whether or not a product is out of stock may be used as an indicator for replenishing products. The arrangement of products on a display shelf may be used as an indicator for moving products forward (for example, if products are only present at the back of the display shelf, a notification may be sent to perform the moving product forward operation).

[0095] It should be noted that when the mobile object MB is an entity moving within a store, the event EV to be detected need not be limited to the customer event EV1 and the product event EV2, and may be changed as appropriate depending on the embodiment. The event EV may include any event that may occur within the store. In another example, the event EV may include at least one of a store clerk event related to a store clerk in the store, a sales promotion event related to a product promotion held within the store, and an equipment event related to equipment present within the store, together with or instead of at least one of the customer event EV1 and the product event EV2. The store clerk event may include any phenomenon related to the store clerk. The store clerk event may include, for example, the work status of the store clerk, the category of work performed by the store clerk, etc. The work of the store clerk may include product promotion activities such as attracting customers. The sales promotion event may include any phenomenon related to product promotion. Product promotion may include, for example, the installation of a special shelf, the placement of advertisements (such as POP advertisements: Point of Purchase Advertising), attracting customers by the store clerk, etc. The promotional event may include, for example, the status of the promotional location, etc. The promotional event may include, for example, the behavior of customers at the promotional location (whether or not they are looking at promotional products, etc.), the status of promotional products (number of remaining products, etc.), the status of promotional activities by store staff, etc. At least a part of the promotional event may be based on the customer interface. An equipment event may be detected as at least one of an event EV1, a product event EV2, and a store clerk event. An equipment event may include any phenomenon related to the equipment. Equipment may include, for example, a shopping basket, a shopping cart, a locker, etc. An equipment event may include, for example, the number of items remaining in a shopping basket, the number of items remaining in a shopping cart, the availability of lockers, etc.

[0096] Similar to the example of FIG. 5, the information processing device 5 may receive a query 60 from the terminal device T1. The information processing device 5 may search a database (memory device MR) to extract elements that match the query 60 from the database. As a result, the information processing device 5 may obtain a search result 65 that is composed of elements that match the query 60. The information processing device 5 may return the obtained search result 65 to the terminal device T1. Through this series of information processing, the information processing device 5 can provide information similar to that shown in FIG. 5.

[0097] 6, when at least one of a target image, a target posture, and a target time is specified as the query 60, the information processing device 5 may further extract the image analysis result 25 associated with the extracted search element. As a result, the search result 65 may further include the image analysis result 25 associated with the extracted search element in addition to the extracted search element (at least one of the image 20, the posture measurement result 39, and the imaging time TT).

[0098] Furthermore, the query 60 may include search target information for image analysis, together with or instead of at least one of the target image, target posture, and target time. The target information may be specified as appropriate depending on the embodiment. The target information may be configured to specify, for example, the category of the object / event to be searched, the name of the object (product name, etc.), text information attached to the object, the content of the phenomenon, and other attributes. The target information may be expressed in any data format, for example, text, numerical values, etc. When the query 60 includes the target information, the information processing device 5 may compare the image analysis results 25 for each image 20 stored in the storage device MR with the target information, and based on the comparison results, extract image analysis results 25 that match the target information from the image analysis results 25 stored in the storage device MR. Whether the image analysis results 25 match the target information may be determined as appropriate based on the degree of text match, the degree of satisfaction of numerical conditions, etc. As a result, the search results 65 may include the image analysis results 25 extracted as matching the target information. When the image analysis result 25 includes multiple items, the search result 65 may consist of only information on the item specified as the target information, or may be configured to include information on items other than the item specified as the target information. For example, in a situation where the image analysis includes detection of an object OB, the object OB includes a product OB1, and the image analysis result 25 includes the product name and the estimated location of the product OB1, the query 60 may be specified to include the product name to be searched. In contrast, the search result 65 may consist of only the product name that matches the specified product name as the extracted image analysis result 25, or may be configured to further include the estimated location of the product OB1.

[0099] In one example, the information processing device 5 may extract the image analysis result 25 and further extract at least one of the image 20, the posture measurement result 39, and the image capture time TT associated with the extracted image analysis result 25. As a result, the search result 65 may include, in addition to the extracted image analysis result 25, at least one of the image 20, the posture measurement result 39, and the image capture time TT associated with the extracted image analysis result 25. Note that if the image analysis result 25 includes an extraction result (partial image) of the range in which the object OB appears, in the above information search, the extraction result of the range in which the object OB appears may be matched with the target image together with or instead of the image 20. If the image analysis result 25 includes the estimated position of the object OB, in the above information search, the estimated position of the object OB may be matched with the target posture together with or instead of the posture measurement result 39.

[0100] As a specific example, assuming the above-described scene in a store, user U1 may request a search for a target product by specifying the product name as a query 60. In one example, the target product name (the name of the target product) may be provided from coupon information for the target product. In response to receiving the query 60 specifying the product name from terminal device T1, the information processing device 5 may search for image analysis results 25 that match the query 60 (specified product name). The information processing device 5 may extract the image analysis results 25 that match the query 60 and further extract images 20 and posture measurement results 39 associated with the extracted image analysis results 25. The information processing device 5 may return the extracted images 20, posture measurement results 39, and image analysis results 25 to terminal device T1 as search results 65. The terminal device T1 may appropriately output the images 20 obtained as the search results 65. This allows user U1 to confirm the scene of the specified target product. If the image analysis result 25 includes an extraction result of the area in which the product OB1 is captured, the terminal device T1 may output the extraction result of the area in which the target product is captured together with or instead of the image 20. In one example, the terminal device T1 may perform a web search for information about the target product by providing the extraction result of the area in which the target product is captured to a search engine. The terminal device T1 may also appropriately output a posture measurement result 39 obtained as a search result 65. This allows the user U1 to confirm the location of the target product. In one example, the terminal device T1 may perform route guidance from the current location of the user U1 to the location of the target product. The posture measurement result 39 obtained as the search result 65 may be used as the location of the target product. Alternatively, if the image analysis result 25 includes an estimated location of the product OB1, the estimated location of the target product included in the image analysis result 25 may be used as the location of the target product. Note that the attribute information used to search for the target product is not limited to the product name. Attribute information other than the product name may also be used as the search query 60. The information processing device 5 may replace the product name with other attribute information and execute the above search process. For example, the user U1 may specify, as the query 60, character information attached to the target object.This allows for obtaining search results 65 related to the target object with the specified character information. The target object may be the above-mentioned product, or an object other than a product, such as a signboard or advertisement. In one example, the terminal device T1 may perform route guidance to the location of an object other than a product (target object). According to one example of this embodiment, as long as the image 20 is obtained, even if the location of the target object has been changed, the changed location of the target object can be tracked.

[0101] Furthermore, similar to the search based on the attribute information of the target product described above, the user U1 may specify the attribute information (category, content, etc.) of the target event as target information (query 60). In response, the information processing device 5 may provide search results 65 related to the specified target event. In one example, if the search results 65 include at least one of the posture measurement result 39 and the estimated location of the target event, the terminal device T1 may perform route guidance to the location of the target event. According to one example of the present embodiment, as long as the image 20 is obtained, even if the location of the target event is changed, the location of the changed target event can be tracked.

[0102] Further, for example, similar to the specific example of FIG. 5 , the user U1 may specify a target image (product image) showing a target product as the query 60. In response, the information processing device 5 may search for an image 20 matching the query 60 and provide the search result 65 to the terminal device T1. In addition, in the example of FIG. 6 , the image analysis result 25 may include an extraction result (partial image) of the area showing the product OB1. The information processing device 5 may search for the image analysis result 25 matching the query 60 by comparing the target image specified as the query 60 with the extraction result of the product OB1. In this search, similar to the search for the image 20, the search result 65 related to the target product can be provided to the user U1. In one example, if the search result 65 includes at least one of the posture measurement result 39 and the estimated position of the target product, the terminal device T1 may perform route guidance to the location of the target product.

[0103] Also, for example, similar to the specific example of FIG. 5 above, user U1 may specify an arbitrary position or range of positions within a store as a target posture (query 60). In response, the information processing device 5 may provide search results 65 related to the specified position or range. Additionally, in the example of FIG. 6 , the search results 65 may include image analysis results 25 associated with the extracted posture measurement results 39. The image analysis results 25 may include extraction results (partial images) of the range in which the product OB1 appears. In one example, the terminal device T1 may perform a web search for information on the target product by providing the extraction results of the range in which the target product appears to a search engine.

[0104] According to one example of the present embodiment, by storing the posture measurement result 39 and the image analysis result 25 in association with the image 20 in the storage device MR, it is possible to provide information based on a combination of the posture measurement result 39, the image analysis result 25, and the image 20. For example, as shown in the example of FIG. 6 above, it is possible to provide a search system (information processing device 5) for accessing other elements from at least one of an image, a posture, and an image analysis result. According to one example of the present embodiment, the image analysis includes detection of an object OB appearing in the image 20, thereby providing information about the object OB. According to one example of the present embodiment, the object OB includes a product OB1, thereby providing information about the product OB1. According to one example of the present embodiment, the image analysis includes detection of an event EV appearing in the image 20, thereby providing information about the event EV. According to one example of the present embodiment, the event EV includes a customer event EV1, thereby providing information about the customer event EV1. Furthermore, according to one example of the present embodiment, the event EV includes a product event EV2, thereby providing information about the product event EV2. Note that, as with FIG. 5 , the form of searching the database is not limited to the above example and may be modified as appropriate depending on the embodiment. In another example, the information processing device 5 may be operated directly to execute a database search without going through the terminal device T1. Furthermore, the terminal device T1 may execute a database search by directly accessing the storage device MR without going through the information processing device 5.

[0105] (others) The form of information provision does not have to be limited to the above search example, and may be changed as appropriate depending on the embodiment. That is, the system for providing information does not have to be limited to the above search system. At least a portion of the accumulated data (image 20, posture measurement result 39, image analysis result 25, and image capture time TT) may be utilized for any purpose other than search. The information processing device 5 may execute predetermined information processing utilizing the accumulated data together with or instead of the above search processing.

[0106] (1) Provided data FIG. 7 schematically illustrates an example of provided data according to this embodiment. In one example, the information processing device 5 may output at least a portion of the accumulated data as is. The information processing device 5 may generate provided data by processing at least a portion of the accumulated data and output the generated provided data. In FIG. 7, as an example of processing, a situation is assumed in which a point PT related to an image capture is plotted on a map MP of the environment. The map MP may be obtained as appropriate. For example, when a mobile object MB is operated within a store, the map MP may be obtained from floor information such as a floor map or floor guide. The plotted point PT may be the acquisition point of the image 20 (posture measurement result 39) or a point obtained as a result 25 of image analysis (such as the estimated position of an object OB or the estimated position of an event EV). The map MP including each plotted point PT may be output as an example of provided data.

[0107] When the posture measurement result 39 is used as the point PT, the point where the image 20 was acquired (the range where the image has been captured) can be viewed from above based on the output map MP. When the estimated position of the object OB is used as the point PT, at least one of the position of each object OB and the degree of detection of the object OB in each area can be confirmed based on the output map MP. In one example, if a mobile object MB is operated in a store and an object OB is a product, a product map (a map showing the location of the product) can be generated by plotting the estimated location of the product on the map MP. Even if the location of the product is changed, a product map showing the new location of the product can be obtained by using the image analysis results 25 of the image 20 collected after the change. This makes it possible to track changes to the product. Furthermore, if the estimated location of the event EV is used as the point PT, the occurrence location of each event EV can be confirmed based on the output map MP. In one example, if the event EV includes a customer event EV1, the estimated location of the event EV may be the estimated location of the detected customer. By plotting this estimated location of the customer as the point PT, it is possible to confirm the degree of customer presence in each area of ​​the map MP. For example, it is possible to confirm areas with a large number of customers, areas with a small number of customers, etc.

[0108] In one example, the information processing device 5 may plot data belonging to a target movement unit on the map MP. For example, when a movement unit is defined as one voyage, the information processing device 5 may plot data obtained in one voyage on the map MP. This makes it possible to check environmental information obtained during one voyage (such as the point where the image 20 was acquired, the position of the object OB, and the position of the event EV).

[0109] (2) Real-time analysis In another example, the accumulated data may be used to analyze the environment in real time. For example, when image analysis including detection of an event EV is performed, the information processing device 5 may refer to the acquired image analysis result 25 in real time and detect the occurrence of a specific event. In response to detecting the occurrence of the specific event, the information processing device 5 may output a notification regarding the occurrence of the specific event.

[0110] As a specific example, if the image analysis result 25 includes a detection result of product event EV2, the information processing device 5 may detect the occurrence of a specific product-related event by referring to the acquired detection result of product event EV2. The specific product-related event to be detected may include, for example, a low remaining quantity of the product, a product being out of stock, or a product being present only at the back of a display shelf. The remaining quantity of the product may be evaluated appropriately based on a predetermined criterion. For example, the information processing device 5 may evaluate that the remaining quantity of the product is low if the remaining quantity of the product is equal to or less than a threshold value. The arrangement of the product on the display shelf may also be evaluated appropriately based on a predetermined criterion. For example, the arrangement of the product on the display shelf may be expressed by the distance from the front edge of the display shelf to the product. If there are no products for which this distance is equal to or less than a threshold value (i.e., there are no products within a predetermined range extending from the front edge of the product shelf toward the back of the product shelf), the information processing device 5 may evaluate that the product is present only at the back of the display shelf. The information processing device 5 may evaluate that a product is present only at the back of the display shelf even when the number of products present within the specified range of the product shelf is small. Each threshold value may be set arbitrarily. The information processing device 5 may appropriately output a notification regarding the detected specific event in response to detecting the occurrence of a specific product-related event. For example, when it detects that the remaining number of products is low or that a product is out of stock, the information processing device 5 may output a notification to a store clerk instructing them to replenish the product on the display shelf. The information processing device 5 may also output a notification to a store clerk instructing them to purchase more products. When it detects that a product is present only at the back of the display shelf, the information processing device 5 may output a notification to a store clerk instructing them to move the product forward.

[0111] Furthermore, if the image analysis result 25 includes a facility event detection result, the information processing device 5 may identify the facility usage status by referring to the acquired facility event detection result. The information processing device 5 may output the identified facility usage status as appropriate. For example, the information processing device 5 may output the remaining number of items in a shopping basket, the remaining number of items in a shopping cart, the availability of lockers, etc. The equipment usage status may also be output.

[0112] (3) Post-mortem analysis In another example, the accumulated data may be used to analyze the environment after the fact. For example, if the image analysis result 25 includes a detection result of an event EV, the information processing device 5 may analyze the characteristics of the event EV occurring in the environment based on the accumulated image analysis result 25. The information processing device 5 may output the analysis result as appropriate.

[0113] As a specific example, if the event EV includes a customer event EV1, the information processing device 5 may analyze customer behavior from the image analysis result 25 (detection result of customer event EV1). Customer behavior is an example of a feature of an event EV. For example, the information processing device 5 may identify a location where a large number of customers are present by referring to the detection result of customer event EV1. The number of customers may be evaluated appropriately based on a predetermined criterion. For example, the information processing device 5 may evaluate a location where a large number of customers are present if the number of customers exceeds or is equal to or greater than a threshold. The information processing device 5 may output the identified location where a large number of customers are present. This allows the location where a large number of customers are present to be identified. The information processing device 5 may also output an image 20 of the identified location. The reason for the large number of customers can be analyzed based on the output image. The location where a large number of customers are present may be the posture measurement result 39 associated with the detection result of customer event EV1 evaluated as having a large number of customers. The image 20 of the identified location may be the image 20 associated with the detection result of customer event EV1 evaluated as having a large number of customers. Furthermore, for example, the information processing device 5 may refer to the detection result of the customer event EV1 to determine whether or not a customer is looking at a specific object (product, sign, advertisement, etc.). The information processing device 5 may output the result of determining whether or not a specific object is being looked at. Locations where there are many customers and whether or not a specific object is being looked at are examples of the analysis results of customer behavior. The analysis results of customer behavior may be used for any purpose, such as store design (layout of display shelves, placement of products / signs / advertisements, etc.).

[0114] Furthermore, if the event EV includes at least one of a customer event EV1, a product event EV2, a store clerk event, and a promotional event, the information processing device 5 may collect the detection results of the promotional features by referring to the image analysis results 25 (detection results of the event EV). The promotional features may include, for example, the status of the promotional activities by the store clerk, the status of the promotional location, the behavior of customers at the promotional location, the status of the promotional products, etc. The information processing device 5 may output the collected detection results of the promotional features. The promotional measures can be evaluated based on the output detection results of the promotional features. This makes it possible to consider improvements to the promotional measures.

[0115] Also, assume a situation in which a mobile MB is operated within a store, a target product is sold at multiple locations, and the mobile MB moves through each location. In this situation, the information processing device 5 may extract images 20 showing each location by searching a database using information about the target product as a query 60. The information about the target product may be, for example, the product name, product image, product attribute information, etc. The information processing device 5 may output each extracted image 20. By comparing the output images 20, it is possible to analyze the differences between the locations selling the target product. For example, it is possible to identify a location with a large number of customers from among multiple locations selling the target product.

[0116] Each of the above analyses may be performed for each target period (time period, date, day of the week, etc.). This allows for analyzing differences in the environment for each period. For example, by analyzing the customer behavior for each time period (morning, noon, and night), it is possible to analyze differences in customer behavior for each time period.

[0117] In addition, in one example, the information processing device 5 provides a list of detection results of the events EV and a prompt including an analysis instruction to the large-scale generative model, thereby detecting the characteristics of the events EV occurring in the environment. The features may be analyzed by a large-scale generative model. The large-scale generative model may include, for example, a large-scale language model (LLM), a large-scale visual language model (VLM), a large-scale speech model, etc. The information processing device 5 may output the analysis results obtained from the large-scale generative model as appropriate.

[0118] (4) Data linkage In addition, in one example, the accumulated data may be utilized in conjunction with other data. For example, when the imaging device 2 is operated in a store, the accumulated data may be utilized in conjunction with the purchase data of the target store. Accordingly, the information processing device 5 may refer to either the accumulated data or the purchase data, and vice versa. The purchase data may be known POS (Point Of Sale) data. The purchase data may be collected by a known method and stored in any storage area. This allows the purchase data to be referenced as needed.

[0119] As a specific example, the information processing device 5 may identify a time period during which sales of a target product are good by analyzing purchase data. The degree of sales may be evaluated appropriately based on predetermined criteria. For example, the information processing device 5 may evaluate the sales of the target product as good if the total sales volume of the target product during the target period exceeds a first threshold or is equal to or greater than the first threshold. The information processing device 5 may evaluate the sales of the target product as poor if the total sales volume of the target product during the target period is equal to or less than a second threshold. The second threshold may be the same as the first threshold or may be smaller than the first threshold. The first threshold and the second threshold may each be set arbitrarily. The information processing device 5 may extract images 20 depicting the target product during the specified time period by searching a database using information related to the specified time period and the target product as a query 60. The information processing device 5 may output the extracted images 20. Based on the output images 20, the status of the time period during which sales of the target product are good can be analyzed.

[0120] The information processing device 5 may identify good and bad sales days for the target product by analyzing the purchase data. The information processing device 5 may extract images 20 that depict the target product on the identified day by searching a database using information about the identified day and the target product as a query 60. The information processing device 5 may output the extracted images 20. By comparing the images 20 from the good sales days and the images 20 from the bad sales days, the reason for fluctuations in sales of the target product can be analyzed.

[0121] The information processing device 5 may analyze the purchase data to identify days or time periods when sales of the target product are poor. The information processing device 5 may search a database using information related to the identified day or time period and the target product as a query 60, and extract at least one of an image 20 depicting the target product on the identified day or time period and a result of image analysis 25. The information processing device 5 may output at least one of the extracted image 20 and the result of image analysis 25. For example, based on the fact that the remaining quantity of the target product depicted in the image 20 on a day when sales are poor is low, the remaining quantity of the target product analyzed as product event EV2 is low, or the target product is detected as being out of stock as product event EV2, it can be analyzed that the reason for the decline in sales of the target product is due to inappropriate replenishment of the target product.

[0122] The information processing device 5 may detect a characteristic period in the store by analyzing at least one of the accumulated images 20 and the image analysis results 25. For example, the information processing device 5 may identify a time when there are many customers (image capture time TT) from at least one of the accumulated images 20 and the image analysis results 25 (customer event EV1, etc.). The number of customers may be evaluated appropriately based on a predetermined criterion. For example, the information processing device 5 may evaluate that there are many customers when the number of customers captured in the images 20 or the number of customers detected as the image analysis results 25 exceeds or is equal to or greater than a threshold. The threshold may be set arbitrarily. The information processing device 5 may detect a specific period (time period, day, The information processing device 5 may extract purchasing data for a specific period when there are many customers (e.g., whether a specific product is selling well) based on the extracted purchasing data.

[0123] The information processing device 5 may identify the time (image capture time TT) when the remaining quantity of the target product is low or when the target product is out of stock by analyzing at least one of the accumulated images 20 and the image analysis results 25. The remaining quantity of the target product may be evaluated appropriately based on predetermined criteria. For example, the information processing device 5 may evaluate that the remaining quantity of the target product is low if the remaining quantity of the target product is equal to or less than a threshold value. The threshold value may be set arbitrarily. The information processing device 5 may extract purchase data for a specific period including the identified time from the accumulated purchase data. The information processing device 5 may verify, based on the extracted purchase data, any adverse effects caused by the target product not being replenished. For example, if it is verified from the purchase data that the target product is selling well during a specific period, the information processing device 5 may evaluate that the remaining quantity of the target product is low or the target product is out of stock simply because the target product is selling well (i.e., no adverse effects have occurred), and output the evaluation result. On the other hand, if it is verified from the purchase data that sales of the target product are poor during a specific period, the information processing device 5 may evaluate that an adverse effect is occurring due to the target product not being replenished, and output the evaluation result. In one example, if it is evaluated that an adverse effect is occurring, the information processing device 5 may generate a stocking schedule for the target product so as to make it less likely to cause an adverse effect, for example, by increasing the amount of stocking or the number of stockings, and output the generated schedule.

[0124] (5) Supplementary information The destination of the various pieces of information (1) to (4) output by the information processing device 5 may be selected appropriately depending on the embodiment. The output destination may be, for example, the memory resources of the information processing device 5, an output device of the information processing device 5 (output device 55 described later), another computer, an external storage device, etc. Furthermore, the entity that executes the information processing regarding each piece of information provision described above does not have to be limited to the information processing device 5. That is, at least a part of the information processing by the information processing device 5 may be executed on a computer other than the information processing device 5. The other computer may be, for example, the control device 1, another information processing device 45, or terminal device T1.

[0125] [Correction of measurement results] FIG. 8 schematically shows an example of a scene in which an image 20 and an orientation measurement result 39 according to this embodiment are associated. In one example, the orientation measurement result 39 associated with the image 20 may be the orientation measurement result 30 obtained by the sensor 3. In another example, the orientation measurement result 30 obtained by the sensor 3 may be corrected so as to satisfy a constraint 70 of a movement condition that is given in advance to the moving body MB. The orientation measurement result 39 associated with the image 20 may be the corrected measurement result 32 obtained thereby. In other words, the orientation measurement result 39 associated with the image 20 may be obtained by correcting the measurement result 30 so as to satisfy the constraint 70 of a movement condition that is given in advance to the moving body MB.

[0126] The correction process of the measurement result 30 may be executed by any computer. In one example, the control device 1 may correct the posture measurement result 30 so as to satisfy a predetermined movement condition constraint 70. The movement condition constraint 70 may be acquired as appropriate depending on the embodiment. For example, if the map information 125 includes constraint information, at least a part of the movement condition constraint 70 may be acquired from the map information 125. At least a part of the movement condition constraint 70 may be acquired from any information source other than the map information 125, for example, being given, specified in a program, provided by an operator, or stored as individual data. This correction process allows the control device 1 to acquire the corrected measurement result 32. The control device 1 may store the corrected measurement result 32 in association with the image 20. In one example, by continuously acquiring the posture measurement result 30, a movement trajectory showing the progress of posture over time may be acquired. In this case, correcting the posture measurement result 30 may be correcting at least a part of the movement trajectory. The control device 1 may store the corrected movement trajectory in association with one or more images 20 (image group).

[0127] In the following embodiment, for convenience of explanation, the control device 1 will be described as executing the correction process. However, the computer that executes the correction process is not limited to the control device 1 and may be changed as appropriate depending on the embodiment. In another example, the information processing device 5 or another information processing device 45 may correct the posture measurement result 30 to satisfy the movement condition constraint 70 and store the corrected measurement result 32 in association with the image 20. When the image analysis result 25 is further associated with the image 20, the computer that executes the image analysis and the computer that executes the correction process may be the same or may be at least partially different. According to one example of the present embodiment, by adopting the corrected measurement result 32 as the measurement result 39, it is possible to ensure the accuracy of the posture measurement result 39 associated with the image 20.

[0128] (Correction method) The movement condition constraints 70 may be configured by some kind of restriction (condition) imposed when the moving body MB moves. The movement condition constraints 70 may be determined appropriately depending on the embodiment as long as the posture measurement results 30 can be reliably corrected. In one example, the movement condition constraints 70 may be defined depending on the task of the moving body MB. The control device 1 may correct the measurement results 30 so as to satisfy the movement condition constraints 70 while suppressing changes in the characteristics (speed information, etc.) of the measurement results 30. Any optimization method may be used as a method for correcting the measurement results 30.

[0129] When the sensor 3 is an inertial sensor 3S, errors in the attitude measurement result 30 obtained by the inertial sensor 3S can be caused mainly by the following two factors. The first factor is the possibility of calculation errors, such as noise from the inertial sensor 3S and estimation errors of the measurement value. The second factor is that the measurement value obtained by the inertial sensor 3S is a relative value, and therefore may deviate from the absolute value in the coordinate system of real space (for example, at least one of the direction and the scale may deviate). In one example, in order to reduce errors caused by either of these two factors, the movement condition constraint 70 and the correction method may have the following configuration.

[0130] (1) First method (loop closure) FIG. 9 schematically shows an example of a first method for correcting the posture measurement result 30 (movement trajectory 300) according to this embodiment.

[0131] In one example, the movement condition constraint 70 may include the moving body MB taking the same attitude at multiple times during the operation period of the imaging device 2. Taking the same attitude may include being located at the same place. Taking the same attitude at multiple times may include the attitude at each time being completely the same and the attitude at each time being substantially the same. The range of substantially the same attitude may be defined as appropriate depending on the embodiment.

[0132] During the operation of the imaging device 2, the movement trajectory 300 may be acquired by continuously acquiring the attitude measurement result 30 by the inertial sensor 3S. Correcting the attitude measurement result 30 may be correcting the movement trajectory 300. Since the imaging device 2 is attached to the moving body MB, when the moving body MB assumes the same attitude, the imaging device 2 also assumes the same attitude at the relevant time. Therefore, correcting the movement trajectory 300 may include correcting the movement trajectory 300 so that the measured attitudes (measurement results 30) at each of multiple times on the movement trajectory 300 are consistent. The measured attitude may include the measurement position. This correction process makes it possible to obtain a corrected movement trajectory 320 (measurement result 32).

[0133] The time at which the same posture is assumed may be determined as appropriate depending on the embodiment. may be determined according to the task of the moving body MB. The time at which the moving body MB assumes the same attitude may be determined based on measurement data of the inertial sensor 3S. The time at which the moving body MB assumes the same attitude may be given in advance. The same attitude may be a known attitude, such as a charging attitude at a charging point, or may be an unknown attitude.

[0134] In one example, the moving body MB may be a robotic device MB111 that moves autonomously within a store, among the autonomously moving robotic devices MB11. The same location may be a charging point CP of the robotic device MB111. That is, the same posture may be a charging posture at the charging point CP. The charging point CP may be a location of a charging station installed in the store. For example, the movement condition may be set so that the movement starts from the charging point CP and ends upon returning to the charging point CP. In this case, by detecting the start and end of the movement of the moving body MB using the detection method based on the measurement data described above, it is possible to identify multiple times (start times and end times of movement) when the moving body MB takes the same posture.

[0135] 9, a scene is assumed in which movement starts from a charging point CP and ends by returning to the charging point CP. Therefore, the posture P1 (start position) at the start time of the movement and the posture P2 (end position) at the end time are identified as the same posture (same position). In this case, as shown in FIG. 9, the movement trajectory 300 may be corrected so that the posture P1 at the start time of the movement and the posture P2 at the end time are the same.

[0136] A loop closure calculation method may be used as a calculation method for correcting the measured postures at multiple times so that they coincide with each other. In one example, when the posture is expressed in two dimensions, the measured values ​​of the posture at each time (1 to T) are expressed as "p t =(x t ,y t ) t=1, ,T". If we assume that the measured values ​​of the posture at each time are arbitrarily converted, the measured values ​​of the posture after conversion can be expressed as "q a_t =F(p t ,θ t )". The function F may be defined arbitrarily. For example, the function F can be expressed as "F(p t ,θ t )=p t +θ t " In this case, θ t are parameters of the function F, and are unknown parameters to be optimized. The function F may also be defined as in Equation 1 below.

[0137]

number

[0138] According to one example of this embodiment, by performing correction using the first method, it is expected that calculation errors will be reduced. Also, in one example of this embodiment, the same posture may be the charging posture at the charging point CP. In this way, by utilizing the movement condition that the robot device MB111 passes through the charging point CP, it is expected that the time when the robot device MB111 assumes the same posture will be appropriately identified. As a result, a proper correction of the movement trajectory 300 can be expected.

[0139] (2-1) Method 2-1 (Map Matching) FIG. 10 schematically shows an example of the 2-1 method for correcting the posture measurement result 30 (movement trajectory 300) according to this embodiment.

[0140] In one example, the movement condition constraint 70 may include limiting the movement of the moving object MB within a predetermined movement range. Correcting the movement trajectory 300 may include correcting the movement trajectory 300 so that it fits the shape of the movement range. That is, the movement trajectory 300 may be corrected so that the shape of the actual movement range in the movement trajectory 300 fits the shape of a predetermined (assumed) movement range. This correction process makes it possible to obtain a corrected movement trajectory 321 (measurement result 32).

[0141] The movement range may be a range within the environment in which the moving object MB is expected to pass. The movement range may be defined as appropriate depending on the embodiment. The movement range may be defined at least partially automatically based on information obtained from the environment, or may be defined manually.

[0142] In one example, the mobile body MB may be a robotic device MB111 that autonomously moves within a store. The movement range given as the constraint 70 may be defined within the floor FR of the store. For example, a floor map or floor guide of the store may be provided as appropriate. The floor map or floor guide may define a movable range within the floor FR (such as a range within which customers can pass) and a non-movable range (such as the range of product shelves). In this case, the movable range defined in the floor map or floor guide may be adopted as the movement range as is. The movement range may be automatically defined according to the task of the mobile body MB. For example, if the mobile body MB (robotic device MB111) is a cleaning robot, the cleaning range of the cleaning robot within the floor FR of the store may be defined in advance. The cleaning range may be defined by a known method. In this case, the given cleaning range may be adopted as the movement range as is. The movement range may also be manually defined within the floor FR of the store. The movement range may be defined as the entire floor FR of the store, or as a partial area within the floor FR. For example, if the area around the cash register is prone to congestion and the movement of the mobile object MB is restricted so that it does not enter this area around the cash register, the movement range may be defined as the area within the store floor FR excluding the area around the cash register.

[0143] In the example of FIG. 10 , an outer edge RE1 and an inner edge RE2 are provided within a store floor FR, walls or shelves exist at the outer edge RE1 and the inner edge RE2, and a situation is assumed in which movement is possible between the outer edge RE1 and the inner edge RE2 as an aisle. In this situation, the space between the outer edge RE1 and the inner edge RE2 may be set as a movement range R1. Accordingly, the movement trajectory 300 may be corrected so that the shape of the actual movement range of the movement trajectory 300 matches the shape of the movement range R1. Note that, as illustrated in FIG. 10 , as a result of correction using method 2-1, it may be acceptable for part of the corrected movement trajectory 321 (measurement result 32) to deviate from the given movement range (movement range R1).

[0144] A map matching calculation method may be used as a calculation method for correcting the shape of the given movement range. In one example, when the posture is expressed in two dimensions, the measured values ​​of the posture at each time (1 to T) are expressed as "p t =(x t ,y t ) t=1, ,T”. Within a given moving range (moving range R1), N points (p e_n ) may be randomly sampled. Each point is called "p e_n =(x e_n ,y e_n ) n=1,...,N". If we assume that the measured values ​​of the posture at each time are scaled, rotated, and translated, the measured values ​​of the posture after transformation can be expressed as "q b_t =s b (R b ·p t )+u b " can be expressed as: s b , R b and u b is the scalar for the scale transformation s indicates the value, rotation matrix and translation vector (2D vector). b , R b and u b is the unknown parameter to be optimized. For each point (p e_n ) and the transformed set of measurements (q b_t ) so that the distance between b , R b and u b The parameters of the above may be optimized. The distance between the sets may be calculated using known distance indices such as the Chamfer distance and the Jaccard index. The trajectory 300 may be expressed by a parameter. A known method such as gradient descent may be used as the optimization method. A known optimizer such as Adam may be used for the calculation of the gradient descent method. This calculation allows the movement trajectory 300 to be corrected so as to fit the shape of the given movement range. Note that when the measurement posture is expressed in three dimensions, the above calculation method may be appropriately extended to three dimensions.

[0145] According to one example of this embodiment, by performing correction using method 2-1, it is expected that errors due to deviation from the coordinate system of the real space can be reduced. Also, in one example of this embodiment, the movement range may be defined within the floor FR of the store. In this way, by utilizing the movement condition that the robot device MB111 moves within the store, it is expected that the movement range limit can be appropriately specified. As a result, it is expected that the movement trajectory 300 can be appropriately corrected.

[0146] Note that the 2-1 method may be used together with or instead of the above-described first method. When both the first method and the 2-1 method are employed, either the first method or the 2-1 method may be executed first. When the correction process using the first method is executed first, the correction process using the 2-1 method may be executed on the corrected movement trajectory 320 obtained by the first method. This is expected to result in a movement trajectory 321 (measurement result 32) with further reduced errors. Similarly, when the correction process using the 2-1 method is executed first, the correction process using the first method may be executed on the corrected movement trajectory 321 obtained by the 2-1 method. This is expected to result in a movement trajectory 320 (measurement result 32) with further reduced errors.

[0147] (2-2) Method 2-2 (Particle filter) 11 schematically illustrates an example of the 2-2 method for correcting the posture measurement results (movement trajectory) according to this embodiment. In one example, when the 2-1 method is used for correction, the 2-2 method may be additionally used for correction. The 2-2 method correction process may be performed at any timing after the 2-1 method correction process is performed.

[0148] In one example, when the correction by the 2-2 method is adopted, the movement condition constraints 70 may further include an internal constraint 700 within the movement range. Correcting the movement trajectory 300 (321) may further include correcting the movement trajectory 321 so as to satisfy the internal constraint 700. This correction process can obtain a corrected movement trajectory 322 (measurement result 32).

[0149] The internal constraints 700 may be configured by some restrictions (conditions) imposed on the movement of the moving body MB within the movement range. The internal constraints 700 may be defined appropriately depending on the embodiment. The internal constraints 700 may be defined at least partially automatically based on information obtained from the environment, or may be defined manually.

[0150] In one example, the internal constraint 700 may include requiring the moving body MB not to pass through an obstacle range defined by the presence of an obstacle. Correcting the movement trajectory 321 to satisfy the internal constraint 700 may include correcting the movement trajectory 321 to bypass the obstacle range (i.e., to prevent the moving body MB from passing through the obstacle range). The obstacle may be any object that obstructs the movement of the moving body MB. The type of obstacle may be determined appropriately depending on the embodiment of the moving body MB, the environment, etc. For example, if the moving body MB is a robotic device (robot device MB111) that autonomously moves within a store, the obstacle may include a platform, a shelf (display shelf, etc.), a wall, etc. The obstacle range may be a range in which the entry of the moving body MB is restricted due to the presence of an obstacle. The obstacle range may be statically or dynamically defined. The obstacle range may be defined in advance or identified by observing the environment. For example, an obstacle present in the environment may be detected by observing the environment using a sensor such as an imaging device. The obstacle range may be appropriately identified from the location where the obstacle is detected. In one example, if the moving body MB is present inside a store, the obstacle range may be set inside the store.

[0151] 11, similar to FIG. 10, a situation is assumed in which walls or shelves exist at the outer edge RE1 and the inner edge RE2, and the space between the outer edge RE1 and the inner edge RE2 is set as the movement range R1. In this situation, the space outside the outer edge RE1 and the space inside the inner edge RE2 may be set as obstacle ranges (R2, R3). Accordingly, the movement trajectory 321 may be corrected to avoid the obstacle ranges (R2, R3).

[0152] The internal constraint 700 does not have to be limited to the above-described obstacle range constraint. The internal constraint 700 may include other constraints in addition to or instead of the obstacle range constraint. For example, the other constraints may include a no-entry range that prohibits the entry of a moving object MB. The no-entry range may be a range through which the moving object MB can pass but into which the moving object MB is prohibited. The no-entry range may be defined at least partially automatically based on information obtained from the environment, or may be defined manually. For example, an area near a cash register may be defined as a no-entry range because it is prone to congestion. The no-entry range may be treated in the same way as the above-described obstacle range. In other words, correcting the movement trajectory 321 to satisfy the internal constraint 700 may include correcting the movement trajectory 321 to detour around the no-entry range.

[0153] The obstacle range and the no-entry range are ranges that impose a restriction that the moving body MB does not pass through. Therefore, in another example, at least one of the obstacle range and the no-entry range may be included in the definition of the movement range by being excluded from the movement range rather than being given as the internal constraint 700. Furthermore, the internal constraint 700 does not have to be limited to constraints that impose a no-passage range, such as the obstacle range and the no-entry range, and may include constraints that impose restrictions (conditions) other than the no-passage range, such as a range where passage is unlikely or a range where passage is likely.

[0154] A particle filter calculation method may be used as a calculation method for correcting the trajectory 321 so as to satisfy the internal constraint 700. In one example, when the orientation is expressed in two dimensions, the measured values ​​of the orientation at each time (1 to T) on the trajectory 321 are expressed as "p c_t =(xc_t ,y c_t ) t=1, ,T”. The amount of movement at each time on this movement trajectory 321 can be expressed as “v c_t =p c_t+1 -p c_t " can be expressed as ". The measurement posture of interest is p c_t Assuming that, p c_t +v c_t A plurality of candidate points of movement may be sampled from the surroundings of the target point. Among the sampled candidate points, (A) the candidate point with the movement amount v c_t and (B) satisfying the internal constraint 700. The extracted candidate points are then used to determine the next measurement posture p c_t+1 In one example, satisfying the internal constraint 700 may include not entering an obstacle range (not colliding with an obstacle). c_t+1 Similarly, by sampling multiple candidate points and extracting candidate points that satisfy the above conditions (A) and (B), the next measurement posture p c_t+2 By repeating this calculation from the start time to the end time of the movement, the movement trajectory 321 can be corrected so as to satisfy the internal constraint 700 (to avoid the obstacle range). Note that when the measured posture is expressed in three dimensions, the above calculation method may be appropriately extended to three dimensions.

[0155] According to an example of this embodiment, after reducing errors due to deviation from the coordinate system of the real space by the correction by the 2-1 method, correction by the 2-2 method is performed, thereby further reducing errors in calculation. As a result, a movement trajectory 322 with reduced errors can be obtained. In one example of the present embodiment, the internal constraint 700 may include a condition that the moving object MB does not pass through an obstacle range. By using information about the range where the obstacle exists (obstacle range) as the internal constraint 700, it is possible to expect appropriate correction of the movement trajectory 321.

[0156] Note that the 2-1 method and the 2-2 method may be used together with or instead of the above-described 1st method. When all of the 1st method, the 2-1 method, and the 2-2 method are employed, the correction process by the 1st method may be performed before the correction process by the 2-1 method, between the correction processes by the 2-1 method and the 2-2 method, or after the correction process by the 2-2 method. When the correction process by the 1st method is performed after the correction process by the 2-1 method, and when the correction process by the 2-2 method is performed after the correction process by the 1st method, the correction process by the 2-2 method may be performed on the corrected movement trajectory 320 obtained by the 1st method. This can be expected to result in a movement trajectory 322 (measurement result 32) with further reduced error. Similarly, when the correction process by the 1st method is performed after the correction process by the 2-2 method, the correction process by the 1st method may be performed on the corrected movement trajectory 322 obtained by the correction process by the 2-2 method. This is expected to result in a movement trajectory 320 (measurement result 32) with further reduced errors.

[0157] (3) Other The movement condition constraints 70 and the method of correcting the measurement results 30 are not limited to the above examples and may be changed as appropriate depending on the embodiment. If the sensor 3 is configured with a sensor other than the inertial sensor 3S (such as a positioning module or a communication module), known techniques may be used for the movement condition constraints 70 and the method of correcting the measurement results 30.

[0158] (Execution timing) The acquired measurement results 30 may be corrected afterward. In one example, when a history of a correctable amount of measurement results 30 has been accumulated, the control device 1 may perform a correction process on the measurement results 30 (movement trajectory) accumulated up to that point. The control device 1 may periodically perform a correction process on the accumulated measurement results 30 (movement trajectory). For example, the control device 1 may perform a correction process on the accumulated measurement results 30 (movement trajectory) for each unit of movement. As a specific example, when a unit of navigation is defined, such as starting movement from a specific point and returning to a specific point, the control device 1 may perform a correction process on the accumulated measurement results 30 (movement trajectory) for each navigation.

[0159] Note that, when the sensor 3 is an inertial sensor 3S, the attitude measurement result 30 is obtained as a relative value from the initial attitude (initial value). The measurement result 30 at each time is derived from the difference with the attitude at a past time. In other words, the measurement result 30 at each time is derived based on the measurement result 30 at the past time. In one example, the control device 1 may perform a correction process on the measurement results 30 (movement trajectory) accumulated up to a certain time (hereinafter also referred to as the "correction time"), and then the attitude measurement by the inertial sensor 3S may continue. The attitude measurement result 30 newly obtained by the inertial sensor 3S after the correction time may be accumulated following the corrected measurement results 32 up to the correction time. In other words, the attitude measurement result 30 after the correction time may be derived based on the corrected measurement results 32 up to the correction time. While the inertial sensor 3S continues to measure the attitude, there may be a temporary timing when at least a part of the measurement results collected by the inertial sensor 3S is made up of the corrected measurement results 32, and the rest of the collected measurement results are made up of the measurement results 30. At this timing, the control device 1 may execute a series of information processes related to the image capture, including a process of determining whether the target range 35 has been imaged. According to one example of this embodiment, the attitude measurement results 30 (32) up to the correction time are corrected, so that the accuracy of the attitude measurement results 30 after the correction time can be expected to improve. As a result, by appropriately determining whether the target range 35 has been imaged, it is possible to determine whether the target range 35 has been imaged or not. Therefore, efficient collection of images 20 can be expected.

[0160] §2 Configuration example [Hardware configuration] (Control device) 12 is a diagram illustrating an example of a hardware configuration of the control device 1 according to the present embodiment. In one example, the control device 1 may be configured as a computer to which a control unit 11, a storage unit 12, an external interface 13, an input device 14, and an output device 15 are electrically connected.

[0161] The control unit 11 is configured to execute information processing based on programs and various data. For example, the control unit 11 includes a hardware processor such as a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory). The control unit 11 (CPU) is an example of a processor resource of the control device 1.

[0162] The storage unit 12 is configured to hold any data. For example, the storage unit 12 may include a hard disk drive, a solid state drive, a semiconductor memory, etc. The storage unit 12, RAM, and ROM are examples of memory resources of the control device 1. In one example, the storage unit 12 may store various types of information such as a program 81 and map information 125.

[0163] The program 81 is a program for causing the control device 1 to execute information processing related to the control of the imaging device 2 (FIGS. 16 and 17 described below). The program 81 includes a series of instructions for this information processing. In one example, the program 81 may further include instructions for information processing other than the information processing related to the control of the imaging device 2, such as correction processing of the measurement results 30 (FIG. 18 described below) and information processing related to image analysis (FIG. 19 described below). In one example, at least a portion of the storage device MR may be configured by the storage unit 12. In this case, at least a portion of the accumulated data (images 20, posture measurement results 39, imaging time TT, and image analysis results 25) may be stored in the storage unit 12.

[0164] In one example, at least one of the program 81 and the map information 125 may be stored in a storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to store various information (such as the stored program) by electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The storage unit 12 and the storage medium 91 are examples of non-transitory storage media. The control device 1 may acquire at least one of the program 81 and the map information 125 from the storage medium 91. The storage medium 91 may be a disk-type storage medium (such as a CD or DVD) or a non-disk type storage medium such as a semiconductor memory (such as a flash memory). Any drive device may be used to read the information stored in the storage medium 91. The type of drive device may be selected depending on the storage medium 91. The drive device may be connected to the control device 1 in any manner. The storage medium 91 may include an external storage device. At least a portion of the accumulated data (images 20, posture measurement results 39, image capture times TT, and image analysis results 25) may be stored in a storage medium 91.

[0165] The external interface 13 is configured to connect to an external device via a wired or wireless connection. The external interface 13 may include, for example, a USB (Universal Serial Bus) port, a dedicated port, a communication port (communication module), etc. The type and number of external interfaces 13 may be determined appropriately depending on the embodiment. When the external interface 13 includes a communication port (communication module), the standard of the communication network may be selected arbitrarily. The communication standard may be selected appropriately from, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, etc. In one example, the control device 1 may be connected to the imaging device 2 and the sensor 3 via the external interface 13. In one example, when a communication module is used for the sensor 3, 3 may be configured by a communication port (communication module) of the external interface 13. In one example, when at least a part of the storage device MR is configured by a storage area other than the memory resources of the control device 1, the control device 1 may be connected to the storage device MR via the external interface 13.

[0166] The input device 14 is configured to accept input of information. The input device 14 may be configured, for example, by an imaging device, a microphone, a mouse, a keyboard, a touch panel, an operator, etc. The imaging device of the input device 14 may be the same as or different from that of the imaging device 2. The output device 15 is configured to output information. The output device 15 may be configured, for example, by a display, a speaker, etc. The control device 1 may be operated using the input device 14 and the output device 15. The input device 14 and the output device 15 may be directly connected to the control device 1 or indirectly connected via the external interface 13. The input device 14 and the output device 15 may be at least partially integrated with a touch panel display, etc.

[0167] Note that, with regard to the specific hardware configuration of the control device 1, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or the like. At least one of the external interface 13, the input device 14, and the output device 15 may be omitted. At least one of the program 81 and the map information 125 may be stored in an external storage device such as a NAS. An external storage device is also an example of a non-transitory storage medium. The control device 1 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. The control device 1 may be a computer designed specifically for the service provided, as well as a general-purpose server device, a general-purpose PC, a laptop PC, a terminal device, or the like. In one example, the control device 1 may be configured as a terminal device UT by including an imaging device 2 and a sensor 3.

[0168] (Information processing device) 13 is a diagram illustrating an example of a hardware configuration of the information processing device 5 according to this embodiment. The information processing device 5 according to this embodiment may be configured as a computer in which a control unit 51, a storage unit 52, an external interface 53, an input device 54, and an output device 55 are electrically connected.

[0169] The control unit 51 to the output device 55 and the storage medium 95 of the information processing device 5 may be configured similarly to the control unit 11 to the output device 15 and the storage medium 91 of the control device 1, respectively. The control unit 51 (CPU) is an example of a processor resource of the information processing device 5. The storage unit 52 (and RAM, ROM) is an example of a memory resource of the information processing device 5. In this embodiment, the storage unit 52 stores various information such as a program 85.

[0170] The program 85 is a program for causing the information processing device 5 to execute information processing related to database search (FIG. 20 described later). The program 85 includes a series of instructions for this information processing. In one example, the program 85 may further include instructions for information processing other than information processing related to database search, such as correction processing of the measurement results 30 (FIG. 18 described later) and information processing related to image analysis (FIG. 19 described later). In one example, at least a portion of the storage device MR may be configured by the storage unit 52. In this case, at least a portion of the accumulated data (images 20, posture measurement results 39, imaging times TT, and image analysis results 25) may be stored in the storage unit 52.

[0171] In one example, the program 85 may be stored in a storage medium 95 instead of or together with the storage unit 52. The information processing device 5 may acquire the program 85 from the storage medium 95. The program 85 may be saved in an external storage device. At least a portion of the accumulated data (images 20, posture measurement results 39, imaging times TT, and image analysis results 25) may be stored in the storage medium 95.

[0172] The information processing device 5 may perform data communication with other computers (such as terminal device T1) via an external interface 53 (communication module). In one example, when at least a part of the storage device MR is configured with a storage area other than the memory resources of the information processing device 5, the information processing device 5 may be connected to the storage device MR via the external interface 53. The information processing device 5 may be operated using an input device 54 and an output device 55.

[0173] Note that, with regard to the specific hardware configuration of the information processing device 5, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 51 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA, DSP, GPU, ASIC, etc. At least one of the external interface 53, the input device 54, and the output device 55 may be omitted. The information processing device 5 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. The information processing device 5 may be a computer designed specifically for the service provided, as well as a general-purpose server device, a general-purpose PC, a laptop PC, a terminal device, etc.

[0174] [Software configuration] (Control device) 14 schematically shows an example of the software configuration of the control device 1 according to this embodiment. The control unit 11 of the control device 1 executes instructions included in the program 81 stored in the storage unit 12 using the CPU. As a result, the control device 1 operates as a computer including an acquisition unit 111, a determination unit 112, and an image capture processing unit 113 as software modules. That is, in one example, each software module of the control device 1 may be realized by the control unit 11 (CPU).

[0175] The acquisition unit 111 is configured to acquire a measurement result 30 of the attitude of the imaging device 2 by the sensor 3. The determination unit 112 is configured to determine, based on the stored map information 125, whether or not an image of a target range 35 toward which the imaging device 2 is facing, which is identified from the acquired attitude measurement result 30, has been captured. The imaging processing unit 113 is configured to execute imaging processing by the imaging device 2 when it is determined that an image of the target range 35 has not been captured. On the other hand, the imaging processing unit 113 is configured to omit execution of imaging processing by the imaging device 2 when it is determined that an image of the target range 35 has been captured.

[0176] In one example, when a configuration is adopted in which the control device 1 performs correction processing on the measurement results 30, the software configuration of the control device 1 may further include a correction processing unit 114. The correction processing unit 114 may be configured to correct the posture measurement results 30 so as to satisfy a predetermined movement condition constraint 70, and to store the corrected measurement results 32 in association with the image 20. Also, in one example, when a configuration is adopted in which the control device 1 performs image analysis on the image 20, the software configuration of the control device 1 may further include an image analysis unit 115. The image analysis unit 115 may be configured to perform image analysis on the image 20, and to store the obtained image analysis results 25 in association with the image 20.

[0177] (Information processing device) FIG. 15 is a diagram showing an example of the software configuration of the information processing device 5 according to this embodiment. The control unit 51 of the information processing device 5 executes instructions included in the program 85 stored in the storage unit 52 using the CPU. As a result, the information processing device 5 operates as a computer having a reception unit 511, a search unit 512, and a response unit 513 as software modules. That is, in one example, similar to the control device 1, each software module of the information processing device 5 may also be realized by the control unit 51 (CPU).

[0178] The receiving unit 511 is configured to receive a query 60 from the terminal device T1. The searching unit 512 is configured to search a database (storage device MR) to extract elements matching the query 60 from the database. The replying unit 513 is configured to reply to the terminal device T1 with the obtained search result 65.

[0179] In one example, when a configuration is adopted in which the information processing device 5 performs correction processing on the measurement result 30, the software configuration of the information processing device 5 may further include a correction processing unit 514. Also, in one example, when a configuration is adopted in which the information processing device 5 performs image analysis on the image 20, the software configuration of the information processing device 5 may further include an image analysis unit 515. The correction processing unit 514 and the image analysis unit 515 may be configured in the same way as the correction processing unit 114 and the image analysis unit 115.

[0180] (others) In the example of the present embodiment, the software modules of the control device 1 and the information processing device 5 are all implemented by a general-purpose CPU. However, the method of implementing each of the modules is not limited to this example and may be modified as appropriate depending on the embodiment. Some or all of the software modules may be implemented by one or more dedicated processors or chipsets. Each of the modules may be implemented as a hardware module. With regard to the software configurations of the control device 1 and the information processing device 5, modules may be omitted, replaced, or added as appropriate depending on the embodiment. For example, in the software configuration of the control device 1, at least one of the correction processing unit 114 and the image analysis unit 115 may be omitted. In the software configuration of the information processing device 5, at least one of the correction processing unit 514 and the image analysis unit 515 may be omitted.

[0181] §3 Example of operation [Control of imaging device] FIG. 16 is a flowchart showing an example of a processing procedure for controlling the imaging device 2 by the control device 1 according to this embodiment. The following processing procedure is an example of an information processing method (control method) executed by a computer. The following processing procedure is merely an example, and each step may be changed as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0182] (Step S101) In step S101, the control unit 11 operates as the acquisition unit 111 to acquire the measurement result 30 of the attitude of the imaging device 2 by the sensor 3.

[0183] The imaging device 2 is separately attached to a moving object MB. In one example, the moving object MB may be at least one of an autonomously moving robotic device MB11 and a device MB12 that moves by direct or remote manual operation. In another example, the moving object MB may be a living thing MB2. The living thing MB2 may be at least one of a human being MB21 and a living thing MB22 other than a human being MB21.

[0184] In one example, the sensor 3 may be an inertial sensor 3S. In one example, the control unit 11 may obtain a movement trajectory 300 by continuously obtaining a posture measurement result 30. When the momentum measurement result 30 is acquired, the control unit 11 advances the process to the next step S102.

[0185] (Step S102) In step S102, the control unit 11 operates as the determination unit 112 to determine, based on the stored map information 125, whether or not the target range 35 toward which the imaging device 2 is facing, identified from the acquired posture measurement result 30, has been imaged.

[0186] In one example, the control unit 11 may determine whether or not the target range 35 has been imaged in accordance with a predetermined criterion. For example, the control unit 11 may determine whether or not the target range 35 has been imaged in accordance with the discrepancy between the posture measurement result 30 and the captured posture included in the map information 125. Alternatively, for example, the control unit 11 may determine whether or not the target range 35 has been imaged in accordance with the extent to which the captured range (target range 35) in the measured posture (measurement result 30) has not yet been imaged. Upon obtaining the determination result, the control unit 11 proceeds to the next step S103.

[0187] (Step S103) In step S103, the control unit 11 operates as the imaging processing unit 113, and determines where the process should branch depending on the determination result of step S102. If it is determined that the target range 35 has not been imaged, the control unit 11 proceeds to the next step S104. On the other hand, if it is determined that the target range 35 has been imaged, the control unit 11 omits the process of the next step S104. As a result, the control unit 11 omits the execution of the imaging process by the imaging device 2. If the execution of the imaging process is omitted, the control unit 11 ends the processing procedure related to the control of the imaging device 2 according to this operation example.

[0188] (Step S104) In step S104, the control unit 11 operates as the imaging processing unit 113 to cause the imaging device 2 to perform imaging processing.

[0189] Fig. 17 is a flowchart showing an example of a subroutine of the imaging process (step S104) according to this embodiment. The processing procedure in Fig. 17 is merely an example, and each step may be changed as much as possible. Furthermore, steps may be omitted, replaced, or added to the processing procedure in Fig. 17 as appropriate depending on the embodiment. The processing in step S104 may include the following processing in steps S141 to S144.

[0190] In step S141, the control unit 11 starts up the imaging device 2. In step S142, the control unit 11 controls the imaging device 2 to capture an image of the target range 35. In step S143, the control unit 11 stops the activation of the imaging device 2 after capturing an image of the target range 35. In step S144, the control unit 11 updates the map information 125 to indicate that the target range 35 has been captured. The processing order of steps S141 to S144 is not limited to the example of FIG. 17 and may be changed as appropriate depending on the embodiment. For example, the processing of step S144 may be performed before any of the processing of steps S141 to S143.

[0191] As a result of the processing of step S142, an image 20 can be acquired. In one example, the imaging system SY may further include a storage device MR. The acquired image 20 may be stored in the storage device MR at any timing. In one example, the image 20 may be stored in the storage device MR in association with a measurement result 39 of the attitude of the imaging device 2 at the time of capturing the image 20 (imaging attitude). In one example, the attitude measurement result 39 may be the attitude measurement result 30 acquired in step S101.

[0192] When the execution of the imaging process in step S104 is completed, the control unit 11 This completes the processing procedure relating to the control of the device 2. In one example, the control unit 11 may execute a series of processes from step S101 to step S104 in real time.

[0193] (Features) In this embodiment, based on the determination result of step S102, if the target range 35 identified by the sensor 3 has not yet been imaged, the control device 1 controls the operation of the image capture device 2 to execute the image capture process of step S104, whereas if the target range 35 has already been imaged, the control device 1 omits the execution of the image capture process of step S104. This makes it possible to prevent overlapping execution of image capture processes in an imaged range. Therefore, according to this embodiment, the number of times the image capture process is executed can be reduced compared to when the image capture device 2 is constantly capturing images, and therefore, it is expected that the image 20 can be acquired efficiently.

[0194] [Correction processing] Fig. 18 is a flowchart showing an example of a processing procedure for correcting the posture measurement result 30 according to this embodiment. The processing procedure in Fig. 18 is merely an example, and each step may be changed as much as possible. Furthermore, steps in the processing procedure in Fig. 18 may be omitted, replaced, or added as appropriate depending on the embodiment.

[0195] (Step S201) In step S201, the control unit 11 operates as the correction processing unit 114 to correct the posture measurement result 30 so as to satisfy the constraints 70 of the predetermined movement conditions. As a result, the control unit 11 can obtain the corrected measurement result 32.

[0196] In one example, the sensor 3 may be an inertial sensor 3S. A movement trajectory 300 may be acquired by continuously acquiring attitude measurement results 30 by the inertial sensor 3S. The control unit 11 may correct the acquired movement trajectory 300 so as to satisfy a constraint 70 of a movement condition given in advance to the moving body MB.

[0197] In one example, the movement condition constraint 70 may include that the moving object MB assumes the same posture at multiple times during the operation period of the imaging device 2. The control unit 11 may correct the movement trajectories (300, 321, 322) so that the measured postures (measurement results 30, 32) at multiple times on the movement trajectories (300, 321, 322) coincide with each other. This allows a corrected movement trajectory 320 to be obtained. In one example, the moving object MB may be a robotic device MB111 that autonomously moves within a store. The same posture may be the charging posture of the robotic device MB111 at the charging point CP.

[0198] In one example, the movement condition constraint 70 may include limiting the movement of the moving object MB within a predetermined movement range. The control unit 11 may correct the movement trajectory (300, 320) so that it fits the shape of the movement range. This allows a corrected movement trajectory 321 to be obtained. In one example, the moving object MB may be a robotic device MB111 that moves autonomously within a store. The movement range may be defined within a floor FR of the store.

[0199] In one example, the movement condition constraints 70 may further include an internal constraint 700 within the movement range. The control unit 11 may correct the movement trajectory (320, 321) so as to satisfy the internal constraint 700. This allows for a corrected movement trajectory 322 to be obtained. In one example, the internal constraint 700 may include a requirement that the moving object MB not pass through an obstacle range defined by the presence of an obstacle. The control unit 11 may correct the movement trajectory (320, 321) so as to bypass the obstacle range. When the correction process is completed, the control unit 11 proceeds to the next step S202.

[0200] (Step S202) In step S202, the control unit 11 operates as the correction processing unit 114 to associate the corrected measurement result 32 with the image 20 and store it.

[0201] The destination for saving the image 20 and the measurement results 32 may be selected appropriately depending on the embodiment. In one example, when the imaging system SY further includes a storage device MR, the control unit 11 may save the image 20 and the measurement results 32 in the storage device MR. In the storage device MR, the posture measurement results 39 associated with the image 20 may be the corrected measurement results 32 obtained by the processing of step S201. In one example, the corrected posture measurement results 32 may be saved in the storage device MR simultaneously with the image 20. In another example, after the image 20 is saved in the storage device MR, the corrected posture measurement results 32 may be saved in the storage device MR later. When saving of the corrected measurement results 32 is completed, the control unit 11 ends the processing procedure related to the correction according to this operation example.

[0202] Note that the entity that executes the processes of steps S201 and S202 does not have to be limited to the control unit 11 (control device 1). In another example, the control unit 51 of the information processing device 5 may operate as the correction processing unit 514 to execute the processes of steps S201 and S202. In yet another example, another information processing device 45 may execute the processes of steps S201 and S202.

[0203] Execution of the processing procedure in FIG. 18 may be started at any timing after a history of the correctable amount measurement results 30 has been accumulated. The posture measurement results 30 may be accumulated as appropriate depending on the embodiment. In one example, the posture measurement results 30 may be accumulated by continuously executing the process of step S101. In another example, the accumulated posture measurement results 30 may be acquired via a network, a storage medium, another computer, an external storage device, etc.

[0204] [Image analysis] Fig. 19 is a flowchart showing an example of a processing procedure for image analysis according to this embodiment. The processing procedure in Fig. 19 is merely an example, and each step may be changed as much as possible. Furthermore, steps in the processing procedure in Fig. 19 may be omitted, replaced, or added as appropriate depending on the embodiment. Execution of the processing procedure in Fig. 19 may be started at any timing after image 20 is obtained.

[0205] (Step S301) In step S301, the control unit 11 operates as the image analysis unit 115 to perform image analysis on the image 20. As a result, the control unit 11 can obtain the result 25 of the image analysis.

[0206] In one example, the control unit 11 may perform image analysis to detect an object OB appearing in the image 20. In one example, the moving object MB may be an entity that moves within a store, and the object OB may be a product OB1 sold in the store.

[0207] In one example, the control unit 11 may perform image analysis to detect events EV captured in the image 20, in addition to or instead of detecting the object OB. In one example, the moving object MB may be an entity moving within the store, and the events EV may include at least one of a customer event EV1 related to customers in the store and a product event EV2 related to products sold in the store. When the image analysis is complete, the control unit 11 proceeds to the next step S302.

[0208] (Step S302) In step S302, the control unit 11 operates as the image analysis unit 115 to The image analysis result 25 is stored in association with the image 20.

[0209] The destination for storing the image 20 and the image analysis result 25 may be selected appropriately depending on the embodiment. In one example, if the imaging system SY further includes a storage device MR, the control unit 11 may store the image 20 and the image analysis result 25 in the storage device MR. In the storage device MR, the image 20 may be associated with the posture measurement result 39 and the image analysis result 25. In one example, the image analysis result 25 may be stored in the storage device MR simultaneously with the image 20. In another example, after the image 20 is stored in the storage device MR, the image analysis result 25 may be stored in the storage device MR later. When the storage of the image analysis result 25 is completed, the control unit 11 ends the processing procedure related to the image analysis according to this operation example.

[0210] Note that the entity that executes the processes of steps S301 and S302 does not have to be limited to the control unit 11 (control device 1). In another example, the control unit 51 of the information processing device 5 may operate as the image analysis unit 515 to execute the processes of steps S301 and S302. In yet another example, another information processing device 45 may execute the processes of steps S301 and S302.

[0211] [Search process] Fig. 20 is a flowchart showing an example of a processing procedure for searching a database by the information processing device 5 according to this embodiment. The processing procedure in Fig. 20 is merely an example, and each step may be changed as much as possible. Furthermore, steps in the processing procedure in Fig. 20 may be omitted, replaced, or added as appropriate depending on the embodiment.

[0212] (Step S501) In step S501, the control unit 51 operates as the reception unit 511 and receives the query 60 from the terminal device T1.

[0213] In one example, the storage device MR may store the image 20, the posture measurement result 39, and the imaging time TT in association with each other. The control unit 51 may accept a query 60 including at least one of a target image to be searched, a target posture, and a target time. In one example, the storage device MR may store the image 20, the posture measurement result 39, the image analysis result 25, and the imaging time TT in association with each other. The control unit 51 may accept the query 60 including at least one of a target image to be searched, a target posture, target information, and a target time. Upon accepting the query 60, the control unit 51 proceeds to the next step S502.

[0214] (Step S502) In step S502, control unit 51 operates as search unit 512 to search the database (storage device MR). Through this search, control unit 51 extracts elements that match query 60 from the database. This allows control unit 51 to obtain search result 65 made up of the extracted elements. Upon obtaining search result 65, control unit 51 proceeds to the next step S503.

[0215] (Step S503) In step S503, the control unit 51 operates as the response unit 513 to respond (reply) to the terminal device T1 with the obtained search result 65. After returning the search result 65 to the terminal device T1, the control unit 51 ends the processing procedure related to the search according to this operation example.

[0216] §4 Variations Although the embodiments of the present disclosure have been described above in detail, the above description is merely an example of the present disclosure in every respect. The present invention can be practiced in any combination as long as it does not cause any inconvenience. Furthermore, various improvements or modifications may be made to the above-described embodiments as appropriate.

[0217] §5 Experimental Examples The following experiments were conducted to verify the effects of the present disclosure, but the present disclosure is not limited to the following experimental examples.

[0218] (First Experimental Example) In the first experimental example, a smartphone was used to verify the effectiveness of imaging control according to the processing procedures shown in Figures 16 and 17. Specifically, the smartphone camera was used as the imaging device of the imaging system. After attaching the smartphone to a commercially available autonomously moving robot, the robot was made to run for 44 minutes. A LiDAR (Light Detection and Ranging) device was also attached to the robot, and self-location estimation results were obtained using LiDAR SLAM (Simultaneous Localization and Mapping). The obtained self-location estimation results were used as the posture measurement results. The smartphone was made to control imaging using the camera according to the processing procedures shown in Figures 16 and 17. When the smartphone was more than 1 m away from the previous imaging posture or tilted more than 90 degrees, it was made to determine that the target area had not been imaged and to perform imaging processing using the camera. The number of times imaging processing was performed using the camera was counted by plotting the points where imaging processing was performed using the camera.

[0219] FIG. 21 shows the plotted results of the points where the image capture process was performed in the first experimental example. As shown in FIG. 21, the number of times the image capture process was performed by the camera during the 44-minute drive was 95. In contrast, the smartphone was capable of capturing images 3.5875 times per second, so the number of times images were captured when constantly capturing images for 44 minutes was 9471. This result demonstrates that the control method of this embodiment (FIGS. 16 and 17) enables efficient image capture.

[0220] (Second Experimental Example) In the second experimental example, the effect of correction processing on the movement trajectory obtained by the inertial sensor was verified. Specifically, a rectangular frame-shaped passage was generated on a computer. Points were set at the four corners of the generated passage: top left, top right, top right, and bottom right. The circular route that starts from the top left point, passes through each point in the order top right, top right, and bottom right, and returns to the top left point, was defined as the true value of the movement trajectory (Ground In addition, assuming self-position estimation using an inertial sensor, a position slightly away from each of the four corners was set as the target value, and an arc-like movement trajectory was derived for the portion from the top left point to the top right point, and a linear movement trajectory was derived for the remaining portion. The movement trajectory obtained by connecting these was then rotated and scaled by appropriate values ​​to generate the initial value of the movement trajectory (Initial Inaccurate Trajectory). Under the condition that the points are the same, the correction of the first method is applied to the initial value of the movement trajectory to obtain a provisionally corrected movement trajectory (LC Output). By setting the range and applying the correction of method 2-1 to the movement trajectory (LC Output), Furthermore, the corrected movement trajectory (MM Output) was obtained. Then, the inside and outside of the passage were measured as obstacle areas. By setting it to , and applying the correction of the 2-2 method to the movement trajectory (MM Output), the final The typical movement trajectory (PF Output) was obtained.

[0221] FIG. 22 shows the initial inaccurate trajectory in the second experimental example. FIG. 23 shows the movement trajectory (LC Output) and the true value (Ground Truth) of the movement trajectory after correction by the first method in the second experimental example. FIG. 24 shows the movement trajectory (LC Output) and the true value (Ground Truth) of the movement trajectory after correction by the first method in the second experimental example. 25 shows the movement trajectory (MM Output) and the true value of the movement trajectory (Ground Truth) after correction by the 2-1 method in the second experimental example. FIG. 25 shows the movement trajectory (PF Output) and the true value of the movement trajectory (Ground Truth) after correction by the 2-2 method in the second experimental example.

[0222] Between the initial inaccurate trajectory and the ground truth The average error was 8.71 m, and the maximum error was 12.40 m. In contrast, the average error between the corrected movement trajectory (LC Output) and the true value (Ground Truth) for the first method was 7. The average error was 57m, with a maximum error of 11.04m. This result shows that the correction by Method 1 can improve the accuracy of the movement trajectory. In addition, the average error between the movement trajectory (MM Output) and the true value (Ground Truth) after correction by Method 2-1 was 0.63m. The maximum error was 2.48 m. From this result, it was found that the accuracy of the movement trajectory can be improved even by the correction of the 2-1 method. Furthermore, the average error between the movement trajectory (PF Output) and the true value (Ground Truth) after the correction of the 2-2 method was 1.43 m. The maximum error was 1.83 m. In other words, although the average error worsened with the correction using Method 2-2, the maximum error was reduced. This result shows that the correction using Method 2-2 can also be expected to improve the accuracy of the movement trajectory. The above results show that by correcting the movement trajectory obtained by the inertial sensor so that it satisfies the constraints of the movement conditions, it is possible to ensure the accuracy of the movement conditions. [Explanation of symbols]

[0223] SY: Imaging system, 1...control device, 2...imaging device, 3...sensor, 11...control unit, 12...storage unit, 125...map information, 30...Measurement results, 35...Target range, MB: Mobile

Claims

1. an imaging device separately attached to the moving object; a sensor for measuring the orientation of the imaging device; and control device, An imaging system comprising: The control device a storage unit for storing map information that records the imaged range; and control unit, Equipped with The control unit acquiring a measurement result of the attitude of the imaging device by the sensor; determining whether an image of a target range toward which the imaging device is facing, which is identified from the acquired posture measurement result, has been captured based on the map information stored in the storage unit; When it is determined that the target range has not been photographed, an imaging process is performed by the imaging device; If it is determined that the target range has been imaged, execution of the imaging process by the imaging device is omitted. It is configured as follows: The imaging process includes: activating the imaging device; controlling the imaging device to image the target area; After the target range is captured, the imaging device is stopped from being activated; and updating the map information to indicate that the area of ​​interest has been imaged; Including, Imaging system.

2. The moving body is an autonomously moving robot device. The imaging system according to claim 1 .

3. The moving body is a device that moves by direct or remote manual operation. The imaging system according to claim 1 .

4. The moving object is a living organism. The imaging system according to claim 1 .

5. The organism is a human being. The imaging system according to claim 4 .

6. the sensor is an inertial sensor; The imaging system according to claim 1 .

7. The imaging system is configured by a terminal device integrally including the imaging device, the sensor, and the control device. The imaging system according to claim 1 .

8. a storage device that stores an image captured by the imaging device in association with a measurement result of the orientation of the imaging device at the time of capturing the image, The imaging system according to claim 1 .

9. the posture measurement result associated with the image is obtained by correcting the posture measurement result by the sensor so as to satisfy constraints of a movement condition given in advance to the moving object; The imaging system according to claim 8 .

10. The storage device stores the image in further association with a result of image analysis on the image. The imaging system according to claim 8 .

11. The image analysis includes detecting objects in the image. The imaging system according to claim 10.

12. The mobile object is an entity that moves within the store, The objects include products sold in the store. The imaging system according to claim 11 .

13. The image analysis includes detecting an event that is a phenomenon related to an object shown in the image. The imaging system according to claim 10.

14. The mobile object is an entity that moves within the store, The event includes a customer event related to a customer in the store. The imaging system according to claim 13.

15. The mobile object is an entity that moves within the store, The event includes a product event related to a product sold in the store. The imaging system according to claim 13.

16. 1. A computer-implemented information processing method, comprising: The computer connected to an imaging device separately attached to a moving object and a sensor for measuring the attitude of the imaging device; It keeps map information that records the area that has been photographed, The information processing method includes: acquiring a measurement result of the attitude of the imaging device by the sensor; determining whether an image of a target range toward which the imaging device is facing, which is identified from the acquired posture measurement result, has been captured based on the stored map information; When it is determined that the target range has not been photographed, an imaging process is performed by the imaging device; If it is determined that the target range has been imaged, execution of the imaging process by the imaging device is omitted. This includes: The imaging process includes: activating the imaging device; controlling the imaging device to image the target area; After the target range is captured, the imaging device is stopped from being activated; and updating the map information to indicate that the area of ​​interest has been imaged; Including, Information processing methods.

17. A program for causing a computer to execute an information processing method, The computer connected to an imaging device separately attached to a moving object and a sensor for measuring the attitude of the imaging device; It keeps map information that records the area that has been photographed, The information processing method includes: acquiring a measurement result of the attitude of the imaging device by the sensor; determining whether an image of a target range toward which the imaging device is facing, which is identified from the acquired posture measurement result, has been captured based on the stored map information; When it is determined that the target range has not been photographed, an imaging process is performed by the imaging device; If it is determined that the target range has been imaged, execution of the imaging process by the imaging device is omitted. This includes: The imaging process includes: activating the imaging device; controlling the imaging device to image the target area; After the target range is captured, the imaging device is stopped from being activated; and updating the map information to indicate that the area of ​​interest has been imaged; Including, program.

Images