Information processing device, information processing method, and program

Abstract

This technology provides efficient access to monitoring results for a specific area from the monitoring results obtained by an imaging device. [Solution] An information processing device relating to one aspect of the present disclosure acquires first range information indicating the first imaging range of one or more first imaging devices deployed in the environment, acquires second range information indicating the second imaging range of a second imaging device present in the environment, identifies the relative positional relationship between the second imaging range of the second imaging device and the first imaging range of each of the one or more first imaging devices from the acquired first range information and second range information, extracts the monitoring results in the second imaging range of the second imaging device from the monitoring results regarding the behavior of a subject obtained from the first image captured by each of the one or more first imaging devices, and outputs the extracted monitoring results.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, technologies for imaging the environment with an imaging device and estimating the behavior of a subject existing in the environment by analyzing the obtained captured image have been developed. For example, Non-Patent Document 1 proposes a technique for estimating an object that a person shown in a target image is looking at by using a trained machine learning model.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Non-Patent Document 4

Non-licensed Document 5

Non-licensed Document 6

Non-licensed Document 7

Non-licensed literature 9

[0004] Conventional methods allow for obtaining monitoring results of a subject (direction of gaze, object of interest, etc.) from an imaging device. However, the inventors of this invention have found the following problems with the conventional technology. Specifically, in conventional methods, such as reflecting the monitoring results on the image captured by the imaging device (Non-Patent Literature 1), the monitoring results are provided based on the viewpoint of the imaging device monitoring the subject. The viewpoint of the imaging device is not necessarily appropriate for the range of the target for which monitoring results are desired. For example, the imaging device may be capturing a wide range, capturing from an angle that is difficult to see, etc. Depending on the perspective, it may be difficult to specify the target area. This could mean that accessing the monitoring results for the target area from the monitoring results obtained by the imaging device could be time-consuming.

[0005] In one respect, this disclosure has been made in consideration of these circumstances, and one of its purposes is to provide a technology for efficiently accessing the monitoring results for a specific area from the monitoring results obtained by an imaging device. [Means for solving the problem]

[0006] This disclosure adopts the following configuration to solve the aforementioned problems. Note that the following configurations can be combined as appropriate.

[0007] An information processing device relating to one aspect of this disclosure includes a control unit. The control unit is configured to perform the following actions when monitoring the behavior of a subject: acquiring first range information indicating the first imaging range of one or more first imaging devices deployed in the environment; acquiring second range information indicating the second imaging range of a second imaging device present in the environment; identifying the relative positional relationship between the second imaging range of the second imaging device and the first imaging range of one or more first imaging devices from the acquired first range information and second range information; extracting the monitoring results in the second imaging range of the second imaging device from the monitoring results regarding the behavior of a subject obtained from first images captured by one or more first imaging devices, based on the identified positional relationship; and outputting the extracted monitoring results.

[0008] In this configuration, monitoring results of the subject are obtained by one or more first imaging devices. Based on the relative positional relationship between the first imaging range of the first imaging device and the second imaging range of the second imaging device, the monitoring results for the second imaging range of the second imaging device are extracted from the monitoring results obtained by the first imaging device. In other words, by simply pointing the second imaging device towards the target area, the range from which monitoring results are to be extracted can be specified. Therefore, with this configuration, efficient access to the monitoring results for the target area from the monitoring results obtained by the first imaging device can be expected by using the second imaging device.

[0009] In the information processing device relating to the above aspect, outputting the extracted monitoring results may be configured to output the extracted monitoring results in such a way that they are reflected in the second image captured by the second imaging device, based on the specified positional relationship. The extracted monitoring results may also be provided after being reflected in the first image captured by the first imaging device, but with this method of provision, the viewpoints from which the monitoring results can be viewed are limited. Depending on the orientation of the first imaging device relative to the range of the target, there is a possibility that areas with poor visibility (for example, areas captured from angles that are difficult to see, blind spots, etc.) may occur within the monitoring range of the first imaging device. In contrast, with this configuration, by reflecting the monitoring results in the second image captured by the second imaging device, the monitoring results can be provided from a relatively free and new viewpoint of the second imaging device. For example, by capturing the range of the target from the front with the second imaging device, or by capturing the range of the target from an appropriate direction with the second imaging device, it is possible to expect an improvement in the visibility of the provided monitoring results.

[0010] In the information processing device relating to the above aspect, the monitoring results may include analytical information showing the results of analyzing the subject's behavior. According to this configuration, the monitoring results can provide the results of behavioral analysis (analytical information).

[0011] In the information processing device relating to the above aspect, the analysis information may show the results of analyzing the subject's behavior during a specific period within the period monitored by one or more first imaging devices. According to this configuration, the analysis results of the behavior of the subject are narrowed down to the behavior performed during the specific period. (Analysis information) can be provided.

[0012] In the information processing device relating to the above aspect, the analysis information may show the results of analyzing the behavior of subjects having specific attributes among subjects monitored by one or more first imaging devices. With this configuration, it is possible to provide analysis results (analysis information) of the behavior of subjects having specific attributes, after narrowing it down to those subjects.

[0013] In the information processing apparatus according to the above aspect, the result of analyzing the action indicated by the analysis information may include the degree of the subject's action. According to this configuration, it is possible to provide an analysis result (analysis information) including the degree of action.

[0014] In the information processing apparatus according to the above aspect, the degree of the subject's action may include at least any one of a statistic of the duration of the action by the subject, a statistic of the frequency of the action by the subject, and a statistic of the number of subjects who executed the action. According to this configuration, it is possible to provide, as an analysis result (analysis information), at least any one of the duration of the action executed by the subject, the frequency of the action executed by the subject, and the number of subjects who executed the action.

[0015] In the information processing apparatus according to the above aspect, the result of analyzing the action indicated by the analysis information may include the content of the action by the subject. According to this configuration, it is possible to provide an analysis result (analysis information) including the content of the action.

[0016] In the information processing apparatus according to the above aspect, the analysis information may indicate the result of analyzing the subject's action by at least any one of a heat map, a marker, and a funnel diagram. According to this configuration, it is possible to provide a visual analysis result (analysis information) by at least any one of a heat map, a marker, and a funnel diagram.

[0017] In the information processing apparatus according to the above aspect, the monitoring result may include related information according to the result of analyzing the subject's action. According to this configuration, it is possible to provide, as the monitoring result, related information according to the analysis result of the action.

[0018] In the information processing apparatus according to the above aspect, the subject may include a human. The action of the subject may include a first action of a human. According to this configuration, in a scene where a monitoring result regarding a human action (first action) is obtained, efficient access to the monitoring result of the target range can be expected.

[0019] In the information processing apparatus according to the one aspect, the first action of a human may include at least any one of browsing, proximity, and fingering interaction operations. According to this configuration, in a scene where monitoring results of at least any one of browsing, proximity, and fingering interaction operations are obtained, efficient access to the monitoring results of the target range can be expected.

[0020] In the information processing apparatus according to the one aspect, the monitoring result may be a monitoring result regarding the first action of a human who satisfies a predetermined attribute condition among the humans monitored by one or more first imaging devices. According to this configuration, it is possible to provide a monitoring result regarding the action of a human after narrowing down to a human who satisfies a predetermined attribute condition.

[0021] In the information processing apparatus according to the one aspect, the predetermined attribute condition may include having executed a second action. According to this configuration, it is possible to provide a monitoring result regarding the action of a human after narrowing down to a human who has executed the second action.

[0022] In the information processing apparatus according to the one aspect, the second action is (1) browsing a specific area in the environment , (2) approaching a specific area in the environment, and (3) fingering interaction operation with respect to a specific area in the environment may include at least any one of them. According to this configuration, it is possible to provide a monitoring result regarding the action of a human after narrowing down to a human who has executed at least any one of the actions (1) to (3) in the specific area as the second action.

[0023] In the information processing apparatus according to the one aspect, the environment may include the space of a store. According to this configuration, in a scene where a target (subject) existing in the space of a store is monitored, efficient access to the monitoring results of the target range can be expected.

[0024] In the information processing device relating to the above aspect, the subject may include customers of the store. The monitoring results may relate to the behavior of customers who meet predetermined purchasing conditions among the customers monitored by one or more first imaging devices. With this configuration, it is possible to provide monitoring results regarding the behavior of customers who meet predetermined purchasing conditions.

[0025] In the information processing device relating to the above aspect, the subject may include customers of the store. The actions of the subject may include actions taken by customers toward objects in the store. According to this configuration, it is possible to provide monitoring results regarding customers toward objects in the store.

[0026] In the information processing device relating to the above aspect, at least one of the one or more first imaging devices may be fixed in a specific position. When the first imaging device is fixed in a specific position, the viewpoint of the first imaging device may become limited. When specifying the range of an object on the first image of the first imaging device, the limited viewpoint of the first imaging device may result in areas that are difficult to specify, such as areas captured from angles that are difficult to see, or blind spots. This may increase the effort required to access the monitoring results of the range of the object. In contrast, with this configuration, even if the viewpoint of the first imaging device is limited, efficient access to the monitoring results of the range of the object can be expected by using the second imaging device. In other words, the effect of improved accessibility by using the second imaging device can be particularly expected when at least one of the one or more first imaging devices is fixed in a specific position.

[0027] In the information processing device relating to the above aspect, at least one of the one or more first imaging devices may be fixed in a specific position. In addition, outputting the extracted monitoring results may be configured to output the extracted monitoring results in such a way that the extracted monitoring results are reflected in a second image captured by the second imaging device, based on the specified positional relationship. As described above, if the first imaging device is fixed in a specific position, the viewpoint of the first imaging device may become limited. If a method of providing the monitoring results is adopted in which the monitoring results are reflected in the first image captured by the first imaging device, the viewpoint from which the monitoring results can be viewed becomes even more limited. In contrast, with this configuration, even if the viewpoint of the first imaging device is limited, the visibility of the provided monitoring results can be expected to improve by reflecting the monitoring results in the second image captured by the second imaging device. In other words, the effect of improving visibility by using the second image of the second imaging device can be particularly expected in situations where at least one of the one or more first imaging devices is fixed in a specific position.

[0028] In the information processing device relating to the above aspect, the first range information may include a three-dimensional representation of the environment constructed by providing a three-dimensional reconstruction model with first images captured by one or more first imaging devices. The second range information may include a second image captured by a second imaging device. In addition, identifying the relative positional relationship between the second imaging range of the second imaging device and the first imaging range of each of the one or more first imaging devices may be achieved by comparing the second image with the three-dimensional representation to identify the range of the second image within the three-dimensional representation. With this configuration, an improvement in the accessibility of the environment can be expected by providing a three-dimensional representation of the environment.

[0029] An information processing device relating to one aspect of this disclosure includes a control unit and is connected to an external computer. The external computer is configured to access first range information indicating the first imaging range of each of the one or more first imaging devices deployed in the environment when monitoring the behavior of a subject, and monitoring results regarding the behavior of a subject obtained from first images captured by each of the one or more first imaging devices. The control unit is configured to acquire second range information indicating the second imaging range of a second imaging device present in the environment, transmit the acquired second range information to the external computer, thereby instructing the external computer to identify the relative positional relationship between the second imaging range of the second imaging device and the first imaging range of each of the one or more first imaging devices from the first range information and the second range information, and to extract the monitoring results in the second imaging range of the second imaging device from the monitoring results regarding the behavior of a subject obtained from first images captured by each of the one or more first imaging devices based on the identified positional relationship, to receive the extracted monitoring results from the external computer, and to output the received monitoring results. With this configuration, by using the second imaging device, it is possible to efficiently access the monitoring results for the target area from the monitoring results of the first imaging device.

[0030] The embodiments of this disclosure are not limited to the information processing device described above. As another embodiment of the information processing device relating to each of the above aspects, one aspect of this disclosure may be an information processing method that implements all or part of the above configurations, a program, or a machine-readable storage medium that stores such a program. Here, a machine-readable storage medium may be a non-temporary medium that stores information such as programs by electrical, magnetic, optical, mechanical, or chemical action. A non-temporary storage medium may include storage media (CDs, DVDs, semiconductor memory, etc.), auxiliary storage devices of a computer, external storage devices connected to a computer, etc.

[0031] For example, an information processing method relating to one aspect of this disclosure may be performed by a computer. The information processing method may include: acquiring first range information indicating the first imaging range of one or more first imaging devices deployed in the environment when monitoring the behavior of a subject; acquiring second range information indicating the second imaging range of a second imaging device present in the environment; identifying the relative positional relationship between the second imaging range of the second imaging device and the first imaging range of one or more first imaging devices from the acquired first range information and second range information; extracting monitoring results in the second imaging range of the second imaging device from the monitoring results regarding the behavior of a subject obtained from first images captured by one or more first imaging devices, based on the identified positional relationship; and outputting the extracted monitoring results.

[0032] Furthermore, for example, a program relating to one aspect of this disclosure may be a program that causes a computer to execute an information processing method. The information processing method may include: acquiring first range information indicating the first imaging range of one or more first imaging devices deployed in the environment when monitoring the behavior of a subject; acquiring second range information indicating the second imaging range of a second imaging device present in the environment; identifying the relative positional relationship between the second imaging range of the second imaging device and the first imaging range of one or more first imaging devices from the acquired first range information and second range information; extracting monitoring results in the second imaging range of the second imaging device from the monitoring results regarding the behavior of a subject obtained from first images captured by one or more first imaging devices, based on the identified positional relationship; and outputting the extracted monitoring results.

[0033] Furthermore, for example, the information processing method relating to one aspect of this disclosure may be performed by a computer. The computer may be connected to an external computer. The external computer monitors the behavior of the subject, and each of the first imaging devices deployed in the environment. The system may be configured to access first range information indicating an imaging range, and monitoring results regarding the behavior of a subject obtained from first images captured by each of the one or more first imaging devices. The information processing method may include: acquiring second range information indicating the second imaging range of a second imaging device present in the environment; transmitting the acquired second range information to an external computer, thereby instructing the external computer to identify the relative positional relationship between the second imaging range of the second imaging device and the first imaging range of each of the one or more first imaging devices based on the first range information and the second range information; extracting monitoring results in the second imaging range of the second imaging device from the monitoring results regarding the behavior of a subject obtained from first images captured by each of the one or more first imaging devices based on the identified positional relationship; receiving the extracted monitoring results from the external computer; and outputting the received monitoring results.

[0034] Furthermore, for example, a program relating to one aspect of this disclosure may be a program for causing a computer to execute an information processing method. The computer may be connected to an external computer. The external computer may be configured to access first range information indicating the first imaging range of one or more first imaging devices deployed in the environment when monitoring the behavior of a subject, and monitoring results regarding the behavior of a subject obtained from first images captured by each of the one or more first imaging devices. The information processing method may include acquiring second range information indicating the second imaging range of a second imaging device present in the environment, transmitting the acquired second range information to the external computer to instruct the external computer to identify the relative positional relationship between the second imaging range of the second imaging device and the first imaging range of each of the one or more first imaging devices from the first range information and the second range information, and, based on the identified positional relationship, to extract the monitoring results in the second imaging range of the second imaging device from the monitoring results regarding the behavior of a subject obtained from first images captured by each of the one or more first imaging devices, receiving the extracted monitoring results from the external computer, and outputting the received monitoring results. [Effects of the Invention]

[0035] According to one aspect of this disclosure, efficient access to the monitoring results for the target area from the monitoring results obtained by the imaging device (first imaging device) can be expected. [Brief explanation of the drawing]

[0036] [Figure 1] Figure 1 schematically illustrates an example of a scenario in which this disclosure applies. [Figure 2] Figure 2 schematically shows an example of the monitoring results. [Figure 3A] Figure 3A schematically shows an example of the output format of the monitoring results. [Figure 3B] Figure 3B schematically shows an example of the output format of the monitoring results. [Figure 3C] Figure 3C schematically shows an example of the output format of the monitoring results. [Figure 4] Figure 4 schematically shows an example of the behavior of the person being monitored. [Figure 5] Figure 5 schematically shows an example of the environment. [Figure 6] Figure 6 schematically shows an example of related information. [Figure 7] Figure 7 schematically shows one example of a method for determining spatial relationships. [Figure 8] Figure 8 schematically shows an example of the hardware configuration of an information processing device. [Figure 9] Figure 9 schematically shows an example of the software configuration of an information processing device. [Figure 10] Figure 10 is a flowchart showing an example of a processing procedure for an information processing device. [Figure 11] Figure 11 schematically illustrates an example of another scenario to which this disclosure applies. [Figure 12] Figure 12 shows an example of the first image in an experimental case. [Figure 13] Figure 13 shows the area from which the second image was obtained in the experimental example. [Figure 14] Figure 14 shows the second image, which was captured at the first location. [Figure 15] Figure 15 shows the result of reflecting the analysis results (viewing) in the second image, which was taken at the first location. [Figure 16] Figure 16 shows the result of reflecting the analysis results (grasping) in the second image, which was captured at the first location. [Figure 17] Figure 17 shows the second image, which was captured at the second location. [Figure 18] Figure 18 shows the result of reflecting the analysis results (viewing) in the second image, which was captured at the second location. [Figure 19] Figure 19 shows the result of reflecting the analysis results (grasping) in the second image, which was captured at the second location. [Modes for carrying out the invention]

[0037] Hereinafter, embodiments relating to one aspect of this disclosure will be described with reference to the drawings. However, the embodiments described below are merely illustrative in all respects of this disclosure. Various improvements or modifications may be made without departing from the scope of this disclosure. In implementing this disclosure, specific configurations may be adopted as appropriate depending on the embodiment. In this embodiment, the data appearing is described in natural language, but more specifically, it is specified in pseudo-language, commands, parameters, machine code, electrical signals, etc., that can be recognized by machines such as computers.

[0038] §1 Examples of Application Figure 1 schematically shows an example of a scenario to which this disclosure applies. The information processing device 1 according to this embodiment is one or more computers configured to provide the monitoring results of a subject S1 by the first imaging device 2.

[0039] In this embodiment, one or more first imaging devices 2 are deployed in environment E1 and monitor the behavior A1 of a subject S1. The information processing device 1 acquires first range information 25 indicating the first imaging range 21 of each of the one or more first imaging devices 2 deployed in environment E1 while monitoring the behavior A1 of the subject S1. The information processing device 1 acquires second range information 35 indicating the second imaging range 31 of a second imaging device 3 located within environment E1. From the acquired first range information 25 and second range information 35, the information processing device 1 identifies the relative positional relationship 4 of the second imaging range 31 of the second imaging device 3 with respect to the first imaging range 21 of each of the one or more first imaging devices 2. Based on the identified positional relationship 4, the information processing device 1 extracts the monitoring result 55 in the second imaging range 31 of the second imaging device 3 from the monitoring results 50 regarding the behavior A1 of the subject S1 obtained from the first image 23 captured by each of the one or more first imaging devices 2. The information processing device 1 outputs the extracted monitoring result 55.

[0040] In this embodiment, monitoring results 50 of the subject S1 are obtained by one or more first imaging devices 2. Based on the relative positional relationship 4 between the first imaging range 21 of the first imaging device 2 and the second imaging range 31 of the second imaging device 3, monitoring results 55 for the second imaging range 31 of the second imaging device 3 are extracted from the monitoring results 50 from the first imaging device 2. In other words, by simply pointing the second imaging device 3 towards the target area, the range from which monitoring results 55 are to be extracted can be specified. Therefore, according to this embodiment, by using the second imaging device 3, efficient access to monitoring results 55 for the target area from the monitoring results 50 from the first imaging device 2 can be expected.

[0041] [Imaging device] The first imaging device 2 is deployed to observe the behavior A1 of subject S1 in environment E1. In one example, the first imaging device 2 may be a camera already installed in environment E1. For example, the first imaging device 2 may include a surveillance camera, security camera, etc. The first imaging range 21 may be called the surveillance range. The type of the first imaging device 2 is not particularly limited as long as it can observe the behavior A1 of subject S1, and may be appropriately selected depending on the embodiment. The first imaging device 2 may include, for example, a general RGB camera. In one example, the first imaging device 2 may be configured so that the dimensions of the first imaging range 21 can be adjusted by changing the magnification, etc.

[0042] Deployment in environment E1 may include deployment within environment E1 and deployment outside environment E1. In the latter case, the first imaging device 2 may be deployed to observe a subject S1 present in environment E1 from outside environment E1. For example, if environment E1 is the space of a store, at least one of the one or more first imaging devices 2 may be deployed to observe the inside of the store or the surrounding area (exterior, etc.) of the store from outside the store.

[0043] If the orientation of the first imaging device 2 at the time of capturing the first image 23 is specified, the first imaging range 21 can be specified. Therefore, as long as the orientation at the time of imaging can be specified, the first imaging device 2 may be deployed in any orientation. The orientation of the first imaging device 2 may be at least partially changeable, or it may not be changeable. The orientation includes position and orientation.

[0044] In one example, at least one of the one or more first imaging devices 2 may be fixed in a specific position. That is, at least one of the one or more first imaging devices 2 may be installed in a stationary state at a certain position (specific position). When each of the multiple first imaging devices 2 is fixed in a specific position, the fixing position of each first imaging device 2 may be appropriately determined according to the embodiment. When the first imaging devices 2 are efficiently deployed in the environment E1, the fixing position of each first imaging device 2 may be appropriately determined so as to minimize overlap of the first imaging range 21. The method of fixing in a specific position is not particularly limited and may be appropriately selected according to the embodiment. In one example, the first imaging device 2 may be fixed in a specific position by a known method. The first imaging device 2 may be fixed to a structure (ceiling, wall, etc.) located inside or outside the environment E1, or to any other object (shelf, desk, etc.). The orientation of the first imaging device 2 fixed in a specific position may be variable, or it may be fixed, just like its position. The orientation of the first imaging device 2 may be changed manually. The first imaging device 2 may be configured to allow its orientation to be changed by mechanical control using a mechanical mechanism such as an electric pan / tilt head.

[0045] When the first imaging device 2 is fixed in a specific position, its viewpoint may become limited. If the range of the target from which to extract monitoring results 55 is specified on the first image 23 of the first imaging device 2, the limited viewpoint of the first imaging device 2 may result in areas that are difficult to specify as the target range, such as areas captured from angles that are difficult to see or areas that are blind spots. This may increase the effort required to access the monitoring results 55 for the target range. In contrast, according to one example of this embodiment, even if the viewpoint of the first imaging device 2 is limited by being fixed in a specific position, efficient access to the monitoring results 55 for the target range can be expected by using the second imaging device 3. In other words, the effect of improved accessibility by using the second imaging device 3 can be particularly expected when at least one of the one or more first imaging devices 2 is fixed in a specific position.

[0046] The deployment method of the first imaging device 2 is not limited to this example. In another example, one or more first imaging devices 2 may be configured to be movable. Being movable of the first imaging device 2 may include being able to move autonomously and being able to move manually by a person. For example, the first imaging device 2 may be installed on an object that can move within the environment E1. The movable object may include at least one of a device that moves by manual operation and an autonomously moving robotic device. A device that moves by manual operation may include, for example, a manually operated vehicle, a remotely operated aircraft, a cart that is pushed or operated by hand, and other devices configured to be movable by manual operation. If the environment E1 is a store space, a cart that is pushed or operated by hand may include a shopping cart. An autonomously moving robotic device may include, for example, an autonomously operated vehicle, an autonomously flying aircraft, and other robotic devices. An aircraft (remotely operated aircraft, autonomously flying aircraft) may include a drone. The robotic device may include, for example, a cleaning robot, a serving robot, a transport robot, a guidance robot, a security robot, and the like. In one example, installing the first imaging device 2 on a robotic device may be configured such that the robotic device is equipped with the first imaging device 2. Also, in one example, being manually movable by a person may include being carried by a moving person.

[0047] The second imaging device 3 is used to define the range of subjects from which the monitoring results 55 are extracted. The second imaging device 3 may be any camera. The second imaging device 3 may include, for example, a camera on a terminal device, an existing camera separate from the first imaging device 2, a camera on a robot device, etc. In one example, if the environment E1 is the space of a store, the second imaging device 3 may be installed on a terminal device, and the terminal device equipped with the second imaging device 3 may be carried by customers (visitors), store employees, or patrol personnel. Patrol personnel may include, for example, security guards, police officers, etc. Store employees may include not only persons employed by the store, but also any persons working in the store (cleaners, etc.). Terminal devices may include, for example, mobile phones (smartphones, etc.), tablet devices, notebook PCs (personal computers), It may be a dedicated device or the like. The type of the second imaging device 3 is not particularly limited as long as the range of the target can be defined, and may be appropriately selected depending on the embodiment. The second imaging device 3 may include, for example, a general RGB camera. In one example, the second imaging device 3 may be configured so that the dimensions of the second imaging range 31 can be adjusted by changing the magnification, etc.

[0048] The second imaging device 3 may or may not be movable within the environment E1. Similar to the example of the first imaging device 2 described above, the mobility of the second imaging device 3 may include both autonomous movement and manual movement by human hands. For example, the second imaging device 3 may be installed on an object that is movable within the environment E1. The object that is movable within the environment E1 may be the same as in the example above. Manual movement by human hands may include being carried by a moving person.

[0049] [environment] Environment E1 may be any physical space. For example, Environment E1 may include the space of a store (such as store E11 in Figure 5, which will be described later). Environment E1 may include at least one indoor or outdoor space. Environment E1 may be a space with defined partitions or a space without defined partitions.

[0050] The subject S1 may be any object that appears in the captured image (first image 23). The subject S1 may include objects that appear temporarily in the environment E1. The subject S1 may also include objects that remain in the environment E1 for a long period of time or permanently. The time that the subject S1 stays in the environment E1 is not particularly limited and may be determined as appropriate depending on the embodiment.

[0051] Furthermore, as long as the behavior A1 can be analyzed, the type of subject S1 is not particularly limited and may be appropriately selected depending on the embodiment. In one example, the subject S1 (object) may include at least one of a living organism and a device (machine). The type of living organism and device may be selected depending on the embodiment.

[0052] [Output Method] The extracted monitoring results 55 may be output in any way. For example, outputting the extracted monitoring results 55 may be configured to reflect the extracted monitoring results 55 in the second image 33 captured by the second imaging device 3, based on the specified positional relationship 4. That is, the extracted monitoring results 55 may ultimately be reflected in the second image 33 and output together with the second image 33. Reflection may include visualization, drawing, superimposing, projection, etc. Specific examples of reflection are shown in Figures 3A to 3C described later. The second image 33 may also be called a query image, as it provides the range from which to extract the monitoring results 55 as the second imaging range 31 and is an image that reflects the extracted monitoring results 55.

[0053] The second image 33 may be output by the information processing device 1, or it may be output by a computer different from the information processing device 1. For example, outputting the extracted monitoring result 55 so that the extracted monitoring result 55 is reflected in the second image 33 may include reflecting the extracted monitoring result 55 in the second image 33 and outputting the second image 33 with the extracted monitoring result 55 reflected. Furthermore, outputting the extracted monitoring result 55 so that the extracted monitoring result 55 is reflected in the second image 33 may also include giving an instruction to another computer to reflect the extracted monitoring result 55 in the second image 33, along with the extracted monitoring result 55, in order to output the second image 33 with the extracted monitoring result 55 reflected. The other computer may be controlled to reflect the monitoring result 55 in the second image 33 in response to this instruction and output the second image 33 with the monitoring result 55 reflected.

[0054] If the extracted monitoring results 55 are output from the viewpoint of the first imaging device 2 (for example, by reflecting the monitoring results 55 in the first image 23), the viewpoints from which the monitoring results 55 can be viewed will be limited. Depending on the orientation of the first imaging device 2 relative to the range of the target, there may be areas within the monitoring range of the first imaging device 2 that are difficult to see. Areas with poor visibility may include, for example, areas captured from angles that are difficult to see, or blind spots. For example, capturing from angles that are difficult to see may include viewing the surface of the target from an oblique angle (in particular, viewing it from an oblique direction from a viewpoint close to the surface, i.e., viewing the surface in a direction parallel or nearly parallel to the surface). In contrast, according to one example of this embodiment, by reflecting the monitoring results 55 in the second image 33 captured by the second imaging device 3, the monitoring results 55 can be provided from the relatively free and new viewpoint of the second imaging device 3. For example, by using the second imaging device 3 to image the target area from the front, or by using the second imaging device 3 to image the target area from an appropriate direction, it is possible to expect an improvement in the visibility of the provided monitoring results 55.

[0055] Furthermore, the second image 33 does not necessarily have to be obtained by capturing the target area from the correct direction. Even if the second imaging device 3 captures the target area from an inappropriate direction, by adopting an output method that reflects the monitoring result 55 in the second image 33, the visibility of the monitoring result 55 can be improved (for example, if the second image 33 is re-captured from the correct direction, visibility can be improved). In other words, introducing the viewpoint of the second imaging device 3, which is different from the viewpoint of the first imaging device 2, into the output of the monitoring result 55 provides a new viewpoint when viewing the monitoring result 55, and thus can contribute to improving the visibility of the monitoring result 55. Therefore, regardless of the orientation in which the second image 33 is captured by the second imaging device 3, it is possible to expect improved visibility of the monitoring result 55 (i.e., there is a possibility that the visibility of the monitoring result 55 will be improved).

[0056] To facilitate the improvement in visibility, it is preferable that the second imaging device 3 is movable within the environment E1. The second imaging device 3 may be configured to move autonomously (for example, mounted on a mobile robotic device). The second imaging device 3 may also be configured to move manually by a human (for example, mounted on a terminal device carried by a moving user). In other words, the movement of the second imaging device 3 may include both autonomous movement and manual movement.

[0057] In one example, when the extracted monitoring result 55 is output so as to be reflected in the second image 33, at least one of the one or more first imaging devices 2 may be fixed in a specific position. As described above, if the first imaging device 2 is fixed in a specific position, the viewpoint of the first imaging device 2 may become limited. If the extracted monitoring result 55 is output from the viewpoint of the first imaging device 2, the viewpoint from which the monitoring result 55 can be viewed may become even more limited. In contrast, according to one example of this embodiment, even if the viewpoint of the first imaging device 2 is limited, by reflecting the monitoring result 55 in the second image 33 captured by the second imaging device 3, the viewpoint of the provided monitoring result 55 can be viewed. Improved visibility can be expected. In other words, the effect of improving visibility by utilizing the second image 33 of the second imaging device 3 can be particularly expected in situations where at least one of the one or more first imaging devices 2 is fixed in a specific position.

[0058] The method for outputting the monitoring results 55 is not limited to reflecting them in the second image 33, and may be modified as appropriate depending on the embodiment. In another example, the extracted monitoring results 55 may be reflected in the first image 23 instead of the second image 33, and output together with the first image 23. Alternatively, the extracted monitoring results 55 may be output as is, independently of the first image 23 and the second image 33.

[0059] In one example, regardless of the output method, the extracted monitoring results 55 may be output in real time depending on the specified second imaging range 31. For example, the information processing device 1 may perform in real time the following: acquire second range information 35 indicating the second imaging range 31, identify the relative positional relationship 4 of the second imaging range 31, extract the monitoring results 55 in the second imaging range 31, and output the extracted monitoring results 55. The acquired second range information 35 may indicate the current second imaging range 31, and the relative positional relationship 4 of the current second imaging range 31 may be identified. In the output of the extracted monitoring results 55, the extracted monitoring results 55 may be output so as to be reflected in the current second image 33. This makes it possible to output the monitoring results 55 in real time on the current second image 33.

[0060] As a concrete example, a user may move freely within environment E1 while holding a terminal device equipped with an imaging device (second imaging device 3). In one example, if environment E1 is a store space, the user may be a customer, store clerk, or patrol officer. The user may use the imaging device of the terminal device to image the range of the target for which they wish to confirm the monitoring result 55. Image capture using the imaging device may include obtaining an image to be used for live view and recording an image at the time of operation by operating an imaging button (shutter button, etc.). Since the acquisition of the second range information 35 and the output of the monitoring result 55 are performed in real time, an image (second image 33) reflecting the monitoring result 55 may be output to the screen of the terminal device. The information processing device 1 may be a terminal device, or it may be another computer (server device, etc.) that can communicate data with the terminal device and provides the extracted monitoring result 55 to the terminal device. In this concrete example, by imaging a desired range with the imaging device of the terminal device, the monitoring result 55 for that range can be output on the captured image in real time.

[0061] However, the timing of outputting the monitoring results 55 is not limited to this example and may be changed as appropriate depending on the embodiment. In another example, the extracted monitoring results 55 may be output retrospectively.

[0062] [Monitoring results] The monitoring results (50, 55) may include any information regarding the behavior A1 of the subject S1 obtained from the first image 23 captured by at least one of the first imaging devices 2. The monitoring results (50, 55) may include at least one of direct information obtained from the first image 23 and indirect information derived from the direct information regarding the behavior A1 of the subject S1.

[0063] Figure 2 schematically shows an example of monitoring results (50, 55) according to this embodiment. In one example, the monitoring results (50, 55) may include at least one of the following: analysis information 571, related information 572, location 573, time 574, and attribute 575.

[0064] Analysis information 571 may show the results of analyzing the behavior A1 of subject S1. The content of the analysis may be determined as appropriate depending on the embodiment. In one example, analyzing behavior A1 is the first The analysis may include image analysis of the behavior A1 of subject S1 in image 23. The image analysis may include any type of inference (e.g., identification of subject S1, estimation of the type of behavior A1, estimation of the location where behavior A1 is performed, etc.). In one example, analyzing behavior A1 may further include numerical analysis of the results of the image analysis on the first image 23. The numerical analysis may include, for example, statistical calculations, inferences, etc. The analysis information 571 may be generated for any unit, such as for each location or for each predetermined range.

[0065] In one example, the results of analyzing the behavior indicated by the analysis information 571 may include at least one of the degree of behavior A1 of subject S1 and the content of behavior A1 performed by subject S1. The degree of behavior A1 may be measured by an arbitrary index indicating the frequency of behavior A1. In one example, the degree of behavior A1 of subject S1 may include at least one of the statistics of the duration of behavior A1 performed by subject S1, the statistics of the frequency of behavior A1 performed by subject S1, and the statistics of the number of subjects S1 that performed behavior A1. Frequency may include the number of times. Statistics may include the mean, median, nth percentile, mode, maximum, minimum, sum, etc. In one example, statistics may be calculated as a value per predetermined criterion (e.g., mean). In this case, the criterion for the statistics may be appropriately selected depending on the embodiment, for example, per unit time, per subject, etc. Each statistic may be output as is. Each statistic may be used in a predetermined calculation process, and the result of that calculation process may be output. In one example, a ratio of statistics may be calculated between the target behavior A1 and other behaviors, and the calculated ratio may be output as the degree of behavior A1. The calculation of the ratio is an example of a predetermined arithmetic process. The statistics of other behaviors may be obtained as statistics of a different type of behavior A1 than the target behavior A1 (i.e., the analysis result of behavior A1), or may be obtained from a different information source than the first image 23. In addition, the content of behavior A1 may include any information that identifies behavior A1. In one example, the content of behavior A1 may include the type of behavior A1 (e.g., browsing, approaching, grasping, etc.). Behavior A1 may be shown in any format. In one example, the content of behavior A1 may be configured to indicate the type of behavior A1 by tags or natural language.

[0066] Related information 572 may be obtained according to the results of the analysis of the subject S1's behavior A1. Related information 572 may be appropriately derived from the analysis results of behavior A1 (analysis information 571). For example, it may be determined whether or not the analysis results satisfy predetermined evaluation conditions. Whether or not the predetermined evaluation conditions are satisfied may be determined using a rule-based method or using a trained machine learning model. The trained machine learning model may include large-scale generative models such as large-scale language models, large-scale visual language models, and multimodal models. If the analysis results satisfy the predetermined evaluation conditions, predetermined information corresponding to satisfying the predetermined evaluation conditions may be obtained. If the analysis results do not satisfy the predetermined evaluation conditions, predetermined information corresponding to not satisfying the predetermined evaluation conditions may be obtained. Related information 572 may include predetermined information corresponding to whether or not the analysis results satisfy the predetermined evaluation conditions (for example, policy information 580 described later). In addition, information that matches the analysis results of behavior A1 may be retrieved by using the analysis results of behavior A1 as a query. Related information 572 may include retrieved information. Matching the analysis results may include at least one of matching the analysis results and being similar to the analysis results. The scope of similarity may be appropriately determined depending on the embodiment. Similarity may be evaluated by known methods such as similarity searches. The search target may be appropriately selected depending on the embodiment. The search target may include the web, past history, etc. For example, the search target may include past analysis results or monitoring results. Accordingly, related information 572 may include past analysis results or monitoring results that match the analysis results used as a query. For example, the search target may include analysis results or monitoring results (monitoring results 50) in a range different from the second imaging range 31. Accordingly, related information 572 may include analysis results or monitoring results in other ranges that match the analysis results used as a query. Also, for example, the search target may include analysis results or monitoring results in other environments (other stores, etc.) other than environment E1. Accordingly, related information 572 may include analysis results in other environments (other stores, etc.) that match the analysis results used as a query. It may also include analysis or monitoring results of other suitable environments. The related information 572 may be generated from the analysis results of behavior A1 in the second imaging range 31 after the second imaging range 31 has been specified. The related information 572 may be generated in advance for each location before the second imaging range 31 is specified, and when the second imaging range 31 is specified, the related information 572 for the second imaging range 31 may be extracted from the pre-generated related information 572.

[0067] Location 573 may indicate any location related to behavior A1. The location may include at least one of one or more points and ranges. Location 573 may be used to determine whether or not to extract the corresponding monitoring result 50 as monitoring result 55. The configuration of location 573 is not particularly limited and may be determined as appropriate depending on the embodiment, as long as it can be used to extract the monitoring result 55. In one example, location 573 may indicate the location of behavior A1 within at least one of the first imaging range 21 of the first imaging device 2 that captured the first image 23 showing behavior A1, within the first image 23, and within the environment E1. For behavior A1 (subject S1) to be captured in the first image 23, it may be sufficient for the behavior A1 (subject S1) to be captured in the first image 23 to the extent that it is image-analyzable. If position 573 indicates a range, and the range indicated by position 573 partially overlaps with the second imaging range 31, the corresponding monitoring result 50 may be determined to belong to the second imaging range 31 or not. That is, the corresponding monitoring result 50 may be extracted as a monitoring result 55 in the second imaging range 31, or it may not be extracted. In the latter case, if the range indicated by position 573 is included in the second imaging range 31, the corresponding monitoring result 50 may be extracted as a monitoring result 55. For example, position 573 (location of action A1) may indicate at least one of the location of the subject S1 when action A1 was performed, and the location targeted by action A1. The location targeted by action A1 may include the location to which action A1 is directed (location of an object, etc.) and the location in which the action is involved. For example, if action A1 is browsing, position 573 may indicate the location that was browsed. The browsed location is an example of the location to which action A1 is directed. Furthermore, for example, if action A1 is in proximity, position 573 may indicate a nearby location of subject S1. A nearby location is an example of a location in which action A1 is involved. The location targeted by action A1 may be estimated by any method from the first imaging range 21 of the first imaging device 2 that captured the first image 23 in which action A1 is captured, and the state of action A1 by subject S1 as captured in the first image 23 (position, orientation, etc.). Known methods may be used to estimate the location targeted by action A1.Position 573 may be configured to directly indicate the location targeted by action A1. Position 573 may also be configured to indirectly indicate the location targeted by action A1, for example, by the location of subject S1 and the direction from subject S1. Position 573 may include at least one of the locations that are visible in the first image 23 (belonging to the first imaging range 21) and locations that are not visible in the first image 23 (not belonging to the first imaging range 21). For example, when action A1 is browsing and subject S1 is browsing a location that is not visible in the first image 23, position 573 may be configured to indicate the location not visible in the first image 23 by including the location of subject S1 and the direction of browsing (direction of gaze). Including the location of subject S1 and the direction of browsing is an example of a configuration that indirectly indicates the location targeted by action A1.

[0068] Time 574 may represent any point in time related to action A1. Time 574 may be obtained by any method. For example, time 574 may be the time the first image 23 in which action A1 is captured, or it may be any time derived from the time the first image 23 is captured. At least one of the first imaging device 2 and the computer controlling the first imaging device 2 may be equipped with a timer, and the time the first image 23 is captured may be obtained by the timer. Deriving an arbitrary time from the capture time may include estimating an arbitrary time from the capture time, correcting the capture time, etc. Time 574 may include, for example, the start time of action A1, the time while action A1 is continuing, the end time of action A1, etc. The start time may be the time the first image 23 in which action A1 first appears, or it may be a time estimated from the time the first image 23 in which action A1 appears. The end time may be the time the first image 23 in which action A1 last appears. This could be the time of imaging, or it could be a time estimated from the time of imaging of the first image 23 in which behavior A1 appears. For example, the analysis results (50, 55) may include multiple times 574. For example, the multiple times 574 may be configured to indicate the duration of behavior A1 by including the start time and end time. If the degree of behavior A1 includes a statistic of duration, the duration may be measured from the start time and end time.

[0069] Attribute 575 may include any features of subject S1. Attribute 575 may be obtained by any method. Attribute 575 may be estimated from the first image 23, or obtained from other information sources other than the first image 23. In one example, the image analysis of the first image 23 may include identifying subject S1, and attribute 575 may include the result of identifying subject S1. Identifying subject S1 may include identifying the individuality of subject S1, and identifying the identity of subject S1 without identifying its individuality. The method for identifying subject S1 is not particularly limited and may be appropriately selected depending on the embodiment. Known methods such as comparing with registered data may be used as the method for identifying subject S1. In one example, attribute 575 may include other features of subject S1 together with or in lieu of the identification result of subject S1. For example, if subject S1 is a person, attribute 575 may include age, affiliation, residential area, hobbies, preferences, etc. Furthermore, in one example, attribute 575 may include behavioral history. Behavioral history may be identified by linking it with other items of the monitoring results (50, 55) (analysis information 571, location 573, time 574, etc.). In other words, at least some of the other items of the monitoring results (50, 55) may also be used as attribute 575. Behavioral history may be obtained from other sources.

[0070] In one example of this embodiment, the monitoring results (50, 55) include analysis information 571 showing the results of analyzing the behavior A1 of subject S1, thereby providing the analysis results (analysis information 571) of behavior A1 as the monitoring result 55. In another example of this embodiment, the analysis results of behavior A1 include the degree of the behavior of subject S1, thereby providing the analysis results (analysis information 571) including the degree of the behavior. In yet another example of this embodiment, the degree of behavior A1 of subject S1 includes at least one statistic of duration, frequency, and number of subjects S1, thereby providing any of these statistic as the analysis results (analysis information 571). In yet another example of this embodiment, the analysis results of behavior A1 include the content of the behavior A1 performed by subject S1, thereby providing the analysis results (analysis information 571) including the content of the behavior A1. In any case, the provided analysis results may be used for any purpose. For example, if environment E1 is a store space, the provided analysis results may be used as material for considering measures in the store. Furthermore, according to one example of this embodiment, the monitoring results (50, 55) include related information 572 corresponding to the results of the analysis of the subject S1's behavior A1, thereby providing related information 572 corresponding to the analysis results of behavior A1 as the monitoring result 55. Note that the configuration of the monitoring results (50, 55) is not limited to the example in Figure 2 and may be appropriately modified depending on the embodiment. For example, at least one of the analysis information 571 and the related information 572 may be omitted. At least one of the time 574 and the attribute 575 may also be omitted.

[0071] (Analysis process) To obtain monitoring results (50, 55), analysis processing (image analysis, numerical analysis, etc.) may be performed on each first image 23 as appropriate. At least a portion of the analysis processing for each first image 23 may be performed on the information processing device 1, or on a computer other than the information processing device 1. The imaging interval for the first image 23 and the execution interval for the analysis processing are not particularly limited and may be determined as appropriate depending on the embodiment.

[0072] In one example, the subject S1 may be detected individually by performing image analysis on the first image 23 captured by one or more first imaging devices 2. The action A1 performed by the subject S1 may also be detected individually. That is, the results of the image analysis are obtained for each subject S1 and each action A1. The subject S1 may perform one or more actions A1 at a time. If the subject S1 performs multiple actions A1, multiple actions A1 may be detected for this subject S1. Detecting an action A1 may include estimating the content of the action A1 (for example, identifying the type of action A1). That is, at least a portion of the analysis results of the action A1 (analysis information 571) may be obtained by image analysis (detection of action A1) on the first image 23. The location (position 573) of the action A1 may be identified from at least one of the first imaging range 21 and the first image 23. If the position 573 is configured to indicate the location of the action A1, detecting the action A1 may include estimating the location of the action A1. Detecting the subject S1 may include estimating at least a portion of the attributes 575 of the subject S1 (for example, identifying the subject S1, estimating the age of the subject S1, etc.). Time 574 may be obtained from the acquisition time of the first image 23 used for detection. The acquisition time may be used directly as time 574, or time 574 may be derived from the acquisition time. Subject S1 and behavior A1 may be detected from first image 23 at one or more time points. Subject S1 and behavior A1 may be identified between first images 23 at different time points and between first images 23 from different first imaging devices 2 if multiple first imaging devices 2 are deployed. For example, by using the analysis results from the previous time point, it may be evaluated whether the same behavior (behavior A1) by the same subject (subject S1) is continuing. The duration may be measured as the time during which the same behavior by the same subject continues. Through the analysis process up to this point, analysis results for each subject S1 and each behavior A1 can be obtained.

[0073] In one example, the second image 33 obtained by the second imaging device 3 may be used as auxiliary material to fill the space outside the first imaging range 21 of the first imaging device 2 when analyzing the behavior A1 of the subject S1. For example, if the location targeted by behavior A1 includes a location not belonging to the first imaging range 21, the second imaging range 31 may be specified to include this location by orienting the second imaging device 3 towards the location not belonging to the first imaging range 21. In this case, the second image 33 obtained by the second imaging device 3 may be used to analyze the location targeted by behavior A1. As an example of analysis, the position of the subject S1 and the direction of behavior A1 may be estimated from the first image 23 of the first imaging device 2. Based on the estimated position of the subject S1 and the direction of behavior A1, the location targeted by behavior A1 (the destination of behavior A1) may be identified in the space filled by the second image 33 of the second imaging device 3. As a specific example, if action A1 is browsing, and subject S1 is browsing a location not visible in the first image 23, the browsing location may be identified in the space filled in by the second image 33 of the second image 3 by capturing an image of the browsing location with the second image 3. This filling of space by the second image 33 of the second image 3 may be adopted in both cases: when the monitoring result 55 is reflected in the second image 33 and when the monitoring result 55 is not reflected in the second image 33. According to one example of this embodiment, by using the second image 33 of the second image 3 as auxiliary material to fill the space of environment E1, the space to be analyzed in environment E1 can be expanded. That is, action A1 can be analyzed by expanding to the range not visible in the first image 23 of the first image 2. For example, in the above example, even if subject S1 is browsing a range not visible in the first image 23 of the first image 2, the browsing location of subject S1 can be analyzed.

[0074] Next, numerical analysis may be performed on the analysis results for each subject S1 and each action A1 to obtain at least a portion of the analysis results for action A1 (analysis information 571). The numerical analysis may include integrating the analysis results of multiple subjects S1. For example, for the same action (action A1) in the same location, a statistical value of the duration of action A1 by each subject S1 may be calculated by aggregating the duration of each subject S1. For the same action in the same location, a statistical value of the frequency of action A1 by each subject S1 may be calculated by aggregating the frequency of action A1 by each subject S1. For the same action in the same location, a statistical value of the number of subjects S1 that performed action A1 may be calculated by counting the number of subjects S1.

[0075] In one example, after at least a portion of the analysis results (analysis information 571) has been obtained, related information 572 may be appropriately derived from the obtained analysis results at any time. In one example, at least a portion of the analysis results of behavior A1 may also be used as at least a portion of attribute 575 (behavioral history, etc.). In one example, reference information may be obtained from other sources, and the obtained reference information may be used as at least a portion of attribute 575.

[0076] Reference information may be appropriately linked to subject S1. For example, by using the identification result of subject S1, the reference information may be linked to subject S1. For example, when identifying the individuality of subject S1, the reference information may be linked to the individuality of subject S1. For example, the reference information may include user information, device information, etc. If environment E1 includes the space of a store and subject S1 includes customers of the store, the reference information may include customer information, purchase information, etc. In the above image analysis, detecting subject S1 may include identifying the individuality of subject S1. Depending on the result of identifying the individuality of subject S1, at least a part of the attributes 575 may be obtained by accessing the reference information linked to the individuality of subject S1.

[0077] Furthermore, for example, the reference information may be linked to subject S1 by using the results of identifying the identity of subject S1, rather than using the results of identifying the individuality of subject S1. As a specific example, if environment E1 includes the space of a store, subject S1 may include the store's customers, and the reference information may include purchase information. The purchase information may include the purchased items and the time of purchase. Action A1 may include grasping the items. In the analysis of subject S1, the identity of subject S1 (customer) may be identified. In this scenario, for example, the target purchase information may be linked to the customer whose identity has been identified and whose time of grasping the item (time 574) best matches the time of purchase. For example, the time of grasping the item best matching the time of purchase may be the time when the item was grasped closest to the time of purchase. If multiple items are purchased, the target purchase information may be linked to the customer whose grasped items match each purchased item, and whose sum of the difference between the time of grasping each item and the time of purchase is the smallest. In one example, an imaging device may be positioned in front of the register. This imaging device may be used as the first imaging device 2, or as another imaging device other than the first imaging device 2. The target purchase information may be linked to the customer whose imaging time by the imaging device positioned in front of the register best matches the purchase time among the customers whose identity has been identified. The target purchase information may be used as at least part of the attributes 575 of the linked customer (subject S1). The reference information may be stored in any storage area accessible by the information processing device 1. Any storage area may be, for example, the memory resources of the information processing device 1, the memory resources of another computer, or an external storage device. The external storage device may include a data server such as a NAS (Network Attached Storage).

[0078] Based on the above, monitoring results (50, 55) can be obtained. Note that the order of the analysis processes to obtain the monitoring results (50, 55) is not limited to the above example and may be changed as appropriate depending on the embodiment. Also, each calculation method of the analysis process is not particularly limited and may be selected as appropriate depending on the embodiment. Known methods may be used for the analysis process. In one example, image analysis may be performed by general image analysis (edge ​​extraction, pattern matching, etc.) or by using a pre-trained machine learning model. The pre-trained machine learning model may include large-scale generative models capable of processing images, such as large-scale visual language models and multimodal models. Numerical analysis may be performed by general analysis processes (statistical calculations, etc.) or by using a pre-trained machine learning model. The pre-trained machine learning model may include large-scale generative models capable of numerical calculations, such as large-scale language models, large-scale visual language models, and multimodal models. An example of a specific implementation method will be described later in the section on [Method for determining positional relationships].

[0079] (Processing timing) In one example, all analysis processes may be performed in advance to generate the monitoring result 50. The generated monitoring result 50 may be stored in any memory area. The monitoring result 55 for the second imaging range 31 may be extracted from the stored monitoring result 50. However, the timing of executing the analysis processes and the unit in which the data (information source for the monitoring result 55) is stored are not limited to this example and may be changed as appropriate depending on the embodiment. All analysis processes do not have to be executed at once. The analysis processes may be executed at any timing before outputting the monitoring result 55. The data that serves as the information source for the monitoring result 55 may be stored not in units of monitoring result 50, but in units capable of generating monitoring results (50, 55), such as the first image 23, analysis results for each subject S1, and analysis results for each action A1.

[0080] In another example, provisional analysis results may be generated by performing some analysis processing. These provisional analysis results may be, for example, analysis results for each subject S1, analysis results for each action A1, etc. The generated provisional analysis results may be stored in any memory area. From the stored provisional analysis results, analysis results related to the second imaging range 31 may be extracted, and the remaining analysis processing may be performed on the extracted analysis results to generate the monitoring results 55. Extracting analysis results related to the second imaging range 31 may mean extracting analysis results that belong to at least a portion of the second imaging range 31, or extracting analysis results that belong to at least a portion of the range that includes the second imaging range 31. When extracting analysis results that belong to at least a portion of the range that includes the second imaging range 31, monitoring results (part of the monitoring results 50) may be generated from the extracted analysis results, and the monitoring results 55 for the second imaging range 31 may be extracted from the generated monitoring results.

[0081] Furthermore, the first image 23 may be stored in any memory area. From the stored first image 23, the first image 23 related to the second imaging range 31 may be extracted, and the monitoring result 55 may be generated by performing an analysis process on the extracted first image 23. Similar to the analysis result described above, extracting the first image 23 related to the second imaging range 31 may mean extracting the first image 23 in which at least a part belongs to the second imaging range 31, or extracting the first image 23 in which at least a part belongs to the range encompassing the second imaging range 31. When extracting the first image 23 in which at least a part belongs to the range encompassing the second imaging range 31, the monitoring result (part of the monitoring result 50) may be generated from the extracted first image 23, and the monitoring result 55 of the second imaging range 31 may be extracted from the generated monitoring result.

[0082] When saving data capable of generating monitoring results 50 (first image 23, provisional analysis results, etc.) in order to obtain monitoring results 55 in the second imaging range 31, at least a portion of the monitoring results 50 may be fictitious. That is, extracting the monitoring results 55 in the second imaging range 31 from the monitoring results 50 may include obtaining the monitoring results 55 generated from data saved as data capable of generating monitoring results 50 (first image 23, provisional analysis results, etc.). At least a portion of the monitoring results 50 may not actually be generated. The storage area for saving the information source of the monitoring results 55 (monitoring results 50, provisional analysis results, first image 23, etc.) may be, for example, the memory resources of the information processing device 1, the memory resources of another computer, or an external storage device. The external storage device may include a data server such as a NAS.

[0083] (Output range) The output range of the monitoring results 55 can be determined arbitrarily. For example, all monitoring results obtained in the second imaging range 31 may be output as monitoring results 55. Alternatively, some of the monitoring results obtained in the second imaging range 31 that satisfy predetermined conditions may be selectively output as monitoring results 55. The predetermined conditions can be defined arbitrarily.

[0084] For example, analysis information 571 is obtained during the period monitored by one or more first imaging devices 2. The results of analyzing the behavior A1 of subject S1 during a specific period may be shown. That is, the predetermined conditions may include the fact that it falls within a specific period, and in the output of the monitoring result 55, analysis results obtained in the second imaging range 31 that fall within the specific period may be selectively output.

[0085] The specified period may be specified by any method. For example, the specified period may be specified manually by the operator. The operator may specify the specified period by directly operating the information processing device 1, or by indirectly giving instructions to the information processing device 1 from another computer (terminal, etc.) connected to the information processing device 1. Alternatively, the specified period may be predetermined (for example, A hours from the present, B days from the present, etc.).

[0086] The analysis results (analysis information 571) corresponding to a specific period may be appropriately selected depending on the embodiment. In one example, whether or not a result corresponds to a specific period may be determined by using time 574. That is, among the analysis results in the second imaging range 31, analysis results for which time 574 corresponds to a specific period may be selected as output targets. If time 574 is configured to indicate time (period), corresponding to a specific period may mean that at least a part of the time indicated by time 574 overlaps with the specific period. Conversely, corresponding to a specific period may mean that the time indicated by time 574 is included within the specific period, and analysis results for which only a part of the time indicated by time 574 overlaps with the specific period may be excluded. In one example, if the first image 23 is saved, the first image 23 corresponding to the specific period may be selected from the saved first image 23, and the analysis results for the specific period may be generated from the selected first image 23. If provisional analysis results are saved, provisional analysis results corresponding to a specific period may be selected from the saved provisional analysis results, and the analysis results for the specific period may be generated from the selected analysis results. In addition, in one example, data that has been in use for a certain period may be discarded, and only data corresponding to a specific period (monitoring results 50, provisional analysis results, first image 23, etc.) may be saved. By obtaining monitoring results 55 including analysis information 571 from the saved data, analysis results for a specific period may be obtained without using time 574. According to one example of this embodiment, it is possible to focus on the action A1 of the subject S1 performed during a specific period and provide analysis results (analysis information 571) of that action A1.

[0087] In one example, the analysis information 571 may show the results of analyzing the behavior A1 of subjects S1 having specific attributes among subjects S1 monitored by one or more first imaging devices 2. That is, the predetermined conditions may include that the subject S1 has specific attributes, and in the output of the monitoring result 55, the analysis results relating to subjects S1 having specific attributes may be selectively output from the analysis results obtained in the second imaging range 31.

[0088] Specific attributes may be specified in any way. For example, specific attributes may be specified manually by an operator. The operator may specify specific attributes by directly operating the information processing device 1, or by indirectly giving instructions to the information processing device 1 from another computer (terminal, etc.) connected to the information processing device 1. Alternatively, specific attributes may be predetermined. If the subject S1 is a human being, having specific attributes may include, for example, being of a specific age, belonging to a specific group, living in a specific residential area, having specific hobbies / interests, or having performed a specific action. For example, whether or not a specific action has been performed may be determined based on whether or not a history indicating that the specific action has been performed is included in the action history.

[0089] The analysis results (analysis information 571) of a subject S1 having specific attributes may be appropriately selected depending on the embodiment. In one example, attribute 575 may be used to determine whether or not the subject S1 has specific attributes. That is, among the analysis results in the second imaging range 31, analysis results in which attribute 575 satisfies predetermined attribute conditions (including specific attributes) may be selected as output targets. Having specific attributes includes performing a specific action, and the specific action is monitored. If attribute 575 is included in the behavior A1 of the object being viewed, attribute 575 may be linked to the analysis results of behavior A1. Based on the identification result of subject S1 and the analysis results of behavior A1, it may be determined whether the same subject S1 that obtained analysis results in the second imaging range 31 performed a specific behavior. Depending on this determination result, the analysis results relating to the subject S1 that performed the specific behavior may be selected from the analysis results obtained in the second imaging range 31. For example, if the first image 23 is saved, the analysis results of subject S1 having a specific attribute may be generated by analyzing the behavior A1 of subject S1 having a specific attribute in the saved first image 23. If provisional analysis results are saved, provisional analysis results of subject S1 having a specific attribute may be selected from the saved provisional analysis results, and the analysis results of subject S1 having a specific attribute may be generated from the selected analysis results. Also, for example, by making only subject S1 having a specific attribute the target of the analysis process, the analysis results of subject S1 having a specific attribute may be obtained without using attribute 575. According to one example of this embodiment, it is possible to focus on a subject S1 having specific attributes and then provide the analysis results (analysis information 571) of the behavior A1 of that subject S1.

[0090] In one example, the above period and attribute conditions may be used in combination. That is, the analysis information 571 may show the results of analyzing the behavior A1 of a subject S1 having specific attributes during a specific period. That is, the predetermined conditions may be that the timing of the execution of behavior A1 falls within a specific period and the subject S1 has specific attributes. In the output of the monitoring result 55, the analysis results concerning the subject S1 having specific attributes during a specific period may be selectively output from the analysis results obtained in the second imaging range 31.

[0091] Furthermore, analysis results (analysis information 571) that satisfy predetermined conditions may also be used to derive related information 572. For example, related information 572 may be obtained based on the results of analyzing the behavior A1 of subject S1 during a specific period within the period monitored by one or more first imaging devices 2. Related information 572 may be obtained based on the results of analyzing the behavior A1 of subject S1 having specific attributes among the subjects S1 monitored by one or more first imaging devices 2. Also, related information 572 may be obtained based on the results of analyzing the behavior A1 of subject S1 having specific attributes during a specific period.

[0092] (output format) The extracted monitoring results 55 may be output in any format. Known formats such as text, numbers, figures, and tables may be used for the output format. Text may include characters, numbers, symbols, etc. Figures may include shapes, icons, graphs, maps, etc. For example, the analysis information 571 may show the results of analyzing the behavior A1 of subject S1 using at least one of a heat map 5711, markers 5713, and a funnel diagram 5715.

[0093] Figures 3A to 3C schematically show examples of output formats for monitoring results 55 (analysis information 571) according to this embodiment. Figure 3A assumes a scenario in which a heat map 5711 is adopted as the output format. Figure 3B assumes a scenario in which a marker 5713 is adopted as the output format. Figure 3C assumes a scenario in which a funnel diagram 5715 is adopted as the output format. Figures 3A to 3C assume a scenario in which the extracted monitoring results 55 are reflected in the second image 33 as the output method.

[0094] As shown in Figure 3A, the heatmap 5711 may be configured to divide the image (second image 33) into multiple sections and indicate the degree of behavior A1 for each section using color coding (shades of color, differences in color, etc.). In one example, the degree of behavior A1 may include at least one of the following: statistics on the duration of behavior A1 by subject S1, statistics on the frequency of behavior A1 by subject S1, and statistics on the number of subjects S1 that performed behavior A1. The method of dividing the image and the dimensions of each section are not particularly limited and may be determined as appropriate depending on the embodiment. The correspondence between the degree and the color coding is not particularly limited and may be determined as appropriate depending on the embodiment. For example, the heat map 5711 may be configured to show areas with a higher degree of behavior A1 in darker red. When multiple types of behavior A1 are analyzed, the heat map 5711 may be configured to show the degree of one or more types of behavior A1. In one example, one or more types of behavior A1 may be selected from multiple types of behavior A1, and the heat map 5711 may be configured to show the degree of the selected type of behavior A1. The types of behavior A1 to be displayed may be selected by any method. For example, the types of behavior A1 to be displayed may be selected manually by the operator.

[0095] As shown in Figure 3B, the marker 5713 may be configured to indicate the location where action A1 was performed. In one example, the marker 5713 may be configured to display at least one of the degree and content of action A1 for each location. The marker 5713 may use visual elements such as text, numbers, diagrams, and tables. Diagrams may include shapes, icons, etc. Diagrams may include color coding according to the type, degree, etc. of action A1. In one example, the marker 5713 may be a callout with a tail containing text. The tip of the tail may indicate the location of action A1. The text in the callout may indicate the content and time of action A1. However, the configuration of the marker 5713 is not limited to these examples and may be modified as appropriate depending on the embodiment. Other presentation formats than callouts may be used for the marker 5713. Furthermore, marker 5713 may be configured to indicate at least one of the degree and content of action A1 by visual elements other than text (e.g., numerical values, color coding, etc.) without containing any text. Note that marker 5713 may be replaced with, for example, a callout, pin, tag, flag, indicator, pointer, etc. In one example, a heatmap 5711 may be formed by integrating marker 5713 in the time direction.

[0096] As shown in Figure 3C, the funnel diagram 5715 may be configured to decompose the process into multiple stages and represent the variation in the degree of each stage. Each stage may correspond to the behavior of the subject S1. At least one of the behaviors in the multiple stages may be analyzed as behavior A1. Behaviors other than behavior A1 may be obtained from other information sources other than the first image 23. The degree of behavior in the target stage may be indicated by numerical values ​​measured in the target stage (duration, frequency, number of subjects S1, etc.) or by the transition probability from a stage prior to the target stage. The previous stage may be, for example, the previous stage, the first stage, etc. The process and the behaviors in each stage are not particularly limited and may be appropriately selected depending on the embodiment. As a specific example, if the environment E1 includes the space of a store and the subject S1 includes a customer, the process shown by the funnel diagram 5715 may include the product purchase process. The purchase process may include entering the imaging range, browsing, approaching (stopping in front of the product, etc.), grasping, and purchasing as the behaviors in each stage. Entry, browsing, approach, and grasping may be analyzed as behavior A1, and the degree of each behavior may be obtained as a result of the analysis of behavior A1. The imaging range may be a first imaging range 21, a partial range within the first imaging range 21, or a second imaging range 31. Whether or not the second imaging range 31 has been entered may be determined based on the location (position 573) of the subject S1. The degree of purchase may be obtained from purchase information. Thus, the funnel diagram 5715 may be configured to represent the variation in the degree of each behavior from entering the imaging range to making a purchase.

[0097] As illustrated in Figures 3A to 3C, according to one example of this embodiment, visual analysis results (analysis information 571) can be provided by at least one of the heatmap 5711, markers 5713, and funnel diagram 5715. Note that each output format may be used in combination. Furthermore, the situations in which each output format can be adopted are not limited to situations in which the output method is to reflect the monitoring results 55 in the second image 33. In another example, each output format may also be adopted in situations in which the output method is to reflect the monitoring results 55 in the first image 23.

[0098] [Action] Action A1 may include any type of action that the subject S1 can perform and that can be analyzed from the captured image (first image 23). Action may include at least one of an action and a reaction. Action A1 may include actions that can interact with objects present in the environment E1. Objects present in the environment E1 may include objects, living organisms, etc. Living organisms may include humans, other living organisms, etc. Action A1 of the monitored subject may be appropriately selected depending on the embodiment of the subject S1, etc.

[0099] Figure 4 schematically shows an example of the monitored behavior A1 according to this embodiment. In one example, the subject S1 may include at least one of a living organism and a device S13. The living organism may include at least one of a human S11 and another living organism S12 other than a human S11.

[0100] (human) If the subject S1 includes a human S11, the action A1 of the subject S1 may include the first action A11 of the human S11. The human S11 to be monitored may be appropriately selected depending on the embodiment. For example, if the environment E1 is a store space, the human S11 may include customers (visitors), store clerks, patrol officers, etc. According to this example, efficient access to the monitoring results 55 of the target area can be expected for the first action A11 of the human S11.

[0101] The first action A11 of human S11 may include any type of action and reaction that human S11 is capable of performing. For example, the first action A11 of human S11 may include at least one of browsing, approaching, and finger interaction actions. Browse may involve directing one's gaze towards an object present in environment E1. Approaching may involve positioning oneself near an object present in environment E1. Positioning oneself near an object may mean being below or within a threshold distance from the object. The threshold (distance) for proximity may be defined as appropriate depending on the embodiment. Approaching an object may include stopping near the object. Finger interaction actions may include any action performed with the fingers on an object present in environment E1. Finger interaction actions may include, for example, reaching out to the object, grasping (taking) the object, touching the object with the hand, pushing the object, pointing at the object, etc. According to one example of this embodiment, efficient access to the monitoring results 55 of the object range can be expected for at least one of browsing, approaching, and finger interaction actions. The types of first action A11 are not limited to the examples above and may be modified as appropriate depending on the embodiment. In another example, first action A11 may include entering the imaging range (first imaging range 21, a partial area within the first imaging range 21, second imaging range 31), kicking, sitting, touching (including pushing) with the body, etc., in place of or together with at least one of the above actions. First action A11 may also include actions that make contact with the object with at least a part of the body.

[0102] With respect to the first action A11 of human S11, the output range of the monitoring result 55 may be arbitrarily determined. For example, all monitoring results obtained in the second imaging range 31 may be output as monitoring result 55. From the monitoring results obtained in the second imaging range 31, some monitoring results that satisfy predetermined conditions may be selectively output as monitoring result 55. The predetermined conditions may be arbitrarily defined. The predetermined conditions may include conditions relating to at least one of the above period and attributes.

[0103] In one example, the monitoring result 55 may be a monitoring result 55 relating to the first action A11 of a person S11 that satisfies predetermined attribute conditions among the people S11 monitored by one or more first imaging devices 2. Satisfying predetermined attribute conditions is an example of having specific attributes. According to this example embodiment, it is possible to provide monitoring results 55 relating to the first action A11 of a person S11 that satisfies predetermined attribute conditions.

[0104] The predetermined attribute conditions may be determined as appropriate depending on the embodiment. In one example, the predetermined attribute conditions may include the execution of the second action A21. The second action A21 may be different from the first action A11. The difference between the second action A21 and the first action A11 is (I) second action A2 (1) The second action A21 is of the same type as the first action A11, but at least one of the time (time 574) and place (location 573) in which the second action A21 was performed is different from that of the first action A11, and (2) the second action A21 is of a different type than the first action A11. Performing the second action A21 is an example of performing a specific action. For example, if the second action A21 is included in the action A1 being monitored, it may be determined, based on the identification result of the subject S1 and the analysis result of action A1, whether the same person S11 from whom the monitoring result 55 is obtained in the second imaging range 31 performed the second action A21. Depending on this determination result, the monitoring result 55 of the person S11 who performed the second action A21 may be selected from the monitoring results 55 obtained in the second imaging range 31. According to one example of this embodiment, it is possible to focus on the person S11 who performed the second action A21 and provide the monitoring result 55 regarding the first action A11 of that person S11.

[0105] The second action A21 may be appropriately selected depending on the embodiment. In one example, the second action A21 may include at least one of the following: (1) browsing a specific area within environment E1, (2) approaching a specific area within environment E1, and (3) interacting with a specific area within environment E1 using fingers. The specific area may or may not overlap with the second imaging range 31 at least partially. The specific area may include the area where objects (objects, living organisms, etc.) exist in environment E1. In one example, similar to a specific period, the specific area may be manually specified by the operator or may be predetermined (for example, given as a predetermined value). Also, in one example, if the second action A21 is detected in the first image 23 and the location (position 573) of the detected second action A21 at least partially overlaps with the specific area, the second action A21 may be evaluated as having been performed in the specific area. If the location of the second action A21 does not overlap with the specific area, the second action A21 may be evaluated as not having been performed in the specific area. Alternatively, the second action A21 may be evaluated as having been performed in the specific area only if the location of the second action A21 is included within the specific area. In other words, if the location of the second action A21 only partially overlaps with or does not completely overlap with the specific area, the second action A21 may be evaluated as not having been performed in the specific area. According to one example of this embodiment, it is possible to narrow down the results to human S11 who performed at least one of the actions (1) to (3) in the specific area as the second action A21, and then provide the monitoring result 55 regarding the first action A11 of that human S11.

[0106] For example, if the environment E1 includes the space of a store, the people S11 include customers, and the first action A11 includes an action toward a product, the sales location of a specific product may be designated as a specific area. In this case, for a customer who performs a second action A21 toward a specific product, monitoring results 55 of the first action A11 toward the product as captured in the second imaging range 31 can be provided. As a specific example, if the first action A11 is browsing a product and the second action A21 is grasping a specific product, monitoring results 55 regarding browsing a product as captured in the second imaging range 31 can be provided for a customer who grasped a specific product.

[0107] In one example, the monitoring result 55 may be the monitoring result 55 relating to the first action A11 of human S11 during a specific period within the period monitored by one or more first imaging devices 2. The monitoring result 55 may also be the monitoring result 55 relating to the first action A11 of human S11 that satisfies predetermined attribute conditions during a specific period.

[0108] (Other organisms) If the subject S1 includes other organisms S12, the behavior A1 of the subject S1 may include the first behavior A12 of the other organisms S12. The other organisms S12 may be appropriately selected depending on the embodiment. Other organisms S12 may include, for example, pets such as dogs and cats. According to one example of this embodiment, efficient access to the monitoring results 55 of the target range can be expected for the first behavior A12 of the other organisms S12.

[0109] The first action A12 of the other organism S12 may include any type of action and reaction that the other organism S12 is capable of performing. In one example, the first action A12 of the other organism S12 may include at least one of proximity and contact. Proximity of the other organism S12 may be defined in the same way as proximity of a human S11. Contact may be at least a part of the body of the other organism S12 touching an object present in the environment E1. According to one example of this embodiment, efficient access to the monitoring results 55 of the range of the object can be expected for at least one of proximity and contact. Note that the types of first action A12 are not limited to the above example and may be changed as appropriate depending on the embodiment. In another example, the first action A12 may include entering the imaging range (first imaging range 21, a partial range within the first imaging range 21, second imaging range 31), etc., in place of or together with at least one of the above actions.

[0110] Similar to the first action A11 of human S11 described above, the output range of the monitoring result 55 for the first action A12 of other organisms S12 may be arbitrarily determined. In one example, the monitoring result 55 may be the monitoring result 55 for the first action A12 of other organisms S12 that satisfy predetermined attribute conditions among the other organisms S12 monitored by one or more first imaging devices 2. According to this example, it is possible to provide monitoring results 55 for the first action A12 of other organisms S12 after narrowing the scope to other organisms S12 that satisfy predetermined attribute conditions.

[0111] The predetermined attribute conditions may be determined as appropriate depending on the embodiment. In one example, the predetermined attribute conditions may include having performed the second action A22. Similar to the relationship between the first action A11 and the second action A21 described above, the second action A22 may be different from the first action A12. In one example, if the second action A22 is included in the action A1 of the subject being monitored, it may be determined, based on the identification result of the subject S1 and the analysis result of action A1, whether the same organism S12 that is the same as the other organism S12 from which monitoring results 55 are obtained in the second imaging range 31 has performed the second action A22. In accordance with this determination result, the monitoring result 55 of the other organism S12 that performed the second action A22 may be selected from the monitoring results 55 obtained in the second imaging range 31. According to one example of this embodiment, it is possible to provide monitoring results 55 regarding the first action A12 of the other organism S12 after narrowing it down to the other organism S12 that performed the second action A22.

[0112] The second action A22 may be appropriately selected depending on the embodiment. In one example, the second action A22 may include at least one of (1) approaching a specific area within the environment E1, and (2) contacting a specific area within the environment E1. The specific area may be the same as the specific area in the case of human S11 described above. According to one example of this embodiment, it is possible to provide monitoring results 55 regarding the first action A12 of other organisms S12 by focusing on other organisms S12 that performed at least one of actions (1) and (2) in the specific area as the second action A22.

[0113] In one example, the monitoring result 55 may be the monitoring result 55 concerning the first behavior A12 of another organism S12 during a specific period within the period monitored by one or more first imaging devices 2. The monitoring result 55 may also be the monitoring result 55 concerning the first behavior A12 of another organism S12 that satisfies predetermined attribute conditions during a specific period.

[0114] (Device) If the subject S1 includes the device S13, the action A1 of the subject S1 may include the first action A13 of the device S13. The target device S13 may be appropriately selected depending on the embodiment. In one example, the device S13 may include at least one of a device that moves by manual operation and an autonomously moving robot device. Similar to the movable object described above, a device that moves by manual operation may include, for example, a manually operated vehicle, a remotely operated aircraft, a cart that is pushed or driven, and other devices configured to be movable by manual operation. If the environment E1 is a store space, the cart that is pushed or driven may be a shopping cart. It may include. Autonomous moving robotic devices may include, for example, vehicles capable of autonomous driving, aircraft capable of autonomous flight, and other robotic devices. Aircraft may include drones. Specifically, autonomous moving robotic devices may include cleaning robots, serving robots, transport robots, guidance robots, security robots, and the like. According to one example of this embodiment, the first action A13 of device S13 can be expected to provide efficient access to the monitoring results 55 of the target area.

[0115] The first action A13 of device S13 may include any type of operation and reaction that device S13 can perform. The first action A13 of device S13 may include autonomous actions by device S13 and manual actions by an operator (e.g., the actions of a shopping cart being pushed by a customer). In one example, the first action A13 of device S13 may include at least one of the following: approaching an object, touching an object, grasping an object, facing the object, and performing a predetermined process. Approaching device S13 may be defined in the same way as approaching human S11, etc. Touching device S13 may be at least a part of device S13 touching an object present in the environment E1. Touching may include bumping into it. Facing the object may be the object surface (front, etc.) of device S13 facing the object, and may be defined in the same way as viewing by human S11. The predetermined process may be appropriately selected depending on device S13. In one example, the predetermined process may relate to the capabilities of the device S13. For example, if the device S13 is a cleaning robot, the predetermined process may be cleaning. According to one example of this embodiment, efficient access to the monitoring results 55 of the range of the target can be expected in terms of at least one of approaching, contacting, grasping, facing the direction of the target, and performing the predetermined process. Note that the type of first action A13 is not limited to the above example and may be changed as appropriate depending on the embodiment. In another example, the first action A13 may include entering the imaging range (first imaging range 21, a partial range within the first imaging range 21, second imaging range 31) in place of or together with at least one of the above actions.

[0116] Similar to the first action A11 of human S11 described above, the output range of the monitoring result 55 with respect to the first action A13 of device S13 may be arbitrarily determined. In one example, the monitoring result 55 may be the monitoring result 55 for the first action A13 of device S13 that satisfies predetermined attribute conditions among the devices S13 monitored by one or more first imaging devices 2. According to one example of this embodiment, it is possible to provide the monitoring result 55 for the first action A13 of device S13 after narrowing it down to devices S13 that satisfy predetermined attribute conditions.

[0117] The predetermined attribute conditions may be determined as appropriate depending on the embodiment. In one example, the predetermined attribute conditions may include the execution of the second action A23. Similar to the relationship between the first action A11 and the second action A21 described above, the second action A23 may be different from the first action A13. In one example, if the second action A23 is included in the action A1 of the subject to be monitored, it may be determined, based on the identification result of the subject S1 and the analysis result of action A1, whether the same device S13 that obtained the monitoring result 55 in the second imaging range 31 executed the second action A23. Depending on this determination result, the monitoring result 55 of the device S13 that executed the second action A23 may be selected from the monitoring results 55 obtained in the second imaging range 31. According to one example of this embodiment, it is possible to provide the monitoring result 55 regarding the first action A13 of device S13 after narrowing it down to the device S13 that executed the second action A23.

[0118] The second action A23 may be appropriately selected depending on the embodiment. In one example, the second action A23 may include at least one of the following: (1) approaching a specific area within environment E1, (2) touching a specific area within environment E1, (3) grasping an object located in the specific area within environment E1, (4) facing the direction of the specific area within environment E1, and (5) performing a predetermined process in the specific area within environment E1. The specific area may be the same as the specific area in the case of human S11 described above. According to one example of this embodiment, at least one of the actions (1) to (5) in the specific area By narrowing the focus to the device S13 that performed either of the following as the second action A23, we can provide the monitoring results 55 regarding the first action A13 of device S13.

[0119] In one example, the monitoring result 55 may be the monitoring result 55 relating to the first action A13 of device S13 during a specific period within the period monitored by one or more first imaging devices 2. The monitoring result 55 may also be the monitoring result 55 relating to the first action A13 of device S13 that satisfies predetermined attribute conditions during a specific period.

[0120] [Specific examples of application scenarios] Figure 5 schematically shows an example of environment E1 (operational scene) according to this embodiment. In one example, environment E1 may include the space of store E11. Store E11 may include commercial facilities such as supermarkets, convenience stores, drugstores, and restaurants. Store E11 may also include a complex commercial facility such as a shopping mall. In this example embodiment, efficient access to the monitoring results 55 of the target range can be expected for the subject S1 in store E11.

[0121] In one example, the subject S1 in store E11 may include at least one of a human S11, another living being S12, and a device S13. Human S11 may include a customer S111, a store clerk, a patrol officer, etc. Each first action (A11, A12, A13) may be performed on an object present in store E11. Each second action (A21, A22, A23) may also be performed on an object present in store E11. Objects present in store E11 may include, for example, object T111, living beings, etc. Object T111 may include, for example, merchandise, shelves, advertisements, signs, equipment, decorations, etc. Advertisements may include, for example, digital signage, POP (Point of Purchase) advertisements, posters, etc. Digital signage may include shelf signage. That's fine. The directions may include, for example, a map or floor plan. The facilities may include, for example, entrances, exits, rest areas, etc. Rest areas may include, for example, chairs, sofas, etc. Decorations may include, for example, ornaments, objects, plants, etc. Specific areas may be designated to include, for example, specific product shelves, specific areas of product shelves, sales locations for specific products, locations for displaying specific advertisements, locations for posting specific directions, locations for specific facilities, locations for specific decorations, etc.

[0122] In one example, the subject S1 may include a customer S111 of store E11. The action A1 of the subject S1 may include an action A111 performed by the customer S111 toward an object T111 in store E11. The customer S111 is an example of a human S11. Action A111 is an example of a first action A11 performed toward an object T111. In one example, action A111 may include at least one of viewing the object T111, approaching the object T111, and interacting with the object T111 with fingers. Action A111 may be performed within store E11. If the area outside store E11 is included in the first imaging range 21 of at least one of the first imaging devices 2, action A111 may be performed from outside store E11 toward the object T111 in store E11. The area outside store E11 may include the area surrounding store E11. In one example of this embodiment, monitoring results 55 regarding customer S111's actions A111 towards object T111 in store E11 can be provided. In this example, the monitoring results 55 may include analysis information 571 of customer S111. The provided analysis information 571 may be used as material for considering measures at store E11.

[0123] The output range of the monitoring result 55 for the behavior A1 of subject S1 in store E11 (e.g., behavior A111 of customer S111) can be arbitrarily determined. For example, all monitoring results obtained in the second imaging range 31 may be output as monitoring result 55. Of the monitoring results obtained in the second imaging range 31, some monitoring results that satisfy predetermined conditions may be selectively output as monitoring result 55. The predetermined conditions may be arbitrarily defined. The predetermined conditions may include conditions relating to at least one of the above period and attribute. 1. If it is at least one of the other organism S12 and the apparatus S13, the above conditions may be adopted as the predetermined conditions.

[0124] In one example, the subject S1 may include customers S111 of store E11. The monitoring result 55 may be about the actions A111 of customers S111 who satisfy predetermined purchase conditions among the customers S111 monitored by one or more first imaging devices 2. The predetermined purchase conditions are an example of predetermined attribute conditions. The predetermined purchase conditions may include any conditions relating to the purchase of goods by customer S111. In one example, the predetermined purchase conditions may include conditions relating to the attributes of the purchase of goods by customer S111. That is, the predetermined purchase conditions may be defined to evaluate satisfaction according to the attributes of the purchase of goods by customer S111. Purchase attributes may include, for example, the purchased goods, the purchase amount, the day of the week purchased, the date and time of purchase, etc. As a specific example, satisfying the predetermined purchase conditions may include purchasing a specific product. Satisfying the predetermined purchase conditions may also include purchasing goods exceeding a predetermined amount (threshold). In one example, similar to a specific period, the predetermined purchase conditions may be manually specified by the operator or predetermined (for example, given as a predetermined value).

[0125] In order to evaluate whether the predetermined purchase conditions are met, customer S111's purchase information may be collected as appropriate. The purchase information may be POS (Point of Sale) information. The purchase information may be public The information may be collected using an intelligent method. The purchase information may include, for example, a list of purchased items, the price of each purchased item, the total purchase amount, the date and time of purchase, etc. The purchase information may be appropriately linked to the customer S111 detected from the first image 23. In one example, the purchase information may be linked to customer S111 by the above method which utilizes the identification result of the subject S1 (the result of identifying the individuality or identity of customer S111). Based on the purchase information linked to the target customer S111, it may be evaluated whether the target customer S111 satisfies predetermined purchase conditions. Depending on the result of this evaluation, the monitoring results 55 of customer S111 that satisfy the predetermined purchase conditions may be selected from the monitoring results 55 obtained in the second imaging range 31. The selected monitoring results 55 may be output. According to one example of this embodiment, it is possible to provide monitoring results 55 regarding the customer S111's behavior A111 after narrowing it down to customer S111 that satisfies predetermined purchase conditions.

[0126] [Specific examples of related information] Figure 6 schematically shows an example of related information 572 according to this embodiment. In one example, the subject S1 may include a customer S111 of store E11. Related information 572 may include policy information 580 that proposes recommended measures for store E11 depending on whether the results of analyzing customer S111's behavior A111 (analysis information 571) satisfy predetermined evaluation conditions. Whether or not the predetermined evaluation conditions are satisfied may be determined by any method. In one example, whether or not the predetermined evaluation conditions are satisfied may be determined by a rule-based method. Whether or not the predetermined evaluation conditions are satisfied may also be determined using a trained machine learning model. The trained machine learning model may include large-scale generative models such as large-scale language models, large-scale visual language models, and multimodal models. By inputting the results of analyzing customer S111's behavior A111 into the trained machine learning model and executing the computational processing of the trained machine learning model, the result of determining whether or not the results of analyzing customer S111's behavior A111 satisfy predetermined evaluation conditions can be obtained from the trained machine learning model. If the trained machine learning model includes a large-scale generative model, the policy information 580 may be generated by the large-scale generative model. At least part of the calculation process for determining whether predetermined evaluation conditions are met may be executed on the information processing device 1 or on another computer other than the information processing device 1. In the latter case, the information processing device 1 may obtain from the other computer the result of determining whether the result of analyzing the customer S111's behavior A111 meets predetermined evaluation conditions, or the policy information 580 corresponding to the determination result. According to one example of this embodiment, by including the policy information 580 in the related information 572, policy information 580 indicating policies recommended for adoption at store E11 can be provided as monitoring results 55.

[0127] The predetermined evaluation conditions may be configured to appropriately evaluate the results of the analysis of customer S111's behavior A111 (analysis information 571). For example, the results of the analysis of customer S111's behavior A111 may include the degree of customer S111's behavior A111. The predetermined evaluation conditions may be configured to evaluate the quality of store E11's operations related to customer S111's behavior A111 in the second imaging range 31 in accordance with a comparison of the degree of customer S111's behavior A111 with a benchmark. The policy information 580 may be configured to propose a first policy 581 recommended for improving the operations of store E11 that are evaluated as poor in comparison with the benchmark, or a second policy 582 recommended for strengthening the operations of store E11 that are evaluated as good in comparison with the benchmark.

[0128] The degree of behavior A111 may be measured by an arbitrary index indicating the frequency of behavior A111. For example, the degree of behavior A111 may include at least one of the following: a statistic on the duration of behavior A111, a statistic on the frequency of behavior A111, and a statistic on the number of customers S111 who performed behavior A111. The operation of the store E11 to be evaluated may be determined as appropriate depending on the embodiment. The operation of the store E11 to be evaluated may include, for example, publicizing product placement, product placement, displaying product advertisements, sales promotion, patrols, operation of robotic devices, layout, etc. Publicizing product placement may include placing advertisements, displaying product placement on signage, etc. Displaying product advertisements may include placing physical advertisements such as POP advertisements and posters, displaying advertisements on digital signage, etc. Sales promotion may include issuing coupons, etc. The operation of store E11 may be an example of the conditions of environment E1. The conditions of environment E1 may be the situation of environment E1 to which measures are proposed. The type of behavior A111 and the indicators for the degree of behavior A111 may be appropriately selected according to the operation of the store E11 being evaluated. The first measure 581 may be structured to propose, for example, changing the operation of store E11 which is evaluated as poor, or introducing a new operation. The second measure 582 may be structured to propose, for example, extending the period of operation of store E11 which is evaluated as good, or expanding the operation of store E11 which is evaluated as good to other stores.

[0129] If a higher degree value for behavior A111 is associated with better operation, then being evaluated as better than the benchmark may mean that the degree value of behavior A111 is high relative to the benchmark value. For example, the benchmark value may be used directly for comparison, or a reference value derived from the benchmark value may be used for comparison with the degree value of behavior A111. For instance, if the degree value of behavior A111 is equal to or greater than the benchmark value, then the degree value of behavior A111 may be evaluated as high relative to the benchmark value, and the operation of store E11 may be evaluated as better than the benchmark. If the degree value of behavior A111 is equal to or greater than a first reference value derived from the benchmark value, then the degree value of behavior A111 may be evaluated as high relative to the benchmark value, and the operation of store E11 may be evaluated as better than the benchmark. The first reference value may be derived from the benchmark value by a predetermined calculation. For example, the first criterion value may be the sum of the benchmark value plus a threshold (first threshold), or the difference obtained by subtracting the threshold (first threshold) from the benchmark value. On the other hand, being evaluated as "bad" in comparison to the benchmark may mean that the degree of behavior A111 is evaluated as low based on the benchmark value. For example, if the degree of behavior A111 is less than or equal to the benchmark value, the degree of behavior A111 may be evaluated as low based on the benchmark value, and the operation of store E11 may be evaluated as bad in comparison to the benchmark. If the degree of behavior A111 is less than or equal to the second criterion value derived from the benchmark value, the degree of behavior A111 may be evaluated as low based on the benchmark value, and the operation of store E11 may be evaluated as bad in comparison to the benchmark. Similar to the first criterion value, the second criterion value may be derived from the benchmark value by a predetermined calculation. In the example, the second reference value may be the difference obtained by subtracting the threshold (second threshold) from the benchmark value, or the sum (total) obtained by adding the threshold (second threshold) to the benchmark value.

[0130] If a lower value for the degree of action A111 is associated with a better operation, the quality of the operation may be evaluated in the opposite way to the method described above. That is, being evaluated as better than the benchmark may be associated with a lower value for the degree of action A111 relative to the benchmark value. Being evaluated as worse than the benchmark may be associated with a higher value for the degree of action A111 relative to the benchmark value. Each evaluation method may be the same as the method described above. The benchmark value may be compared directly, or a baseline value derived from the benchmark value may be compared with the degree of action A111. In either case, there may or may not be a numerical range in which the operation is neither evaluated as good nor bad when comparing the degree of action A111 with the benchmark. If there is a numerical range in which the operation is neither evaluated as good nor bad, and if neither the evaluation condition for the first measure 581 or the second measure 582 is set, there may be a range in which measure information 580 (first measure 581, second measure 582) cannot be obtained at store E11. Furthermore, the output range of the monitoring result 55 may also be applied to the degree of action A111 to be compared with the benchmark. In one example, the benchmark may be compared to the degree of action A111 that satisfies at least one of the above conditions of period and attribute. According to one example of this embodiment, recommended measures (first measure 581, second measure 582) in accordance with the comparison with the benchmark can be provided as measure information 580.

[0131] (benchmark) The benchmark value may be set as appropriate depending on the embodiment. For example, the benchmark value may be manually specified by the operator, similar to a specific period, or it may be predetermined (for example, given as a default value). Alternatively, the benchmark value may be determined according to the monitoring result 50 (analysis information 571).

[0132] For example, the predetermined evaluation conditions may be defined to evaluate the quality of store E11's operations related to customer S111's behavior A111, based on a comparison between the degree of customer S111's behavior A111 during a specific period within the period monitored by one or more first imaging devices 2 and a benchmark. The benchmark may be defined based on the degree of customer S111's behavior A111 during periods other than the specific period within the period monitored by one or more first imaging devices 2. Behavior A111 during the other periods defining the benchmark may be related to behavior A111 in the second imaging range 31 compared to the benchmark. For example, behavior A111 during the other periods defining the benchmark may be the same as behavior A111 in the second imaging range 31 compared to the benchmark. The other periods may be specified in any way. Similar to the specific period, the other periods may be specified manually by the operator or may be predetermined (for example, given as a default value). Other periods may be specified according to the specific period (for example, C days before, D weeks before, etc.). Other periods may be specified as appropriate so as not to completely coincide with the specific period. Other periods may partially overlap with the specific period, or they may not overlap with the specific period. Other periods may encompass the specific period, or they may be encompassed by the specific period. Defining a benchmark according to the degree of customer S111's behavior A111 in other periods may include directly adopting the value of the degree of behavior A111 in other periods as the benchmark value, and deriving the benchmark value from the value of the degree of behavior A111 in other periods by a predetermined calculation. The predetermined calculation is not particularly limited and may be determined as appropriate depending on the embodiment. The predetermined calculation may include, for example, adding a predetermined value to the value of the degree of behavior A111, or subtracting a predetermined value from the value of the degree of behavior A111. Furthermore, the degree of behavior A111 in other periods defining the benchmark may be measured at the same location as the degree of behavior A111 in the second imaging range 31 compared to the benchmark, or it may be measured at a different location (for example, another range described later). According to one embodiment, recommended measures (first measure 581, second measure 582) can be provided as measure information 580, based on a comparison of the degree of action A111 between a specific period and another period.

[0133] In one example, the benchmark may be defined according to the degree of customer S111's behavior A111 in a range other than the second imaging range 31. The other range may be specified in any way. Similar to a specific period, the other range may be specified manually by the operator or may be predetermined (e.g., given as a default value). The other range may be specified according to the second imaging range 31 (e.g., a range adjacent to the second imaging range 31). The other range may be specified as appropriate so as not to completely coincide with the second imaging range 31. The other range may partially overlap with the second imaging range 31 or may not overlap with the second imaging range 31. The other range may be included in the second imaging range 31 or may be included within the second imaging range 31. The other range may be specified as appropriate so as not to completely coincide with the target range of the second imaging range 31 (e.g., the placement range of the target product). Defining a benchmark based on the degree of behavior A111 in other ranges may include directly adopting the value of behavior A111 in other ranges as the benchmark value, and deriving the benchmark value from the degree of behavior A111 in other ranges by a predetermined calculation. The predetermined calculation is not particularly limited and may be determined as appropriate depending on the embodiment. The predetermined calculation may include, for example, adding a predetermined value to the degree of behavior A111, or subtracting a predetermined value from the degree of behavior A111. In one example, the degree of behavior A111 of customer S111 during a specific period in the second imaging range 31 may be compared with the benchmark. In this case, the degree of behavior A111 of customer S111 in other ranges that defines the benchmark may be measured during the same period as the specific period, or during a period other than the specific period. As above, the other period may be specified as appropriate so as not to perfectly coincide with the specific period. According to one example of this embodiment, recommended measures (first measure 581, second measure 582) can be provided as measure information 580, based on a comparison of the degree of action A111 between the second imaging range 31 and other ranges.

[0134] (Policies) The measures (e.g., Measure 1 581, Measure 2 582) may be determined as appropriate depending on the implementation. The prescribed evaluation conditions (Actions to be evaluated A111, analysis items, benchmarks, etc.) will serve as the criteria for proposing the measures. The prescribed evaluation conditions may be defined as appropriate depending on the proposed measures.

[0135] (1) Measures to increase the number of customers entering the facility In one example, customer S111's action A111 may include entering the monitoring range of the first imaging device 2. The monitoring range may be the first imaging range 21, or a partial range within the first imaging range 21. In one example, similar to a specific period, the monitoring range may be manually specified by the operator or predetermined (for example, given as a predetermined value). The monitoring range may also be defined according to the elements of the store E11, such as being defined to include the range where the products are placed. The predetermined evaluation conditions may include an entry evaluation condition that evaluates whether the number of customers S111 entering the monitoring range of the first imaging device 2 that overlaps with the second imaging range 31 is large or small. That is, based on the entry evaluation condition, it may be evaluated whether the number of customers S111 entering the monitoring range of the first imaging device 2 that overlaps with the second imaging range 31 is small or not.

[0136] In the entry evaluation conditions, whether the number of entering customers S111 is small or not can be evaluated by any method. For example, the benchmark described above may be used to evaluate whether the number of entering customers S111 is small or not. That is, the entry evaluation conditions may be defined to evaluate whether the number of entering customers S111 is small or not by comparing the number of customers S111 entering the monitoring range of the first imaging device 2 that overlaps with the second imaging range 31 with the benchmark. As a specific example, the results of the analysis of customer S111's behavior A111 (analysis information 571) are used in the second imaging range 31 The statistics may include the number of customers S111 who entered the monitoring range of the first imaging device 2, which overlaps with the other parameters. This statistic of the number of customers S111 is an example of the statistic of the number of subjects S1 who performed action A1. In the entry evaluation condition, this statistic of the number of customers S111 may be compared with a benchmark. The statistic of the number of customers S111 and the benchmark may be compared directly, or they may be converted into a ratio with other actions (such as the number of visitors to store E11) before comparison.

[0137] The analysis of customer S111's behavior A111 (analysis information 571) indicates that a small number of customers S111 entering the monitoring area may be due to poor publicity efforts to raise awareness of the monitoring area among customers S111. For example, the monitoring area may include at least one of the product shelves and the area surrounding them. Thus, a small number of customers S111 entering the monitoring area may correspond to insufficient (poor) publicity regarding product placement. Therefore, if, based on the entry evaluation conditions, it is evaluated that a small number of customers S111 enter the monitoring area of ​​the first imaging device 2 that overlaps with the second imaging area 31, the policy information 580 may be configured to propose a policy to increase the number of customers S111 entering the monitoring area of ​​the first imaging device 2 that overlaps with the second imaging area 31. This policy is an example of the first policy 581.

[0138] Measures to increase the number of customers S111 entering the store may correspond to measures to improve customer S111's awareness of the relevant area (the monitoring area of ​​the first imaging device 2 that overlaps with the second imaging area 31). The configuration of the measure information 580 (measures to increase the number of customers S111 entering the store) is not particularly limited and may be determined as appropriate depending on the embodiment, as long as it can encourage such improvement in awareness. For example, the measure information 580 (measures to increase the number of customers S111 entering the store) may consist of a notification that instructs store staff in natural language to improve awareness of the relevant area, such as, "It seems that not many people are visiting the sales floor. How about displaying information and the location of the sales floor on the digital signage in the store?" Furthermore, if digital signage is installed in the store E11 and the information processing device 1 is connected to this digital signage, the measure information 580 may include a control instruction to output information and the location of the sales floor in the relevant area to the digital signage. Thus, outputting the measure information 580 may include controlling the digital signage to output information and the location of the sales floor in the relevant area. According to one example of this embodiment, it is possible to provide measures to improve awareness of locations with a small number of visitors S111.

[0139] (2) Measures to increase the degree of involvement In one example, customer S111's action A111 may include engagement behavior, such as viewing and approaching object T111 in store E11. This engagement behavior is an example of the first action A11. The predetermined evaluation conditions may include engagement evaluation conditions that assess whether the degree of engagement behavior by customer S111 in the second imaging range 31 is high or low. That is, the degree of engagement behavior by customer S111 in the second imaging range 31 may be evaluated based on the engagement evaluation conditions.

[0140] In the involvement evaluation conditions, the degree of involvement by customer S111 may be evaluated by any method. For example, the benchmark described above may be used to evaluate whether the degree of involvement by customer S111 is high or low. That is, the involvement evaluation conditions may be defined to evaluate whether the degree of involvement by customer S111 is high or low in accordance with a comparison of the degree of involvement by customer S111 in the second imaging range 31 with the benchmark. As a specific example, the results of the analysis of customer S111's behavior A111 (analysis information 571) may include the degree of involvement by customer S111. The degree of involvement is an example of the degree of behavior A1. For example, the degree of involvement by customer S111 may include at least one of the statistics of the duration of the involvement by customer S111, the statistics of the frequency of the involvement by customer S111, and the statistics of the number of customers S111 who performed the involvement. In the involvement evaluation conditions, this degree of involvement by customer S111 may be compared with the benchmark. Customer S111 The degree and benchmark of involvement behavior may be compared directly, or they may be compared after being converted into a ratio with other behaviors (for example, the ratio of the number of customers S111 who viewed or were near a customer S111 to the number of customers S111 who entered the relevant area).

[0141] The analysis of customer S111's behavior A111 (analysis information 571) revealed that a low degree of involvement may result from poor publicity to encourage customer S111 to understand the target object T111. For example, the target object T111 may include products sold at store E11. Thus, a low degree of involvement may correspond to insufficient (poor) product publicity. Insufficient product publicity may include poor product placement, poor presentation of product advertisements, etc. Poor presentation of product advertisements may include poor placement of product advertisements, insufficient number of product advertisements, etc. Therefore, if the degree of involvement is evaluated as low in the second imaging range 31 based on the involvement evaluation conditions, the policy information 580 may be configured to propose policies to increase the degree of involvement. This policy is also an example of the first policy 581.

[0142] Measures to increase the degree of engagement may correspond to measures to improve understanding of the object T111. The structure of the measure information 580 (measures to increase the degree of engagement) is not particularly limited and may be determined as appropriate depending on the embodiment, as long as it can encourage such improvement in understanding. For example, the measure information 580 (measures to increase the degree of engagement) may consist of notifications that instruct store staff in natural language to improve their understanding of the object T111, such as, "The product doesn't seem to stand out. How about changing the placement of the product?", "How about increasing product advertising, such as placing POP advertisements on the shelves?", or "How about decorating to make the product stand out?". If a digital signage is installed in store E11 and the information processing device 1 is connected to this digital signage, the measure information 580 may include control instructions to output product advertisements in the second imaging range 31 to the digital signage. Thus, outputting the measure information 580 may include controlling the digital signage to output product advertisements in the second imaging range 31. Furthermore, the policy information 580 may include instructions to cause the large-scale generative model to propose the current product arrangement and a new product arrangement that makes the products stand out compared to the current arrangement. Thus, outputting the policy information 580 may include causing the large-scale generative model to generate a new product arrangement. According to one example of this embodiment, it is possible to provide a policy to improve the understanding of object T111, which has a low degree of involvement behavior. Note that this policy does not have to be limited to store E11. For example, if there are other stores (sister stores, etc.) related to store E11, the policy proposed in store E11 to increase the degree of involvement behavior towards object T111 may be extended to the other stores related to store E11.

[0143] (3) Sales promotion measures In one example, customer S111's action A111 may include grasping a product in store E11. The product is an example of an object T111. Grasping the product is an example of the finger interaction action in the first action A11. The predetermined evaluation conditions may include a conversion evaluation condition that evaluates whether the number of customers S111 who grasped the target product located in the second imaging range 31 is large or small. That is, based on the conversion evaluation condition, it may be evaluated whether the number of customers S111 who grasped the target product located in the second imaging range 31 is large or small.

[0144] In the conversion evaluation criteria, the number of customers S111 who grasped the target product can be evaluated in any way. For example, the benchmark described above may be used to evaluate whether the number of customers S111 who grasped the target product is large or small. That is, the conversion evaluation criteria may be defined to evaluate whether the number of customers S111 who grasped the target product placed in the second imaging range 31 is large or small, in accordance with a comparison of the number of customers S111 who grasped the target product with the benchmark. As a specific example, the behavior A111 of customer S111 was analyzed. The results (analysis information 571) may include statistics on the number of customers S111 who grasped the target product located in the second imaging range 31. This statistic on the number of customers S111 is an example of statistics on the number of subjects S1 who performed action A1. In the conversion evaluation conditions, this statistic on the number of customers S111 may be compared with a benchmark. The statistic on the number of customers S111 and the benchmark may be compared directly, or they may be converted into a ratio with other actions (entry into the relevant range, browsing, etc.) before comparison.

[0145] The analysis of customer S111's behavior A111 (analysis information 571) revealed that a small number of customers S111 holding the target product may be due to poor management efforts to promote conversion of the target product. Poor management efforts to promote conversion of the target product may correspond to insufficient sales promotion of the target product. Therefore, based on the conversion evaluation conditions, if it is evaluated that a small number of customers S111 are holding the target product located in the second imaging range 31, the policy information 580 may be configured to propose a sales promotion policy for the target product. This policy is also an example of the first policy 581.

[0146] Sales promotion measures for the target product may correspond to measures to increase the number of customers S111 who pick up and purchase the target product. The structure of the measure information 580 (sales promotion measures for the target product) is not particularly limited as long as it relates to sales promotion, and may be appropriately determined according to the embodiment. Known methods may be used for sales promotion. For example, the measure information 580 (sales promotion measures for the target product) may consist of a notification instructing store staff in natural language to promote the sale of the target product, such as "How about issuing a coupon for the target product?" The measure information 580 may include a coupon for the target product. Outputting the measure information 580 may include outputting the coupon to the customer S111's user terminal, digital signage, etc. According to one example of this embodiment, measures can be provided to promote the sale of a target product that is sold in the second imaging range 31 and has a small number of customers S111 who purchase it.

[0147] Furthermore, if a sales promotion for the target product is underway and, based on the conversion evaluation conditions, it is evaluated that a large number of customers S111 have grasped the target product located in the second imaging range 31, the policy information 580 may be configured to propose a policy to continue the ongoing sales promotion for the target product. This policy is an example of the second policy 582. The sales promotion policy for the target product, and the policy to continue the ongoing sales promotion for the target product, do not have to be confined to store E11. For example, if there are other stores (such as affiliated stores) related to store E11, the sales promotion policy for the target product proposed in store E11 may be extended to the other stores related to store E11. If a policy to continue the ongoing sales promotion for the target product is proposed in store E11, the ongoing sales promotion for the target product may be extended to the other stores related to store E11. If the sales promotion is also underway in other stores, the policy to continue the ongoing sales promotion may be extended to those stores as well.

[0148] Furthermore, the action A111 used to evaluate conversion is not limited to grasping a product and may be modified as appropriate depending on the embodiment. In another example, grasping a product in store E11 may be replaced with approaching a product. The number of customers S111 who grasped the target product may be replaced with the degree to which customers S111 approached the target product. The degree of approach to the target product is an example of the degree of action A1. In one example, the degree of approach to the target product by customers S111 may include at least one of the statistics of the duration of approach by customers S111, the frequency of approach by customers S111, and the number of approaching customers S111. If, based on the conversion evaluation conditions, the degree of approach to the target product located in the second imaging range 31 is evaluated as low, the policy information 580 may be configured to propose a sales promotion measure for the target product. Based on the conversion evaluation criteria, if the degree of proximity to the target product located in the second imaging range 31 is evaluated as high, the policy information 580 may be configured to propose a policy to continue the ongoing sales promotion for the target product.

[0149] (4) Measures regarding the frequency of ad output In one example, one or more digital signage displays may be installed in store E11. One or more digital signage displays may display multiple advertisements. The display of each advertisement may be controlled in any way (random, predetermined order, etc.). The display time (time) for each advertisement may be specified as appropriate. When multiple digital signage displays are installed, the relationship between each digital signage display and the advertisements it displays is not particularly limited and may be determined as appropriate depending on the embodiment. Each digital signage display may be controlled to display all advertisements. At least some of the multiple digital signage displays may be controlled to display only some of the multiple advertisements. Customer S111's action A111 may include viewing advertisements displayed on the digital signage. Digital signage is an example of an object T111. Viewing advertisements displayed on digital signage is an example of viewing in the first action A11. If multiple advertisements are displayed on the target digital signage, the advertisements viewed by customer S111 may be identified based on the viewing time (time 574) and the display time of each advertisement. The predetermined evaluation conditions may include viewing evaluation conditions that evaluate whether customer S111 viewed a large or small amount of the target advertisements displayed on the digital signage deployed in the second imaging range 31. That is, based on the viewing evaluation conditions, it may be evaluated whether customer S111 viewed a large or small amount of the target advertisements displayed on the digital signage deployed in the second imaging range 31.

[0150] In the viewing evaluation conditions, the extent to which the target advertisement was viewed may be evaluated by any method. For example, the benchmark described above may be used to evaluate whether the target advertisement was viewed a lot or a little. That is, the viewing evaluation conditions may be defined to evaluate whether the extent to which the target advertisement was viewed a lot or a little is determined by comparing the extent to which customer S111 viewed the target advertisement displayed on the digital signage deployed in the second imaging range 31 with the benchmark. As a specific example, the results of the analysis of customer S111's behavior A111 (analysis information 571) may include the extent to which customer S111 viewed the target advertisement. The extent to which the target advertisement was viewed is an example of the extent of behavior A1. For example, the extent to which the target advertisement was viewed may include at least one of the statistics of the duration of viewing the target advertisement, the statistics of the frequency of viewing the target advertisement, and the statistics of the number of customers S111 who viewed the target advertisement. In the viewing evaluation conditions, this extent to which the target advertisement was viewed may be compared with the benchmark. The degree of browsing and benchmarking by customer S111 may be compared directly, or they may be compared after being converted into a ratio with other behaviors.

[0151] Analysis of customer S111's behavior A111 (analysis information 571) indicates that a high degree of viewing of the target advertisement may occur due to the effective presentation of the advertisement for the target product. Therefore, based on the viewing evaluation conditions, if it is evaluated that a high degree of viewing has occurred of the target advertisement displayed on the digital signage deployed in the second imaging range 31, the policy information 580 may be configured to propose a policy to increase the frequency of displaying the target advertisement. This policy is an example of the second policy 582. Increasing the frequency of displaying the target advertisement may include increasing the number of times the target advertisement is displayed, extending the display time of the target advertisement, extending the display period of the target advertisement, etc.

[0152] If it relates to increasing the output frequency of the target advertisement, the configuration of the policy information 580 (a policy to increase the output frequency of the target advertisement) is not particularly limited and may be determined as appropriate depending on the embodiment. For example, policy information 580 (a policy to increase the output frequency of the target advertisement) may consist of a notification instructing the store clerk to increase the output frequency of the target advertisement in natural language, such as "Why don't you try increasing the output frequency of the target advertisement?". When the information processing device 1 is connected to one or more digital signage displays, policy information 580 may include a control command to increase the output frequency of the target advertisement. Thus, outputting policy information 580 may include controlling one or more digital signage displays to increase the output frequency of the target advertisement. According to one example of this embodiment, among the multiple advertisements output on one or more digital signage displays, We can provide measures to increase the frequency of displaying ads with a high view count.

[0153] Furthermore, if, based on the viewing evaluation conditions, the target advertisement displayed on the digital signage deployed in the second imaging range 31 is evaluated as having been viewed infrequently, the policy information 580 may be configured to propose a policy to reduce the frequency of displaying the target advertisement. This policy is an example of the first policy 581. Reducing the frequency of displaying the target advertisement may include reducing the number of times the target advertisement is displayed, shortening the display time of the target advertisement, shortening the display period of the target advertisement, or stopping the display of the target advertisement altogether. Stopping the display of the target advertisement may include replacing the target advertisement with another advertisement. The policies to increase the frequency of displaying the target advertisement and the policies to decrease the frequency of displaying the target advertisement do not have to be limited to store E11. For example, if there are other stores (sister stores, etc.) related to store E11, the policies to increase the frequency of displaying the target advertisement or to decrease the frequency of displaying the target advertisement proposed in store E11 may be extended to other stores related to store E11.

[0154] (Other specific examples) Furthermore, the scenarios in which the policy information 580 can be provided as related information 572 are not limited to scenarios in which the environment E1 includes the space of store E11 and the subject S1 includes customer S111. In the method of providing policy information 580 shown in Figure 6 above, at least one of customer S111 and store E11 may be changed. Customer S111 may be replaced with a subject S1 other than customer S111 (a person S11 other than customer S111, another living organism S12, a device S13, etc.). Store E11 may be replaced with an environment E1 other than store E11.

[0155] In other words, the policy information 580 may be configured to propose recommended measures in environment E1 depending on whether the results of the analysis of subject S1's behavior A1 satisfy predetermined evaluation conditions. The results of the analysis of subject S1's behavior A1 may include the degree of subject S1's behavior A1. The predetermined evaluation conditions may be defined to evaluate the quality of the environment E1 related to subject S1's behavior A1 depending on a comparison between the degree of subject S1's behavior A1 in the second imaging range 31 and a benchmark. The policy information 580 may be configured to propose a first measure 581 recommended to improve the environment E1 that is evaluated as poor by comparison with the benchmark, or a second measure 582 recommended to strengthen the environment E1 that is evaluated as good by comparison with the benchmark. The predetermined evaluation conditions may be defined to evaluate the quality of the environment E1 related to subject S1's behavior A1 depending on a comparison between the degree of subject S1's behavior A1 in a specific period of time within the period monitored by one or more first imaging devices 2 and a benchmark. The benchmark may be defined according to the degree of behavior A1 of the subject S1 during periods other than a specific period within the period monitored by one or more first imaging devices 2. The benchmark may also be defined according to the degree of behavior A1 of the subject S1 in areas other than the second imaging range 31.

[0156] In one example, subject S1 may include human S11. Human S11's first action A11 may include suspicious behavior. Suspicious behavior may be defined as appropriate depending on the embodiment. Suspicious behavior may include fraudulent acts, unusual acts, damaging acts, etc. The predetermined evaluation conditions may include security evaluation conditions that evaluate whether the degree of suspicious behavior by human S11 in the second imaging range 31 is high or low. In one example, the above benchmark may be used to evaluate the security evaluation conditions. That is, the security evaluation conditions may be defined to evaluate whether the degree of suspicious behavior by human S11 in the second imaging range 31 is high or low in accordance with a comparison of the degree of suspicious behavior by human S11 in the second imaging range 31 with the benchmark. The degree of suspicious behavior is an example of the degree of action A1. If, based on the security evaluation conditions, the degree of suspicious behavior by human S11 in the second imaging range 31 is evaluated as high, the policy information 580 may be configured to propose a policy to increase the frequency of patrols in the second imaging range 31. This measure is an example of Measure 581. Patrols may be carried out by human operators or by robotic devices (security robots, etc.). It is also possible that the policy information 580 consists of a notification encouraging an increase in the frequency of patrols, or a control instruction causing the robot device to increase the frequency of patrols. According to one example of this embodiment, it is possible to provide a policy to strengthen patrols in places where suspicious activity is frequent. In one example, environment E1 may include the space of store E11. In this case, it is possible to provide a policy to strengthen patrols of store E11 in places where suspicious activity is frequent.

[0157] In one example, environment E1 may include the space of store E11. Subject S1 may include the store clerk of store E11. The clerk is an example of a human S11. The clerk's actions may include grasping merchandise in store E11. Merchandise is an example of an object T111. Grasping merchandise is an example of the interaction of fingers in the first action A11. The predetermined evaluation conditions may include a stocking evaluation condition that evaluates whether the degree to which the clerk grasped the target merchandise placed in the second imaging range 31 is high or low. In one example, the above benchmark may be used to evaluate the stocking evaluation condition. That is, the stocking evaluation condition may be defined to evaluate whether the degree to which the clerk grasped the target merchandise placed in the second imaging range 31 is high or low, in accordance with a comparison between the degree to which the clerk grasped the target merchandise and the benchmark. The degree to which the target merchandise was grasped is an example of the degree of action A1. Based on the stocking evaluation conditions, if it is evaluated that a large number of target products placed in the second imaging range 31 are grasped by store staff, the policy information 580 may be configured to propose a policy to increase the sales volume of the target products. This policy is an example of the second policy 582. Increasing the sales volume of the target products may include at least one of increasing the inventory of the target products and increasing the number of target products displayed in the store. The policy information 580 may consist of a notification encouraging an increase in the sales volume of the target products. The policy information 580 may consist of an order instruction for ordering the target products. The policy information 580 may consist of a manufacturing instruction that commands the manufacturing equipment to produce the target products. The policy information 580 may consist of a control instruction that commands a robotic device (such as a transport robot) to stock the target products. According to one example of this embodiment, it is possible to provide a policy to strengthen sales of target products that are frequently grasped and stocked by store staff among the products sold in the second imaging range 31. This policy may be extended to other stores as well.

[0158] In one example, the subject S1 may include the device S13. The first action A13 of the device S13 may include contact with the object. Contact may include collision. In one example, the object may include fixtures such as shelves. If the environment E1 includes the space of a store E11, the shelves may include merchandise shelves for displaying goods. The predetermined evaluation conditions may include a contact evaluation condition that evaluates whether the degree to which the device S13 contacts the object present in the second imaging range 31 is high or low. In one example, the above benchmark may be used to evaluate the contact evaluation condition. That is, the contact evaluation condition may be defined to evaluate whether the degree to which the device S13 contacts the object present in the second imaging range 31 is high or low, in accordance with a comparison between the degree to which the device S13 contacts the object present in the second imaging range 31 and the benchmark. The degree to which the device contacts the object is an example of the degree of action A1. Based on the contact evaluation conditions, if it is evaluated that the device S13 is frequently in contact with an object in the second imaging range 31, the policy information 580 may be configured to propose measures to suppress contact by the device S13. This measure is an example of the first policy 581. Measures to suppress contact by the device S13 may include, for example, changing the movement path of the device S13 or changing the state of the environment E1. If the device S13 is an autonomously moving robot, changing the movement path of the device S13 may include setting a no-entry zone. If the object includes furniture, changing the state of the environment E1 may include changing the layout of the furniture. The policy information 580 may also consist of a notification prompting the suppression of contact by the device S13. If the device S13 is an autonomously moving robot, the policy information 580 may consist of a control instruction that commands the robot to avoid the object (e.g., prohibiting entry into the area encompassing the object). If the target includes furniture and a device for moving the furniture (such as a robot) is deployed in environment E1, the policy information 580 may consist of a control instruction that commands the device to change the layout of the furniture. According to one example of the configuration, measures can be provided to suppress contact of the device S13.

[0159] [Method for determining spatial relationships] Each range information (25, 35) may be configured to directly or indirectly indicate the imaging range (21, 31) of each imaging device (2, 3). In one example, each range information (25, 35) may include each image (23, 33) or a representation of the environment E1 estimated from each image (23, 33). This allows each range information (25, 35) to indirectly indicate the imaging range (21, 31) of each imaging device (2, 3). Alternatively, each range information (25, 35) may include the orientation and field of view of each imaging device (2, 3). This allows each range information (25, 35) to directly indicate the imaging range (21, 31) of each imaging device (2, 3). In one example, the field of view of each imaging device (2, 3) may be known from the performance of each imaging device (2, 3). If the position of the imaging devices (2, 3) is fixed, the position of the imaging devices (2, 3) may be known. If the orientation of the imaging devices (2, 3) is also fixed, the orientation of the imaging devices (2, 3) may also be known. If the imaging devices (2, 3) are movable and their orientation is variable, the orientation of the imaging devices (2, 3) may be determined by any method. Known methods may be used to determine the orientation of the imaging devices (2, 3). Similarly, if the orientation of the imaging devices (2, 3) is changeable, the orientation of the imaging devices (2, 3) may be determined by any method. Known methods may be used to determine the orientation of the imaging devices (2, 3). The relative positional relationship 4 of the second imaging range 31 may be determined by any method according to the configuration of each range information (25, 35).

[0160] Figure 7 schematically shows an example of a method for determining the positional relationship 4 according to this embodiment. In one example, the first range information 25 may include a three-dimensional representation 250 of the environment E1 constructed by providing the first images 23 captured by one or more first imaging devices 2 to a three-dimensional reconstruction model 60. The construction of the three-dimensional representation 250 may be performed on the information processing device 1 or on a computer other than the information processing device 1. The three-dimensional representation 250 may be stored on the information processing device 1 or on another computer. If the three-dimensional representation 250 is stored on another computer, access to the three-dimensional representation 250 may be an example of obtaining the first range information 25. The three-dimensional representation 250 may be appropriately configured to represent the three-dimensional space of the environment E1. Known data formats such as three-dimensional coordinates may be used for the three-dimensional representation 250.

[0161] The second range information 35 may include a second image 33 captured by the second imaging device 3. Acquiring the second image 33 may be one example of acquiring the second range information 35. Identifying the relative positional relationship 4 of the second imaging range 31 of the second imaging device 3 with respect to the first imaging range 21 of each of the one or more first imaging devices 2 may be performed by matching the second image 33 with the three-dimensional representation 250 to identify the range of the second image 33 within the three-dimensional representation 250. The matching of the second image 33 may be performed on the information processing device 1 or on another computer other than the information processing device 1. In the latter case, identifying the range of the second image 33 within the three-dimensional representation 250 may be performed by having another computer identify the range of the second image 33 within the three-dimensional representation 250. The monitoring result 50 may be obtainable on the three-dimensional representation 250. The monitoring result 55 in the second imaging range 31 may be appropriately acquired as the monitoring result 50 of the range of the second image 33 identified within the three-dimensional representation 250. According to one example of this embodiment, by providing the three-dimensional representation 250 of the environment E1, it is possible to expect an improvement in the viewability of the environment E1. In one example, when adopting a form in which the monitoring result 55 is reflected in the second image 33, the relative three-dimensional position of each pixel of the second image 33 may be identified on the three-dimensional representation 250. The monitoring result 55 may be extracted according to the identified three-dimensional position of each pixel, and the extracted monitoring result 55 may be reflected in each pixel of the second image 33.

[0162] As an example of a specific implementation method, the computer may determine the relative positional relationship 4 of the second imaging range 31 and extract the monitoring results 55 in the second imaging range 31 using the following procedure. In the following example, it is assumed that the second range information 35 is configured to indicate the second imaging range 31 by including the second image 33 of the second imaging device 3.

[0163] In the first step, the computer may estimate the imaging parameters of the first imaging device 2, the relative three-dimensional position of each pixel of the first image 23 as seen from the first imaging device 2, and the depth value of each pixel of the first image 23 by using a three-dimensional reconstruction model 60 for each of the first images 23 of the first imaging device 2. The imaging parameters may define the first imaging range 21 in three-dimensional space (environment E1). The imaging parameters may include, for example, the orientation and field of view of the first imaging device 2. The depth value of each pixel may be the distance from the first imaging device 2 to the object captured by each pixel. The space composed of the imaging parameters, the relative three-dimensional position of each pixel of the first image 23, and the depth value of each pixel of the first image 23 may be an example of a three-dimensional representation 250. For the three-dimensional reconstruction model 60, known software such as Visual Geometry Grounded Transformer (Non-Patent Literature 2) or COLMAP (Non-Patent Literature 3) may be used. For example, when constructing the 3D representation 250, one or more first images 23 from one or more first imaging devices 2 may be used. The first images 23 used to construct the 3D representation 250 do not need to show the subject S1 to be monitored. Furthermore, using a model for target information may be done by inputting the target information into the model and executing the model's calculations.

[0164] In the second step, the computer may use a three-dimensional reconstruction model 60 on the second image 33 of the second imaging device 3 to estimate the imaging parameters of the second imaging device 3 and the relative three-dimensional position of each pixel of the second image 33 as seen from the first imaging device 2. For example, the imaging parameters of the second imaging device 3 may be the imaging parameters of the second imaging device 3 relative to the first imaging device 2. Estimating the imaging parameters of the second imaging device 3 and the relative three-dimensional position of each pixel of the second image 33 using the three-dimensional reconstruction model 60 may be an example of identifying the range of the second image 33 within the three-dimensional representation 250 (i.e., identifying the relative positional relationship 4 of the second imaging range 31). The first and second steps may be performed in at least partially parallel.

[0165] In the third step, the computer may use the various information obtained in the first and second steps to extract the monitoring results 55 in the second imaging range 31 in any way. For example, in step 3-1, the computer may use a monocular depth estimation model on the first image 23 used to construct the three-dimensional representation 250 to estimate the depth value of each pixel in the first image 23 used to construct the three-dimensional representation 250. For each pixel in the first image 23, the computer may calculate the average ratio of the depth value estimated by the three-dimensional reconstruction model 60 in the first step and the depth value estimated by the depth estimation model. For the depth estimation model, known software such as Depth Anything V2 (Non-Patent Literature 4) may be used, for example.

[0166] In step 3-2, the computer may estimate the depth value of each pixel in each first image 23 (frame) obtained continuously by one or more first imaging devices 2 by using a depth estimation model. The computer may normalize the depth value of each pixel in each first image 23 by multiplying the estimated depth value of each pixel by the average of the ratios calculated in step 3-1. Based on the normalized depth value of each pixel in each first image 23 and the imaging parameters of the first imaging device 2, the computer may estimate the relative three-dimensional position of each pixel in each first image 23 as seen from the first imaging device 2.

[0167] In step 3-3, the computer uses a detection model for each first image 23 continuously obtained by each of the one or more first imaging devices 2, and in each first image 23 The area in which the subject S1 is captured may be detected. The detection model may include, for example, Wholebody34 (non-patented). Known software such as Reference 5), YOLO (Non-Patent Document 6), and SegFormer (Non-Patent Document 7). The detection model may include a segmentation model. The computer may estimate the relative three-dimensional position of the subject S1 as seen from the first imaging device 2, based on the three-dimensional position of each pixel estimated in step 3-2 within the range of the detected region. As an example, the computer may calculate the average value of the three-dimensional positions of each pixel estimated in step 3-2 within the range of the detected region of the subject S1. The computer may obtain the calculated average value of the three-dimensional positions as an estimate of the relative three-dimensional position of the subject S1 as seen from the first imaging device 2. The computer may estimate the relative three-dimensional position of the entire subject S1, or it may estimate the relative three-dimensional position of the subject S1 in parts, such as the face region, hand region, etc.

[0168] For example, when estimating a location far from the subject S1, such as browsing, as the target location for action A1, in steps 3-4, the computer may estimate the target location for action A1 by using an action A1 estimation model for the detected subject S1. The estimated value of the target location for action A1 may include, for example, the direction vector of action A1 from subject S1. Known software may be used for the action A1 estimation model. For example, if action A1 includes browsing, the target location (direction vector) for browsing may be estimated using an estimation model such as Non-Patent Document 8 or Non-Patent Document 9. Note that if the relative 3D position of subject S1 is used as the location for action A1 for all monitored action A1s, steps 3-4 may be omitted.

[0169] In steps 3-5, the computer may detect the action A1 performed in the second imaging range 31 by utilizing the relative three-dimensional position of each pixel of the second image 33 estimated in step 2, and the relative three-dimensional position of the subject S1 estimated in step 3-3. If the location of the action A1 is estimated in step 3-4, the computer may further utilize the estimated location to detect the action A1 performed in the second imaging range 31.

[0170] For example, if the subject S1 includes a human S11 and action A1 (first action A11) includes grasping the object, the computer may calculate the distance between the three-dimensional position of each pixel in the second image 33 estimated in the second step and the three-dimensional position of the hand region of human S11 estimated in the third-third step. The computer may determine whether there is a region in the second image 33 where the calculated distance satisfies a threshold condition. The threshold condition may be, for example, that the calculated distance is less than or equal to a threshold. The threshold may be arbitrarily defined. If there is a region where the calculated distance satisfies the threshold condition, the computer may detect that grasping of the object has been performed in the second imaging range 31. The computer may identify the region where the calculated distance satisfies the threshold condition as the region of the object grasped by human S11. On the other hand, if there is no region where the calculated distance satisfies the threshold condition, the computer may evaluate that grasping of the object has not been performed in the second imaging range 31.

[0171] Furthermore, for example, if action A1 (first action A11) includes browsing, and the direction vector of the location to be browsed is estimated by steps 3-4, the computer may calculate the direction vector for each pixel from the 3D position of human S11 estimated by step 3-3 to the 3D position of each pixel of the second image 33 estimated in step 2. The computer may calculate the difference between the calculated direction vector for each pixel and the browsing direction vector estimated in steps 3-4. The computer may determine whether there is a region in the second image 33 where the calculated difference satisfies a threshold condition. The threshold condition may be, for example, that the calculated difference is less than or equal to a threshold. The threshold may be arbitrarily defined. If there is a region where the calculated difference satisfies the threshold condition, the computer may detect that browsing of the target has been performed in the second imaging range 31. This area may be identified as the area of ​​the object viewed by human S11. On the other hand, if there is no area where the calculated difference satisfies the threshold condition, the computer may conclude that no object viewing has been performed in the second imaging range 31.

[0172] The computer may identify the content of each behavior A1 by subject S1 based on whether or not each behavior A1 is detected. The computer may calculate the degree of behavior A1 of subject S1 by integrating the detection results of each subject S1 and each behavior A1. Based on these, the computer may derive analysis information 571 showing the results of analyzing behavior A1 by subject S1. The computer may also appropriately derive related information 572 from the results of analyzing behavior A1.

[0173] As a result, the computer can acquire the monitoring results 55 of the second imaging range 31. In this example of implementation, the imaging by the second imaging device 3 introduces a new viewpoint, the second imaging device 3 (second image 33). This allows the computer to acquire the monitoring results 55 of behavior A1 in the range that is not captured in the first image 23 of the first imaging device 2 by imaging that range with the second imaging device 3. In other words, by introducing the second imaging device 3 to specify the range from which to extract the monitoring results 55, the range from which to acquire the monitoring results 55 can be expanded. The analysis of behavior A1 in the third to fifth step is an example of analyzing behavior A1 using the second image 33 of the second imaging device 3 as auxiliary material to fill the space of store E11. Furthermore, by reflecting the monitoring results 55 in the second image 33 of the second imaging device 3, it is possible to improve the visibility of the provided monitoring results 55. Note that the computer that performs each of the above calculation processes may include at least one of the information processing device 1 and other computers other than the information processing device 1. At least a portion of each of the above calculation processes may be executed on the information processing device 1 or on another computer.

[0174] The method for extracting the monitoring results 55 is not limited to the above example and may be modified as appropriate depending on the embodiment. In another example, each range information (25, 35) may include the orientation and field of view of each imaging device (2, 3). The relative positional relationship 4 of the second imaging range 31 of the second imaging device 3 may be determined from the orientation and field of view of each imaging device (2, 3) by known techniques such as computer vision. The relative positional relationship 4 may include, for example, the relative position of each pixel of the second image 33 as seen from the first imaging device 2. Based on the determined relative positional relationship 4, the monitoring results 55 in the second imaging range 31 may be extracted as appropriate.

[0175] §2 Example Configuration [Hardware configuration] Figure 8 schematically shows an example of the hardware configuration of the information processing device 1 according to this embodiment. In one example, the information processing device 1 may be configured as a computer in which a control unit 11, a storage unit 12, an external interface 13, an input device 14, and an output device 15 are electrically connected.

[0176] The control unit 11 is configured to perform information processing based on the program and various data. In one example, the control unit 11 includes a hardware processor such as a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory). That's fine. The control unit 11 (CPU) is an example of a processor resource.

[0177] The storage unit 12 is configured to hold arbitrary data. In one example, the storage unit 12 may include a hard disk drive, a solid-state drive, semiconductor memory, etc. The storage unit 12, RAM, and ROM are examples of memory resources. In one example of this embodiment, the storage unit 12 may store various information such as program 81. Program 81 is a program that causes the information processing device 1 to execute information processing (Figure 10, described later) related to providing the monitoring results of the subject S1. Program 81 includes a series of instructions for said information processing.

[0178] In one example, program 81 may be stored in a storage medium 91 instead of, or together with, the storage unit 12. The storage medium 91 is configured to store various types of information (stored programs, etc.) by electrical, magnetic, optical, mechanical, or chemical means so that a machine such as a computer can read the information. The storage unit 12 and the storage medium 91 are examples of non-temporary storage media. The information processing device 1 may retrieve program 81 from the storage medium 91. The storage medium 91 may be a disk-type storage medium (CD, DVD, etc.) or a non-disk-type storage medium such as semiconductor memory (flash memory, etc.). Any drive device may be used to read the information stored in the storage medium 91. The type of drive device may be selected according to the storage medium 91. The drive device may be connected to the information processing device 1 by any method. The storage medium 91 may include an external storage device.

[0179] In one example, the monitoring result 50 of the subject S1 may be stored in at least one of the storage unit 12 and the storage medium 91. This allows the information processing device 1 to retain the monitoring result 50 of the subject S1. The monitoring result 50 may consist of at least one of the provisional analysis results and each of the first images 23.

[0180] The external interface 13 is configured to connect to an external device by wire or wireless connection. In one example, the external interface 13 may include a USB (Universal Serial Bus) port, a dedicated port, a communication port (communication module), etc. The type and number of external interfaces 13 may be determined as appropriate depending on the embodiment. If the external interface 13 includes a communication port, the communication network standard may be arbitrarily selected. The communication standard may be appropriately selected from, for example, the Internet, wireless communication network, mobile communication network, telephone network, dedicated network, etc. In one example, the information processing device 1 may be connected to other devices (first imaging device 2, second imaging device 3, other computers, external storage devices, etc.) via the external interface 13.

[0181] The input device 14 is configured to accept information input. In one example, the input device 14 may include an imaging device, microphone, mouse, keyboard, touch panel, touchpad, or control. The output device 15 is configured to output information. In one example, the output device 15 may include a display or speaker. The information processing device 1 may be operated using the input device 14 and the output device 15. The input device 14 and the output device 15 may be directly connected to the information processing device 1, or they may be indirectly connected via an external interface 13. The input device 14 and the output device 15 may be integrated in at least part by a touch panel display or the like.

[0182] Regarding the specific hardware configuration of the information processing device 1, components can be omitted, replaced, and added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. Hardware processors include microprocessors, FPGAs (field-programmable gate arrays), DSPs (digital signal processors), and GPs. It may consist of a U (Graphics Processing Unit), an ASIC (application-specific integrated circuit), etc. External interface 13, input device 14 and output device 15 At least one of these may be omitted. Program 81 may be stored in an external storage device such as a NAS. Monitoring results 50 may be stored in at least one of another computer and / or an external storage device. An external storage device is also an example of a non-temporary storage medium. The information processing device 1 may consist of multiple computers. In this case, the hardware configuration of each computer may or may not be the same. The information processing device 1 may be a computer designed specifically for the services provided, as well as a general-purpose server device, a general-purpose PC (Personal Computer), a notebook PC, a terminal device, etc. A terminal device is a... This may include smartphones, tablet devices, etc. For example, the information processing device 1 may be a terminal device equipped with a second imaging device 3.

[0183] [Software Configuration] Figure 9 schematically shows an example of the software configuration of the information processing device 1 according to this embodiment. The control unit 11 executes instructions contained in the program 81 stored in the storage unit 12 using the CPU. As a result, the information processing device 1 operates as a computer equipped with a first acquisition unit 111, a second acquisition unit 112, a specific unit 113, an extraction unit 114, and an output processing unit 115 as software modules. In other words, in one example, each software module of the information processing device 1 may be implemented by the control unit 11 (CPU).

[0184] The first acquisition unit 111 is configured to acquire first range information 25 indicating the first imaging range 21 of each of the one or more first imaging devices 2 deployed in the environment E1 while monitoring the behavior A1 of the subject S1. The second acquisition unit 112 is configured to acquire second range information 35 indicating the second imaging range 31 of the second imaging device 3 located within the environment E1. The identification unit 113 is configured to identify the relative positional relationship 4 of the second imaging range 31 of the second imaging device 3 with respect to the first imaging range 21 of each of the one or more first imaging devices 2 from the acquired first range information 25 and second range information 35. The extraction unit 114 is configured to extract the monitoring result 55 in the second imaging range 31 of the second imaging device 3 from the monitoring results 50 regarding the behavior A1 of the subject S1 obtained from the first image 23 captured by each of the one or more first imaging devices 2, based on the identified positional relationship 4. The output processing unit 115 is configured to output the extracted monitoring result 55.

[0185] In this example, each software module of the information processing device 1 is implemented by a general-purpose CPU. However, the method of implementing each of the above modules is not limited to this example and may be modified as appropriate depending on the embodiment. Some or all of the above software modules may be implemented by one or more dedicated processors or chipsets. Each of the above modules may be implemented as a hardware module. Regarding the software configuration of the information processing device 1, modules may be omitted, replaced, and added as appropriate depending on the embodiment.

[0186] §3 Example of Operation Figure 10 is a flowchart showing an example of a processing procedure for providing the monitoring results (monitoring results 55) of the subject S1 by the information processing device 1 according to this embodiment. The information processing device 1 (control unit 11) is configured to execute the following steps in accordance with the instructions included in the program 81. The following processing procedure is an example of an information processing method executed by a computer. The following processing procedure is merely an example. Each step may be changed as much as possible. In addition, steps in the following processing procedure can be omitted, replaced, and added as appropriate depending on the embodiment.

[0187] (Step ST101) In step ST101, the control unit 11 operates as a first acquisition unit 111. That is, the control unit 11 acquires first range information 25 indicating the first imaging range 21 of each of the one or more first imaging devices 2 deployed in the environment E1 while monitoring the behavior A1 of the subject S1. The first imaging devices 2 monitor the behavior A1 of the subject S1 by capturing a first image 23. The first range information 25 may be appropriately configured to specify the range of the first image 23 (first imaging range 21) in the environment E1.

[0188] In one example, the first range information 25 may include a three-dimensional representation 250 of the environment E1 constructed by providing the first images 23 captured by each of the one or more first imaging devices 2 to the three-dimensional reconstruction model 60. In one example, at least one of the one or more first imaging devices 2 may be fixed in a specific position. In one example, the subject S1 may include a human S11, and the action A1 of the subject S1 may include the first action A11 of the human S11. In one example, the human S11 The first action A11 may include at least one of proximity, browsing, and finger interaction actions. In one example, the environment E1 may include the space of store E11. In one example, the subject S1 may include a customer S111 of store E11. In one example, the action A1 of subject S1 may include an action A111 by customer S111 toward an object T111 in store E11. In one example, the action A111 of customer S111 may include entering the monitoring range of the first imaging device 2. In one example, the action A111 of customer S111 may include engaging with the object T111 in store E11 by browsing and approaching. In one example, the action A111 of customer S111 may include grasping an item in store E11. In one example, acquiring the first range information 25 may consist of connecting to an external computer (another computer) that can access the first range information 25. Upon acquiring the first range information 25, the control unit 11 proceeds to the next step ST102.

[0189] (Step ST102) In step ST102, the control unit 11 operates as a second acquisition unit 112. That is, the control unit 11 acquires second range information 35 indicating the second imaging range 31 of the second imaging device 3 located within the environment E1. The second range information 35 may be appropriately configured to identify the range of the second image 33 (second imaging range 31) in the environment E1.

[0190] In one example, the second range information 35 may include the second image 33 captured by the second imaging device 3. The information processing device 1 may acquire the second range information 35 (second image 33) directly or indirectly from the second imaging device 3. Indirect acquisition may be done via an external computer (another computer), a storage medium, an external storage device, etc. Once the second range information 35 is acquired, the control unit 11 proceeds to the next step ST103.

[0191] (Step ST103) In step ST103, the control unit 11 operates as a specific unit 113. That is, the control unit 11 identifies the relative positional relationship 4 of the second imaging range 31 of the second imaging device 3 with respect to the first imaging range 21 of each of the one or more first imaging devices 2, based on the acquired first range information 25 and second range information 35.

[0192] In one example, if the first range information 25 includes a three-dimensional representation 250 of the environment E1 constructed from the first image 23, and the second range information 35 includes the second image 33, the control unit 11 may identify the range of the second image 33 within the three-dimensional representation 250 as a relative positional relationship 4 by comparing the second image 33 with the three-dimensional representation 250. In one example, identifying the relative positional relationship 4 of the second imaging range 31 may be done by transmitting the second range information 35 to an external computer, thereby causing the external computer to identify the relative positional relationship 4 of the second imaging range 31. Once the relative positional relationship 4 is identified, the control unit 11 proceeds to the next step ST104.

[0193] (Step ST104) In step ST104, the control unit 11 operates as an extraction unit 114. That is, based on the identified positional relationship 4, the control unit 11 extracts the monitoring results 55 in the second imaging range 31 of the second imaging device 3 from the monitoring results 50 regarding the behavior A1 of the subject S1 obtained from the first images 23 captured by each of the one or more first imaging devices 2.

[0194] In one example, the monitoring result 55 may include analysis information 571 showing the results of analyzing the behavior of subject S1. In one example, the results of analyzing the behavior shown by the analysis information 571 may include at least one of the degree of subject S1's behavior A1 and the content of behavior A1 performed by subject S1. In one example, the degree of subject S1's behavior A1 may include statistics on the duration of behavior A1 performed by subject S1, statistics on the frequency of behavior A1 performed by subject S1, and the subject who performed behavior A1. The analysis information 571 may include at least one of the statistics of the number of S1. For example, the analysis information 571 may show the results of analyzing the behavior A1 of subject S1 using at least one of a heat map 5711, markers 5713, and a funnel diagram 5715. For example, the analysis information 571 may show the results of analyzing the behavior A1 of subject S1 during a specific period within the period monitored by one or more first imaging devices 2. For example, the analysis information 571 may show the results of analyzing the behavior A1 of subject S1 having specific attributes among the subjects S1 monitored by one or more first imaging devices 2. For example, the monitoring results 55 may include related information 572 corresponding to the results of analyzing the behavior A1 of subject S1.

[0195] In one example, if the subject S1 includes a human S11 and the action A1 of the subject S1 includes the first action A11 of the human S11, the monitoring result 55 may be the monitoring result 55 relating to the first action A11 of a human S11 that satisfies predetermined attribute conditions among the human S11 monitored by one or more first imaging devices 2. In one example, the predetermined attribute conditions may include performing a second action A21. In one example, the second action A21 may include at least one of (1) browsing a specific area in the environment E1, (2) approaching a specific area in the environment E1, and (3) an interaction action of the fingers with a specific area in the environment E1. In one example, if the environment E1 includes the space of a store E11 and the subject S1 includes a customer S111 of the store E11, the monitoring result 55 may relate to the action A111 of a customer S111 that satisfies predetermined purchasing conditions among the customer S111 monitored by one or more first imaging devices 2.

[0196] For example, if the subject S1 includes customer S111 of store E11, the related information 572 may include policy information 580 that proposes recommended measures for store E11, depending on whether the results of the analysis of customer S111's behavior A111 (analysis information 571) satisfy predetermined evaluation conditions. For example, the results of the analysis of customer S111's behavior A111 may include the degree of customer S111's behavior A111. The predetermined evaluation conditions may be defined to evaluate the quality of store E11's operations related to customer S111's behavior A111, based on a comparison of the degree of customer S111's behavior A111 in the second imaging range 31 with a benchmark. The policy information 580 may be configured to propose a first policy 581 recommended to improve store E11's operations that are evaluated as poor compared to the benchmark, or a second policy 582 recommended to strengthen store E11's operations that are evaluated as good compared to the benchmark.

[0197] For example, the predetermined evaluation conditions may be defined to evaluate the quality of store E11's operations related to customer S111's behavior A111, based on a comparison between the degree of customer S111's behavior A111 during a specific period within the period monitored by one or more first imaging devices 2 and a benchmark. The benchmark may be defined based on the degree of customer S111's behavior A111 during periods other than the specific period within the period monitored by one or more first imaging devices 2. For example, the benchmark may be defined based on the degree of customer S111's behavior A111 in ranges other than the second imaging range 31.

[0198] For example, if customer S111's action A111 includes entering the monitoring range of the first imaging device 2, the predetermined evaluation conditions may include an entry evaluation condition that evaluates whether the number of customers S111 entering the monitoring range of the first imaging device 2 that overlaps with the second imaging range 31 is large or small. If, based on the entry evaluation condition, it is evaluated that the number of customers S111 entering the monitoring range of the first imaging range 21 that overlaps with the second imaging range 31 is small, the policy information 580 may be configured to propose a policy to increase the number of customers S111 entering the monitoring range of the first imaging device 2 that overlaps with the second imaging range 31.

[0199] For example, if customer S111's action A111 includes engagement behavior involving at least one of viewing and approaching object T111 in store E11, the predetermined evaluation condition is engagement evaluation, which assesses whether the degree of engagement behavior by customer S111 in the second imaging range 31 is high or low. Value conditions may be included. If, based on the involvement evaluation conditions, the degree of involvement behavior is evaluated as low in the second imaging range 31, the policy information 580 may be configured to propose measures to increase the degree of involvement behavior.

[0200] For example, if customer S111's action A111 includes picking up a product in store E11, the predetermined evaluation conditions may include a conversion evaluation condition that evaluates whether the number of customers S111 who picked up the target product located in the second imaging range 31 is large or small. If, based on the conversion evaluation condition, the number of customers S111 who picked up the target product located in the second imaging range 31 is evaluated to be small, the policy information 580 may be configured to propose a sales promotion measure for the target product.

[0201] For example, extracting the monitoring result 55 may be done by having an external computer extract the monitoring result 55. The control unit 11 may obtain the extracted monitoring result 55 from the external computer. Once the monitoring result 55 is extracted, the control unit 11 proceeds to the next step ST105.

[0202] (Step ST105) In step ST105, the control unit 11 operates as an output processing unit 115. That is, the control unit 11 outputs the extracted monitoring result 55.

[0203] The output configuration and output destination may be determined as appropriate depending on the embodiment. In one example, the control unit 11 may output the extracted monitoring results 55 as they are. The control unit 11 may perform predetermined information processing (such as reflecting the results in an image) on the extracted monitoring results 55 and output the result of performing the predetermined information processing. In one example, the output destination may be RAM, storage unit 12, output device 15, storage medium 91, another computer, external storage device, etc.

[0204] For example, the control unit 11 may output the extracted monitoring result 55 so that it is reflected in the second image 33 captured by the second imaging device 3, based on the specified positional relationship 4. For example, when acquiring the second image 33, the control unit 11 may reflect the extracted monitoring result 55 in the second image 33 based on the specified positional relationship 4 and output the second image 33 with the monitoring result 55 reflected. Alternatively, for example, if the second image 33 is to be output by another computer, the control unit 11 may transmit the specified positional relationship 4 and the extracted monitoring result 55 to the other computer, thereby instructing the other computer to reflect the extracted monitoring result 55 in the second image 33. In this case, outputting the monitoring result 55 so that it is reflected in the second image 33 may be configured by transmitting the specified positional relationship 4 and the extracted monitoring result 55, thereby instructing the other computer to reflect the extracted monitoring result 55 in the second image 33 and output the second image 33 with the monitoring result 55 reflected.

[0205] When the monitoring result 55 is output, the control unit 11 terminates the processing procedure related to this example of operation. The control unit 11 may repeatedly execute the processing of steps ST101 to ST105 at any timing. The analysis processing for the first image 23 (processing to generate the monitoring result) may be executed at any timing before the completion of step ST104. In one example, at least a part of the analysis processing for the first image 23 may be executed in parallel with the processing of step ST104. Also in one example, the control unit 11 may execute the processing of steps ST102 to ST105 in real time. This makes it possible to output the monitoring result 55 for the range that the second imaging device 3 is currently facing (current second imaging range 31). By further adopting a form in which the monitoring result 55 is reflected in the second image 33, the monitoring result 55 can be output in real time on the current second image 33. Also in one example, the control unit 11 may repeatedly execute the processing of steps ST102 to ST105 to enable the second imaging device 3 to It can continuously output monitoring results 55 for the targeted area.

[0206] (Features) In this embodiment, monitoring results 50 of the subject S1 are obtained by one or more first imaging devices 2. Through the processing of steps ST101 to ST104, based on the relative positional relationship 4 between the first imaging range 21 of the first imaging device 2 and the second imaging range 31 of the second imaging device 3, monitoring results 55 for the second imaging range 31 of the second imaging device 3 are extracted from the monitoring results 50 of the first imaging device 2. In other words, by simply pointing the second imaging device 3 towards the target range, the range from which monitoring results 55 are to be extracted can be specified. Therefore, according to this embodiment, by using the second imaging device 3, efficient access to monitoring results 55 for the target range from the monitoring results 50 of the first imaging device 2 can be expected.

[0207] §4 Variant While embodiments of this disclosure have been described in detail above, the above description is merely illustrative in all respects. The processes and means described in this disclosure can be freely combined and implemented as long as no technical inconsistencies arise. Furthermore, various improvements or modifications may be made to the above embodiments as appropriate. For example, the following modifications are possible. In the following, the same reference numerals are used for components similar to those in the above embodiments, and explanations of points similar to those in the above embodiments have been omitted as appropriate. Each component of the following modified examples can be appropriately combined with the components of the above embodiments.

[0208] <4.1> Figure 11 schematically illustrates an example of another scenario to which this disclosure applies. In one example, the identification of the relative positional relationship 4 and the extraction of the monitoring results 55 may be performed by an external computer EC1. Specifically, the information processing device 1 may include a control unit 11. The information processing device 1 may be connected to the external computer EC1. In one example, the information processing device 1 may be connected to the external computer EC1 via an external interface 13.

[0209] The external computer EC1 may be configured to access first range information 25 indicating the first imaging range 21 of each of the one or more first imaging devices 2 deployed in the environment E1, and monitoring results 50 regarding the behavior A1 of subject S1 obtained from the first images 23 captured by each of the one or more first imaging devices 2, when monitoring the behavior A1 of subject S1. Access to such information may include, for example, storing such information, generating such information, retrieving such information from any storage area, and being able to connect to other computers that store such information.

[0210] The information processing device 1 (control unit 11) may acquire second range information 35 indicating the second imaging range 31 of the second imaging device 3 located within the environment E1. The information processing device 1 (control unit 11) may transmit the acquired second range information 35 to the external computer EC1, thereby instructing the external computer EC1 to identify the relative positional relationship 4 of the second imaging range 31 of the second imaging device 3 with respect to the first imaging range 21 of each of the one or more first imaging devices 2, based on the first range information 25 and the second range information 35, and to extract the monitoring result 55 for the second imaging range 31 of the second imaging device 3 from the monitoring results 50 regarding the behavior A1 of the subject S1 obtained from the first image 23 captured by each of the one or more first imaging devices 2, based on the identified positional relationship 4. At least one of the calculation processes of identifying the relative positional relationship 4 and extracting the monitoring result 55 may be performed on the external computer EC1, or on another computer other than the external computer EC1. In the latter case, the external computer EC1 may have another computer perform at least one of the following: identify the relative positional relationship 4 and extract the monitoring results 55, and obtain the results from yet another computer. The information processing device 1 (control unit 11) extracts The monitoring result 55 may be received from an external computer EC1. The information processing device 1 (control unit 11) may output the received monitoring result 55.

[0211] In this modified example, the first acquisition unit 111 and the identification unit 113 may be omitted from the software configuration of the information processing device 1. The second acquisition unit 112 may simply be called the "acquisition unit". The extraction unit 114 may be configured to transmit the acquired second range information 35 to the external computer EC1, thereby instructing the external computer EC1 to identify the relative positional relationship 4 of the second imaging range 31 and to extract the monitoring results 55 in the second imaging range 31. The output processing unit 115 may be configured to receive the extracted monitoring results 55 from the external computer EC1 and to output the received monitoring results 55. In this modified example as well, by using the second imaging device 3, efficient access to the monitoring results 55 of the target range from the monitoring results 50 from the first imaging device 2 can be expected.

[0212] The external computer EC1 may consist of one or more computers. If the external computer EC1 consists of multiple computers, connecting to the external computer EC1 may mean connecting to at least some of the computers. To extract the monitoring results 55, the information processing device 1 may give instructions to all the computers that make up the external computer, or it may give instructions to some of the computers. In the latter case, the computer that receives the instructions from the information processing device 1 may extract the monitoring results 55 by giving instructions directly or indirectly to the remaining computers.

[0213] §5 Experimental Examples To verify the feasibility of this disclosure, the following experiments were conducted. However, this disclosure is not limited to the following experimental examples.

[0214] A surveillance camera installed in a retail store was used as the first imaging device, and images from the surveillance camera were collected as the first image. A camera from a commercially available smartphone was used as the second imaging device.

[0215] Figure 12 shows an example of the first image obtained by a surveillance camera. Figure 13 shows the range from which the second image was obtained using a smartphone. The second image was obtained by capturing images of the ranges indicated by 1 (first location) and 2 (second location) in Figure 13 using a smartphone. For the analysis processing of the first image and the method for determining the relative positional relationship of the second imaging range, the above example of a specific implementation method (steps 1, 2, 3-1, 3-2, 3-3, 3-4, and 3-5) was adopted. For the 3D reconstruction model, Visual Geometry Grounded Transformer (Non-Patent Literature 2) was adopted. For the depth estimation model, Depth Anything V2 (Non-Patent Literature) was adopted. Reference 4) was adopted. For the subject detection model, Wholebody34 (Non-Patent Document 5) was adopted. The estimation model described in Non-Patent Document 8 was used to estimate the location (direction vector) of the object being viewed. In steps 3-5, a commercially available computer was used to detect viewing and grasping, calculate the frequency of detected viewing and grasping, and generate a heatmap from the calculated frequencies. The generated heatmap was then reflected in the second image.

[0216] Figure 14 shows the second image obtained by imaging the first location. Figure 15 shows the result of reflecting the viewing analysis results in the second image of the first location. Figure 16 shows the result of reflecting the grasping analysis results in the second image of the first location. Figure 17 shows the second image obtained by imaging the second location. Figure 18 shows the result of reflecting the viewing analysis results in the second image of the second location. Figure 19 shows the result of reflecting the grasping analysis results in the second image of the second location. From these results, it was possible to verify that the above embodiment is feasible. In addition, as shown in Figures 12 to 19, the monitoring results became visible even in areas with poor visibility from the surveillance camera (first imaging device). From these results, it was possible to introduce a second imaging device (smartphone) and the second imaging device We were able to verify that the visibility of the monitoring results can be improved by reflecting the monitoring results in the image obtained by the placement (second image). [Explanation of symbols]

[0217] 1...Information processing device, 11...Control unit, 12...Storage unit, 2...First imaging device, 21...First imaging range, 23...First image, 25…First range information, 3...Second imaging device, 31...Second imaging range, 33...Second image, 35...Second range information, 4…Positional relationship, 50... Monitoring results, 55... (extracted) monitoring results, E1…Environment, S1…Subject, A1…Action

Claims

1. To acquire first range information indicating the first imaging range of one or more first imaging devices deployed in the environment while monitoring the behavior of a subject, To acquire second range information indicating the second imaging range of the second imaging device present in the aforementioned environment, From the acquired first range information and second range information, the relative positional relationship of the second imaging range of the second imaging device to the first imaging range of each of the one or more first imaging devices is determined. Based on the identified positional relationship, the monitoring results concerning the subject's behavior obtained from the first images captured by each of the one or more first imaging devices, specifically the monitoring results within the second imaging range of the second imaging device, and Output the extracted monitoring results. A control unit configured to perform the following: The second imaging device is movable within the environment to change its position. Information processing device.

2. The second imaging device is movable within the environment by being carried by a person moving within the environment or by being installed on an object moving within the environment. The information processing apparatus according to claim 1.

3. The aforementioned subjects include humans, The actions of the subject include the first action of the human, The information processing apparatus according to claim 1.

4. The first human action includes at least one of browsing, approaching, and finger interaction actions. The information processing apparatus according to claim 3.

5. The aforementioned monitoring results are monitoring results concerning the first behavior of a human being who satisfies predetermined attribute conditions among the humans monitored by the one or more first imaging devices. The information processing apparatus according to claim 3.

6. The aforementioned predetermined attribute conditions include having performed the second action, The information processing apparatus according to claim 5.

7. The second action described above is, (1) Viewing a specific area within the environment, (3) Proximity to a specific area within the environment, (3) Interaction movements of the fingers with a specific area within the environment Including at least one of the following: The information processing apparatus according to claim 6.

8. Outputting the extracted monitoring results is configured to output the extracted monitoring results so that they are reflected in the second image captured by the second imaging device, based on the identified positional relationship. The information processing apparatus according to any one of claims 1 to 7.

9. The monitoring results include analytical information showing the results of analyzing the subject's behavior. The information processing apparatus according to any one of claims 1 to 7.

10. The aforementioned analysis information represents the results of analyzing the subject's behavior during a specific period within the period monitored by the one or more first imaging devices. The information processing apparatus according to claim 9.

11. The aforementioned analysis information shows the results of analyzing the behavior of subjects having specific attributes among the subjects monitored by the one or more first imaging devices. The information processing apparatus according to claim 9.

12. The results of the analysis of the behavior indicated by the aforementioned analysis information include the degree of the subject's behavior, The information processing apparatus according to claim 9.

13. The degree of the subject's behavior includes at least one of the following: a statistic on the duration of the behavior performed by the subject, a statistic on the frequency of the behavior performed by the subject, and a statistic on the number of subjects who performed the behavior. The information processing apparatus according to claim 12.

14. The results of the analysis of the behavior indicated by the aforementioned analysis information include the content of the behavior by the subject, The information processing apparatus according to claim 9.

15. The aforementioned analysis information shows the results of analyzing the subject's behavior using at least one of a heatmap, markers, and a funnel diagram. The information processing apparatus according to claim 9.

16. The aforementioned monitoring results include relevant information corresponding to the results of the analysis of the subject's behavior. The information processing apparatus according to any one of claims 1 to 7.

17. The aforementioned environment includes the store space, The information processing apparatus according to any one of claims 1 to 7.

18. The subjects include customers of the store, The monitoring results relating to the behavior of customers who meet predetermined purchasing conditions among the customers monitored by the one or more first imaging devices, The information processing apparatus according to claim 17.

19. The subjects include customers of the store, The actions of the subject include the actions of the customer toward the objects in the store. The information processing apparatus according to claim 17.

20. At least one of the one or more first imaging devices is fixed in a specific position. The information processing apparatus according to any one of claims 1 to 7.

21. The first range information includes a three-dimensional representation of the environment constructed by providing the first images captured by each of the one or more first imaging devices to a three-dimensional reconstruction model. The second range information includes a second image captured by the second imaging device, Identifying the relative positional relationship between the second imaging range of the second imaging device and the first imaging range of each of the one or more first imaging devices is achieved by comparing the second image with the three-dimensional representation to determine the range of the second image within the three-dimensional representation. The information processing apparatus according to any one of claims 1 to 7.

22. A method of information processing performed by a computer, To acquire first range information indicating the first imaging range of one or more first imaging devices deployed in the environment while monitoring the behavior of a subject, To acquire second range information indicating the second imaging range of the second imaging device present in the aforementioned environment, From the acquired first range information and second range information, the relative positional relationship of the second imaging range of the second imaging device to the first imaging range of each of the one or more first imaging devices is determined. Based on the identified positional relationship, the monitoring results concerning the subject's behavior obtained from the first images captured by each of the one or more first imaging devices, specifically the monitoring results within the second imaging range of the second imaging device, and Output the extracted monitoring results. Includes, The second imaging device is movable within the environment to change its position. Information processing methods.

23. A program that causes a computer to execute an information processing method, The aforementioned information processing method is To acquire first range information indicating the first imaging range of one or more first imaging devices deployed in the environment while monitoring the behavior of a subject, To acquire second range information indicating the second imaging range of the second imaging device present in the aforementioned environment, From the acquired first range information and second range information, the relative positional relationship of the second imaging range of the second imaging device to the first imaging range of each of the one or more first imaging devices is determined. Based on the identified positional relationship, the monitoring results concerning the subject's behavior obtained from the first images captured by each of the one or more first imaging devices, specifically the monitoring results within the second imaging range of the second imaging device, and Output the extracted monitoring results. Includes, The second imaging device is movable within the environment to change its position. program.

24. A program that causes a computer to execute an information processing method, The aforementioned computer is connected to an external computer, The aforementioned external computer is When monitoring the behavior of a subject, first range information indicating the first imaging range of each of the one or more first imaging devices deployed in the environment, and Monitoring results regarding the subject's behavior obtained from the first images captured by each of the one or more first imaging devices described above. It is configured to be accessible, The aforementioned information processing method is To acquire second range information indicating the second imaging range of the second imaging device present in the aforementioned environment, By transmitting the acquired second range information to the external computer, the external computer is instructed to identify the relative positional relationship between the second imaging range of the second imaging device and the first imaging range of each of the one or more first imaging devices, based on the first range information and the second range information, and to extract the monitoring results in the second imaging range of the second imaging device from the monitoring results regarding the behavior of the subject obtained from the first images captured by each of the one or more first imaging devices, based on the identified positional relationship. Receiving the extracted monitoring results from the external computer, and Output the received monitoring results. Includes, The second imaging device is movable within the environment to change its position. program.

Images