Information processing apparatus, information processing method, and information processing system

Abstract

To properly determine the presence / absence of an object within an imaging range.SOLUTION: An information processing apparatus includes: an image storing unit which stores a video captured by an imaging unit; a generation unit which generates tracking data obtained by tracking an object included in the video stored by the image storing unit and a carrying body which carries the object; and a determination unit which determines the presence or absence of the object within an imaging range of the imaging unit on the basis of the tracking data generated by the generation unit.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing device, an information processing method, and an information processing system. [Background technology]

[0002] Conventionally, there has been provided a non-contact object detection technology using an image captured by a camera (see, for example, Patent Document 1). For example, such an object detection technology uses an image captured by a camera to detect an object located within the angle of view (capture range) of the camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2022-524787 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is room for improvement in the above conventional technology. For example, the conventional technology does not take into consideration the case where an object is located within the camera's angle of view (shooting range) but is located in a blind spot from the camera. Thus, there is room for improvement in the conventional technology in terms of how to deal with the case where an object located within the shooting range is not properly detected.

[0005] An object of the present invention is to provide an information processing device, an information processing method, and an information processing system that appropriately determine the presence or absence of an object within an imaging range. [Means for solving the problem]

[0006] The information processing device of this embodiment has an image storage unit that stores images captured by a shooting unit, a generation unit that generates tracking data that tracks an object included in the image stored by the image storage unit and a transporting subject that transports the object, and a determination unit that determines the presence or absence of the object within the shooting range of the shooting unit based on the tracking data generated by the generation unit. Effect of the Invention

[0007] According to the present invention, the presence or absence of an object within an imaging range can be appropriately determined. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an information processing system according to an embodiment. [Diagram 2] FIG. 2 is a perspective view illustrating an example of a lighting device according to an embodiment. [Diagram 3] FIG. 3 is a block diagram showing a configuration of a lighting device according to an embodiment. [Figure 4] FIG. 4 is a block diagram showing a configuration of a server device according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of an AI information storage unit according to the embodiment. [Figure 6] FIG. 6 is a diagram showing an example of image capture by a lighting device having an image capture function. [Figure 7] FIG. 7 is a diagram showing an example of an image including an object captured by the lighting device. [Figure 8] FIG. 8 is a flowchart showing an example of a process for estimating the volume of an object. [Figure 9] FIG. 9 is a diagram showing an example of image capture by a lighting device having an image capture function. [Figure 10] FIG. 10 is a diagram showing an example of an image including a person captured by the lighting device. [Figure 11] FIG. 11 is a flowchart showing an example of a process for determining the presence or absence of an object. [Figure 12]FIG. 12 is a diagram illustrating an example of costs related to detection. [Figure 13] FIG. 13 is a flowchart showing an example of control of object detection processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] An information processing device (e.g., server device 300; the same applies below) according to an embodiment described below has an image storage unit (e.g., memory unit 320; the same applies below) that stores video captured by an imaging unit (e.g., camera 21; the same applies below), a generation unit 335 that generates tracking data that tracks an object included in the video stored by the image storage unit and a transporting entity that transports the object, and a determination unit 337 that determines the presence or absence of an object within the imaging range of the imaging unit based on the tracking data generated by the generation unit 335.

[0010] In addition, in the information processing device according to the embodiment described below, the determination unit 337 determines whether or not an object, which is a moved target object whose position is changed by being carried by a carrying subject, is located within the shooting range of the shooting unit.

[0011] In addition, in the information processing device according to the embodiment described below, the determination unit 337 determines whether or not the object is located in a position that is a blind spot with respect to the field of view of the imaging unit, based on a comparison between the change in the object's position indicated by the tracking data and the change in the position of the transporting subject.

[0012] In addition, in the information processing device according to the embodiment described below, the determination unit 337 determines whether or not an object transported by a transporting entity, which is either a person, a moving body that moves in response to a person's operation, or a moving body that moves automatically, is located within the shooting range of the shooting unit.

[0013] Furthermore, the information processing system 1 according to the embodiment described below includes a lighting device 100 having an imaging unit that captures an image, an image storage unit that stores the image captured by the imaging unit, a generation unit 335 that generates tracking data that tracks an object included in the image stored by the image storage unit and a transporting subject that transports the object, and a determination unit 337 that determines the presence or absence of the object within the imaging range of the imaging unit based on the tracking data generated by the generation unit 335.

[0014] [Embodiment] [Information processing system configuration] From here, the device configuration of the information processing system 1 according to the embodiment and the function of each device will be described. First, the device configuration of the information processing system 1 according to the embodiment will be described based on Fig. 1. Fig. 1 is a diagram showing the information processing system according to the embodiment.

[0015] The information processing system 1 shown in Fig. 1 includes a lighting device 100, a repeater 200, a server device 300, and a terminal device 400. The information processing system 1 may include a plurality of lighting devices 100, a plurality of repeaters 200, a plurality of server devices 300, and a plurality of terminal devices 400. The lighting device 100 is connected to the repeater 200 via a LAN (Local Area Network) or the like so as to be able to communicate with each other by wire or wirelessly. The lighting device 100 communicates with the server device 300 via the repeater 200.

[0016] The lighting device 100 is a lighting device with a camera. For example, the lighting device 100 is a device in which a camera configuration is integrated into a lighting fixture. The lighting device 100 is an edge device equipped with a light source 31 and a camera 21. For example, the lighting device 100 is an information processing device (computer) that executes processing related to capturing an image. Note that the image in this application is a concept that includes a moving image (video) and a still image.

[0017] For example, the lighting device 100 is a lighting device that is installed on the ceiling of a structure such as a factory or a warehouse, and illuminates the internal space of the structure when turned on. FIG. 1 shows a case where the lighting device 100 is installed in the internal space of a factory or a warehouse where an object SB such as luggage is managed. Here, the lighting device 100 will be described with reference to FIG. 2. FIG. 2 is a perspective view showing an example of a lighting device according to an embodiment. The lighting device 100 according to the embodiment is a ceiling-mounted lighting device, which is a so-called base light. In the example shown in FIG. 2, the lighting device 100 has a camera unit 20 and a lighting unit 30. Note that the lighting device 100 is not limited to a base light, and may be a ceiling light, a downlight, a spotlight, or the like.

[0018] Camera unit 20 is an imaging device that captures an image of a predetermined area around lighting device 100. In Fig. 1, camera unit 20 captures an image of an imaging range AR that corresponds to the angle of view of camera 21. For example, imaging range AR indicates the imaging range of camera 21. For example, camera unit 20 captures images such as moving images (video) or still images, targeting the imaging range AR.

[0019] For example, when an object SB is located within the shooting range AR, the camera unit 20 captures an image including the object SB. Also, when an entity (also referred to as a "carrying entity") carrying an object SB such as luggage managed in the space is located within the shooting range AR, the camera unit 20 captures an image including the carrying entity (also referred to as a "carrier" or "moving object"). For example, an object SB etc. is a transported target (moved object) in comparison with a moving object (carrying entity). For example, when a person HM, who is an example of a carrying entity (also referred to as a "person"), is located within the shooting range AR, the camera unit 20 captures an image including the person HM.

[0020] The transporting subject is not limited to a person, but may include various equipment (moving bodies) as long as it is capable of transporting the object SB. For example, when a forklift FL, which is an example of a transporting subject, is located within the shooting range AR, the camera unit 20 captures an image including the forklift FL. Note that, although a forklift is illustrated as an example of a transporting subject other than a person in FIG. 1, the transporting subject other than a person may be various equipment (moving bodies) such as an automatic guided vehicle, an autonomous mobile robot, a cart, etc. In other words, the transporting subject is not limited to a person, but may be various equipment as long as it is capable of transporting the object SB, i.e., the position of the object SB can be changed.

[0021] The repeater 200 is a device having a function of performing relay processing. For example, the repeater 200 is a so-called gateway that connects a network within a facility in which the lighting device 100 is installed to an external network such as the Internet.

[0022] The server device 300 is an information processing device that executes various information processes described later. The server device 300 is a device used as a server of a client-server system. For example, the server device 300 is an information processing device (computer) that functions as a so-called cloud server. The server device 300 executes an object detection process that detects an object from an image captured by the camera 21. The server device 300 executes a process that detects a transport subject such as a person from an image captured by the camera 21. The server device 300 controls various detection processes and executes processes based on the results of various detections, which will be described in detail later.

[0023] Server device 300 is connected to relay 200 and terminal device 400 via a predetermined communication network such as the Internet, either wired or wirelessly. Server device 300 may be connected to relay 200 and terminal device 400 in any manner as long as information can be transmitted and received, and may be connected to relay 200 and terminal device 400 via wired or wireless communication. For example, server device 300 may communicate with terminal device 400 via any device (such as a gateway device).

[0024] The terminal device 400 is a device having a function of displaying information. For example, the terminal device 400 is an information processing device (computer) used by a user. For example, the terminal device 400 is a mobile terminal device that can communicate with any server device 300 via a predetermined network. FIG. 1 shows a case where the terminal device 400 is a notebook PC (Personal Computer). Note that the terminal device 400 may be not only a notebook PC, but also a smart device such as a smartphone or a tablet, or a computer such as a desktop PC.

[0025] The terminal device 400 displays information provided by the server device 300. For example, the terminal device 400 transmits information requesting information (request information) to the server device 300, and displays the information received from the server device 300. For example, the terminal device 400 transmits request information specifying information to the server device 300, and displays the information received from the server device 300. For example, the terminal device 400 displays content including report information, etc.

[0026] [Lighting equipment configuration example] Next, the configuration of the lighting device 100 according to the embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the lighting device according to the embodiment. For example, the lighting device 100 is an information processing device (computer) that functions as a lighting device with a camera. As shown in Fig. 3, the lighting device 100 has a communication unit 10, a camera unit 20, a lighting unit 30, a control unit 40, and a storage unit 50.

[0027] The communication unit 10 transmits and receives information to and from other devices such as the repeater 200. The communication unit 10 is realized, for example, by a predetermined communication circuit or a NIC (Network Interface Card). The communication method by the communication unit 10 may be, for example, wireless communication such as a local area network (LAN). Note that the communication method by the communication unit 10 may be any communication method, for example, wired communication, infrared communication, etc.

[0028] Camera unit 20 has a camera 21, a camera control unit 22, and a storage unit 23. Camera 21 has an imaging element that electronically acquires an image, such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and captures an image corresponding to a predetermined angle of view. For example, camera 21 has a fixed imaging range (imaging range AR in FIG. 1), and captures an image of the fixed imaging range.

[0029] The camera control unit 22 controls the photographing by the camera 21. In addition, the camera control unit 22 outputs the image photographed by the camera 21 to the control unit 40.

[0030] Storage unit 23 stores programs and data for implementing various controls of camera unit 20. Storage unit 23 may store images captured by camera 21 or images (image data) captured by camera 21. Storage unit 23 is implemented by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0031] The lighting unit 30 includes a light source 31, a lighting control unit 32, and a storage unit 33. The light source 31 includes a semiconductor light emitting element such as an LED (Light Emitting Diode). The light source 31 is turned on or off in a manner corresponding to a control signal output from the lighting control unit 32.

[0032] The lighting control unit 32 controls turning on and off of the light sources 31. The lighting control unit 32 turns on the light sources 31 that are to be turned off, or turns off the light sources 31 that are to be turned on. In the case of the lighting unit 30 in which the light source 31 can be dimmed, the lighting control unit 32 may control the dimming degree of the light source 31.

[0033] The storage unit 33 stores programs and data for implementing various controls of the lighting unit 30. The storage unit 33 is implemented by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 33 may be provided in the storage unit 50 described later.

[0034] The storage unit 50 stores various information related to information processing. The storage unit 50 stores programs for implementing various controls of the control unit 40. The storage unit 50 is implemented by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. For example, the storage unit 50 may store information stored in the storage unit 23 or information stored in the storage unit 33.

[0035] The control unit 40 is realized, for example, by a central processing unit (CPU) or a micro processing unit (MPU) executing various programs stored in an internal memory using a RAM as a working area. The control unit 40 can also be realized, for example, by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The control unit 40 has an acquisition unit 41, a processing unit 42, and a transmission unit 43, and realizes or executes the functions and actions of information processing described below. The internal configuration of the control unit 40 is not limited to the configuration shown in FIG. 3, and may be other configurations as long as they perform information processing described below.

[0036] The acquisition unit 41 acquires various information. The acquisition unit 41 acquires various information from the camera unit 20. The acquisition unit 41 acquires images captured by the camera unit 20 from the camera unit 20. The acquisition unit 41 acquires various information from the lighting unit 30. The acquisition unit 41 acquires various information from the memory unit 50. The acquisition unit 41 acquires various information from an external information processing device. The acquisition unit 41 receives various information from an external information processing device via the communication unit 10. The acquisition unit 41 receives information from the repeater 200. The acquisition unit 41 receives information from the server device 300 via the repeater 200.

[0037] The processing unit 42 processes the images captured by the camera unit 20 and executes a predetermined process. The processing unit 42 performs image processing. The processing unit 42 stores various information in the storage unit 50. Here, the processing unit 42 stores the images captured by the camera 21 in a manner according to settings from the terminal device 400. For example, the processing unit 42 may store only one week's worth of images in the storage unit 50 and erase images that have been captured more than one week ago. Furthermore, when a setting is made to upload images to the cloud, the processing unit 42 may upload the captured images to the server device 300 without registering them in the storage unit 50.

[0038] The transmitting unit 43 transmits various information to an external device via the communication unit 10. The transmitting unit 43 transmits various information to the repeater 200. The transmitting unit 43 transmits images captured by the camera unit 20 to an external device. The transmitting unit 43 transmits the images to the server device 300 or the terminal device 400 via the repeater 200.

[0039] [Server device configuration example] Next, the configuration of the server device 300 according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the server device according to the embodiment. As shown in Fig. 4, the server device 300 has a communication unit 310, a storage unit 320, and a control unit 330.

[0040] The communication unit 310 is realized by, for example, a predetermined communication circuit or a NIC. For example, the communication unit 310 can communicate with external devices such as the repeater 200 and the terminal device 400. For example, the communication unit 310 can communicate with the terminal device 400 used by the administrator of the information processing system 1.

[0041] The storage unit 320 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 320 stores various information related to information processing. The storage unit 320 functions as an image storage unit that stores images such as moving images (video) or still images captured by the camera 21. That is, the storage unit 320 stores images captured by the camera 21. The storage unit 320 stores images captured by the camera 21 with a fixed shooting range. For example, the storage unit 320 according to the embodiment includes an AI information storage unit 321, an image information storage unit 322, and a processing information storage unit 323, as shown in FIG. 4.

[0042] The storage unit 320 may store various information according to the purpose, not limited to the above. For example, the storage unit 320 stores various information received from the lighting device 100, the repeater 200, etc. in addition to the above.

[0043] The AI ​​information storage unit 321 stores information about AI (artificial intelligence) used in processing. For example, the AI ​​information storage unit 321 stores information about a model learned by machine learning using AI-related technology. For example, the AI ​​information storage unit 321 stores information (model data) about a trained model (model) learned (generated) by a learning process. FIG. 5 is a diagram illustrating an example of an AI information storage unit according to an embodiment. In the example shown in FIG. 5, the AI ​​information storage unit 321 includes items such as "model ID," "purpose," and "model data."

[0044] "Model ID" indicates identification information for identifying a model. "Use" indicates the use of the corresponding model. "Model data" indicates the data of the model. Figure 5 etc. shows an example in which conceptual information such as "MDT1" is stored in "model data", but in reality, various information constituting the model is included, such as information on the model configuration (network configuration) and information on parameters. For example, "model data" includes information including the nodes in each layer of the network, the functions employed by each node, the connection relationships of the nodes, and the connection coefficients set for the connections between the nodes.

[0045] In FIG. 5, the model (model M1) identified by the model ID "M1" indicates that the application is "object detection". In other words, the model M1 indicates that it is a model that outputs information indicating an object contained in an input image. For example, the model M1 outputs, for an input image, information indicating features such as the color and shape of a detected object and an area (segmentation) that the object occupies in the image. For example, the model M1 outputs, for an input image, a label indicating an object based on features such as the color and shape of a detected object (for example, information indicating the type of object) and information indicating an area (segmentation) that the object occupies in the image. In addition, it indicates that the model data of the model M1 is model data MDT1. Note that the model M1 may be a model that detects objects other than a transport subject such as a person or a forklift, or may be a model that detects a transport subject such as a person or a forklift as an object. In the case where model M1 is a model that detects transporting subjects such as people and forklifts as objects, server device 300 may use information from the detection results using model M1, excluding detection results corresponding to transporting subjects such as people and forklifts, as the result of the object detection process.

[0046] Also, the model (model M2) identified by the model ID "M2" indicates that the application is "person detection". In other words, model M2 indicates that it is a model that outputs information indicating a person (person) included in an input image. For example, model M2 outputs, for an input image, information indicating features such as the color and shape of a detected person and the area (segmentation) that the person occupies in the image. For example, model M2 outputs, for an input image, information indicating a label indicating a person based on features such as the color and shape of a detected person (for example, a transporting entity ID that identifies a transporting entity) and the area (segmentation) that the person occupies in the image. Also, it indicates that the model data of model M2 is model data MDT2.

[0047] Also, the model (model M3) identified by the model ID "M3" indicates that the application is "forklift detection." In other words, model M3 indicates that it is a model that outputs information indicating a forklift contained in an input image. For example, model M3 outputs, for an input image, information indicating features such as the color and shape of the detected forklift, and the area (segmentation) that the forklift occupies in the image. For example, model M3 outputs, for an input image, information indicating a label indicating a forklift based on features such as the color and shape of the detected forklift (for example, a transporting entity ID that identifies the transporting entity), and the area (segmentation) that the forklift occupies in the image. Also, it indicates that the model data of model M3 is model data MDT3.

[0048] 5, a case where a forklift is detected as a transporting subject other than a person is described as an example, but when the transporting subject is a device other than a forklift, such as an automated guided vehicle, a model that detects the moving subject may be used. In this case, the AI ​​information storage unit 321 may store a model (e.g., model M4) that detects an automated guided vehicle.

[0049] The AI ​​information storage unit 321 may store various information according to the purpose, not limited to the above. For example, the AI ​​information storage unit 321 may store a model (e.g., model M5) that detects a transport subject (moving object) including a person, a forklift, an unmanned transport vehicle, etc.

[0050] For example, the AI ​​information storage unit 321 may store data used in learning (learning data) in association with a trained model (model). For example, the learning data includes various information related to the data used in learning the corresponding model, such as data (input information) and answer information (output information) corresponding to the data. In this case, the server device 300 may execute a learning process using the learning data stored in the AI ​​information storage unit 321 to generate a model to be stored in the AI ​​information storage unit 321.

[0051] For example, the AI ​​information storage unit 321 may store information indicating the cost of using each model. In this case, for example, the AI ​​information storage unit 321 may store information indicating that the cost (analysis cost) of using the model M1 that detects an object is high. For example, the AI ​​information storage unit 321 may store information indicating that the cost of using the model M2 that detects a person (figure) is low. For example, the AI ​​information storage unit 321 may store information indicating that the cost of using the model M3 that detects a forklift is low.

[0052] The image information storage unit 322 stores information about various images such as moving images (video) and still images. The image information storage unit 322 stores images such as moving images and still images captured by the lighting device 100. For example, the image information storage unit 322 may store not only the images themselves such as moving images and still images, but also file path names indicating the storage locations of these images. The image information storage unit 322 is not limited to the above, and may store various types of information according to the purpose.

[0053] The processing information storage unit 323 stores information used for various information processes. The processing information storage unit 323 stores information used for generation processing. For example, the processing information storage unit 323 stores information used for generation processing of tracking data. For example, the processing information storage unit 323 stores information used for generation processing of report information to be provided.

[0054] The processing information storage unit 323 stores information used in the estimation process. For example, the processing information storage unit 323 stores information used in the volume estimation process of an object. The processing information storage unit 323 stores information used in the determination process. For example, the processing information storage unit 323 stores information used in the determination process of the presence or absence of an object. Note that the processing information storage unit 323 is not limited to the above, and may store various kinds of information according to the purpose.

[0055] The control unit 330 is realized, for example, by a CPU, an MPU, or the like, executing various programs stored in an internal memory using a RAM as a working area. The control unit 330 can also be realized, for example, by an integrated circuit such as an ASIC or an FPGA. The control unit 330 has an acquisition unit 331, a first detection unit 332, a second detection unit 333, a control processing unit 334, a generation unit 335, an estimation unit 336, a determination unit 337, and a provision unit 338, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of the control unit 330 is not limited to the configuration shown in FIG. 4, and may be other configurations as long as they perform information processing described below.

[0056] The acquisition unit 331 acquires various information. The acquisition unit 331 acquires various information from the storage unit 320. The acquisition unit 331 receives various information from an external device via the communication unit 310. The acquisition unit 331 receives information from the repeater 200. For example, the acquisition unit 331 may receive information collected by the lighting device 100 from the repeater 200. The acquisition unit 331 acquires an image captured by the camera 21.

[0057] The first detection unit 332 functions as an object detection processing unit that executes an object detection process to detect an object from an image acquired by the acquisition unit 331. The first detection unit 332 detects an object from an image stored in the storage unit 320. The first detection unit 332 executes the object detection process using a model M1 that is an object detection model. The first detection unit 332 generates information indicating a detection result of the object detection process.

[0058] The second detection unit 333 functions as a person detection unit that detects a person from the image acquired by the acquisition unit 331. The second detection unit 333 executes a person detection process (also called a "person detection process") using a model M2 that is a person detection model. The second detection unit 333 generates information indicating a detection result of the person detection process.

[0059] The second detection unit 333 may execute a transport subject detection process to detect various transport subjects, not limited to a person (person). For example, the second detection unit 333 executes the forklift detection process using a model M3 that is a forklift detection model.

[0060] The control processing unit 334 executes a control process for controlling various detections. For example, the control processing unit 334 executes the control process based on various information acquired by the acquisition unit 331. The control processing unit 334 executes the control process based on various information stored in the storage unit 320. The control processing unit 334 executes the control process based on various information received from an external device.

[0061] The control processing unit 334 controls detection by the first detection unit 332 and the second detection unit 333. The control processing unit 334 controls the execution state of the object detection process by the first detection unit 332 depending on whether or not the situation regarding the shooting range of the camera 21 satisfies a predetermined condition. If the situation satisfies the predetermined condition, the control processing unit 334 causes the first detection unit 332 to execute the object detection process, and if the situation does not satisfy the predetermined condition, the control processing unit 334 causes the first detection unit 332 to stop executing the object detection process. If a carrying subject carrying an object is located within the shooting range of the camera 21, the control processing unit 334 determines that the predetermined condition is satisfied. If a person is detected by the second detection unit 333, the control processing unit 334 determines that the predetermined condition is satisfied.

[0062] The generating unit 335 executes a generation process to generate various information. The generating unit 335 stores the information generated by the generation process in the storage unit 320. The generating unit 335 executes the generation process based on various information received from an external device. For example, the generating unit 335 executes the generation process based on various information acquired by the acquiring unit 331. For example, the generating unit 335 executes the generation process based on various information stored in the storage unit 320.

[0063] The generating unit 335 executes the generating process based on the detection results by the first detecting unit 332 and the second detecting unit 333. The generating unit 335 executes the generating process using the information estimated by the estimating unit 336. The generating unit 335 executes the generating process using the information determined by the determining unit 337. The generating unit 335 generates tracking data that tracks an object included in an image stored by the storage unit 320 and a transporting subject that transports the object. The generating unit 335 generates tracking data that tracks an object based on a change (transition) in the position of the object detected in each frame (still image) of the video stored by the storage unit 320. The generating unit 335 also generates tracking data that tracks the transporting subject based on a change (transition) in the position of the transporting subject detected in each frame (still image) of the video stored by the storage unit 320.

[0064] The generating unit 335 generates various information such as report information. The generating unit 335 generates various information such as content including report information to be displayed on the terminal device 400. For example, the generating unit 335 generates content (screen) including report information to be provided to the terminal device 400 by appropriately using various techniques related to image generation, image processing, etc. For example, the generating unit 335 generates a screen (content) to be provided to the terminal device 400 by appropriately using various techniques such as Java (registered trademark). The generating unit 335 may generate content including report information to be provided to the terminal device 400 based on the format of CSS (Cascading Style Sheets), JavaScript (registered trademark), or HTML (Hyper Text Markup Language). For example, the generating unit 335 may generate content in various formats such as JPEG (Joint Photographic Experts Group), GIF (Graphics Interchange Format), and PNG (Portable Network Graphics).

[0065] The estimation unit 336 executes an estimation process to estimate various pieces of information. The estimation unit 336 stores the information estimated by the estimation process in the storage unit 320. The estimation unit 336 executes the estimation process based on various pieces of information received from an external device. For example, the estimation unit 336 executes the estimation process based on various pieces of information acquired by the acquisition unit 331. For example, the estimation unit 336 executes the estimation process based on various pieces of information stored in the storage unit 320.

[0066] The estimation unit 336 executes the estimation process based on the detection results by the first detection unit 332 and the second detection unit 333. The estimation unit 336 executes the estimation process using the information generated by the generation unit 335. The estimation unit 336 executes the estimation process using the information determined by the determination unit 337.

[0067] The estimation unit 336 estimates the volume of the one object based on the coordinate information of the one object in each frame (still image) including the one object estimated to be the same object by the analysis of the detection result by the first detection unit 332 and the information indicating the size of the one object in each frame. The estimation unit 336 estimates the volume of the one object based on the coordinate information of the one object in each frame including the one object moving in the image and the information indicating the size of the one object in each frame.

[0068] The estimation unit 336 estimates the volume of a moving object based on coordinate information of the moving object in each frame in which the moving object is included in an image captured within a fixed shooting range, and information indicating the size of the moving object in each frame.

[0069] The determination unit 337 executes a determination process for determining various pieces of information. The determination unit 337 stores the information determined by the determination process in the storage unit 320. The determination unit 337 executes the determination process based on various pieces of information received from an external device. For example, the determination unit 337 executes the determination process based on various pieces of information acquired by the acquisition unit 331. For example, the determination unit 337 executes the determination process based on various pieces of information stored in the storage unit 320.

[0070] The determination unit 337 executes a determination process based on the detection results by the first detection unit 332 and the second detection unit 333. The determination unit 337 executes a determination process using information generated by the generation unit 335. The determination unit 337 executes a determination process using information estimated by the estimation unit 336.

[0071] The determination unit 337 determines the presence or absence of an object within the shooting range of the camera 21 based on the tracking data generated by the generation unit 335. The determination unit 337 determines whether or not an object, which is a moved target object whose position is changed by being transported by a transporting subject, is located within the shooting range of the camera 21.

[0072] The determination unit 337 determines whether or not the object is located in a position that is a blind spot with respect to the field of view of the camera 21, based on a comparison between the change in the object position indicated by the tracking data and the change in the position of the transporting subject. The determination unit 337 determines whether or not the object transported by the transporting subject, which is any one of a person, a moving body that moves in response to a person's operation, and a moving body that moves automatically, is located within the shooting range of the camera 21.

[0073] The providing unit 338 executes a providing process of transmitting various information to an external device. The providing unit 338 transmits various information to an external device via the communication unit 310. The providing unit 338 transmits various information to the terminal device 400 or the repeater 200. The providing unit 338 transmits information generated by the generating unit 335 to the terminal device 400. The providing unit 338 transmits information to be displayed by the terminal device 400 to the terminal device 400. The providing unit 338 transmits report information to the terminal device 400 by the generating unit 335. The providing unit 338 provides report information indicating a detection result based on a detection result of an object by the first detection unit 332 and a detection result of a person by the second detection unit 333.

[0074] [Example of processing in an information processing system] From here, various processing examples will be described in detail based on the above content. In the following, when the information processing system 1 or the server device 300 is described as a processing subject, the processing may be performed by any device included in the information processing system 1 depending on the device configuration included in the information processing system 1. For example, depending on the device configuration of the information processing system 1, not only the server device 300 but also the lighting device 100, the repeater 200, the terminal device 400, etc. may execute the processing described below.

[0075] [Example of estimating the volume of an object] For example, the information processing system 1 executes a process (also referred to as a "first process") of estimating the volume of an object included in an image captured by the camera 21. Below, problems related to the first process, an example of the first process executed by the information processing system 1, and the like will be described.

[0076] Conventional object detection using multi-eye photography with stereo cameras requires multiple cameras to be installed in a range that shares a common angle of view, and in order to estimate the volume of an object, multiple cameras that share a common angle of view must be installed, resulting in issues such as increased camera installation costs.

[0077] Therefore, the information processing system 1 executes a first process of estimating the volume of an object using only one camera 21. The server device 300 estimates the volume of an object based on coordinate information of the object in each frame in which the object is estimated to be the same object by analyzing the detection result of the object detection process that detects the object from the image captured by the single camera 21, and information indicating the size of the object in each frame. For example, the information processing system 1 captures an image of an internal space such as a factory or a warehouse using a lighting device 100 with one camera 21, and detects the object. The information processing system 1 tracks the detected object while detecting the object, and collects information on the position and size of the object by linking the object detected for each image. The information processing system 1 estimates the volume of the object from the collected information.

[0078] An example of this point will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a diagram showing an example of photographing by a lighting device having a photographing function. Fig. 7 is a diagram showing an example of an image including an object photographed by the lighting device.

[0079] 6 and 7 show an example in which object SB1, which is a cardboard box, moves in one direction (for example, from right to left) along the floor surface within the shooting range AR of camera 21. Note that in Figs. 6 and 7, object SB1 is shown as object SB1-1, object SB1-2, and object SB1-3 depending on the position of object SB1, but when describing without making any particular distinction between objects SB1-1, SB1-2, and SB1-3, object SB1 will be described as object SB1.

[0080] 6, the object SB1 is changed (moved) to a position shown by object SB1-1, a position shown by object SB1-2, and a position shown by object SB1-3 in this order. For example, in FIG. 6, the object SB1 is carried (moved) along a floor surface by a carrying subject such as a person.

[0081] Frame FR11 in Fig. 7 corresponds to a frame (still image) captured when object SB1 is at the position shown by object SB1-1 in Fig. 6. Frame FR12 in Fig. 7 corresponds to a frame captured later than frame FR11, and corresponds to a frame (still image) captured when object SB1 is at the position shown by object SB1-2 in Fig. 6. Frame FR13 in Fig. 7 corresponds to a frame captured later than frame FR12, and corresponds to a frame (still image) captured when object SB1 is at the position shown by object SB1-3 in Fig. 6.

[0082] 6 and 7, the server device 300 executes an object detection process to detect objects from an image including frames FR11 to FR13 captured by one camera 21. Then, the server device 300 analyzes the object detection result to estimate that an object SB1-1 included in frame FR11, an object SB1-2 included in frame FR12, and an object SB1-3 included in frame FR13 are the same object SB1.

[0083] Then, the server device 300 estimates the volume of the object SB1 based on the coordinate information of the object SB1 in each of the frames FR11 to FR13 in which the object SB1 is included and information indicating the size of the object SB1 in each of the frames FR11 to FR13.

[0084] 7, the area of ​​the object SB1 in the image becomes smaller as the object position approaches the center of the image (frames FR11 to FR13, etc.). Therefore, the server device 300 estimates the shape of the object from the coordinate position of the object SB1 in the image and the amount of change in length of the object SB1 in each of the two directions, the X direction and the Y direction, that is, the amount of change in area in two dimensions (X direction and Y direction), and estimates the volume based on the estimated shape.

[0085] From here, a first process flow in the information processing system 1 will be described. Fig. 8 is a flowchart showing an example of an object volume estimation process. Note that, although the information processing system 1 will be described below as the processing subject, the process shown in Fig. 8 may be performed by any of the devices such as the lighting device 100, the repeater 200, the server device 300, the terminal device 400, etc., depending on the device configuration included in the information processing system 1.

[0086] The information processing system 1 starts object detection analysis of a video (step S101). For example, the server device 300 starts object detection processing using a video (image) captured by the camera 21.

[0087] The information processing system 1 reads the image into the AI ​​(step S102). For example, the server device 300 inputs the image into the model M1, etc.

[0088] The information processing system 1 performs object detection analysis (step S103). For example, the server device 300 executes object detection processing using a video (image) captured by the camera 21.

[0089] If an object is not detected (step S104: No), the information processing system 1 executes the process of step S109 without executing the processes of steps S105 to S108. For example, if an object is not detected in a target frame (such as a still image), the server device 300 executes the process of step S109.

[0090] When the information processing system 1 detects an object (step S104: Yes), it records the feature quantities such as the color and shape of the object (step S105). For example, when the server device 300 detects an object in a target frame (still image, etc.), it stores the feature quantities such as the color and shape of the object detected in the frame in the storage unit 320.

[0091] If a similar object has not been detected (step S106: No), the information processing system 1 executes the process of step S109 without executing the processes of steps S107 to S108. For example, if a similar object has not been detected in a frame prior to the target frame, the server device 300 executes the process of step S109.

[0092] If a similar object has already been detected (step S106: Yes), the information processing system 1 estimates that it is the same object (step S107). For example, if there is an object whose feature amount is similar to that of the detected object among the objects whose feature amount is stored in the storage unit 320, the server device 300 determines that a similar object has already been detected and estimates that it is the same object.

[0093] Then, the information processing system 1 records the object information (step S108). For example, the server device 300 stores in the storage unit 320 object information in which an object detected in a target frame is associated with the same object detected in a frame prior to the target frame. For example, the server device 300 may make the information (object ID) identifying the object detected in the target frame the same as the information (object ID) identifying the same object detected in a frame prior to the target frame, thereby making it possible to identify the same object in each frame. For example, the object information includes various information used for processing, such as the feature amount of the object detected in the target frame, information indicating the area (coordinate data, length in the vertical direction and length in the horizontal Y direction, etc.) that the object occupies in the frame (still image).

[0094] If there is a next frame (step S109: Yes), the information processing system 1 returns to step S103 and repeats the process. For example, if there is a frame after the target frame, the server device 300 returns to step S103 and repeats the process. Note that, when the server device 300 reads an image into the AI ​​for each frame (still image), the server device 300 may return to step S102 and repeat the process.

[0095] If there is no next frame (step S109: No), the information processing system 1 calls the object information (step S110). For example, if there is no frame after the target frame, the server device 300 acquires object information of an object whose volume has not been calculated from the object information stored in the storage unit 320.

[0096] Then, the information processing system 1 estimates the shape of the object from the coordinate data of the object and the lengths of the object in the X and Y directions (step S111). For example, the server device 300 estimates the shape of the object from the lengths of the X direction, which is a direction along the floor of the space in which the object is located, and the Y direction, which is a direction along the floor perpendicular to the X direction.

[0097] Then, the information processing system 1 estimates the height of the object (step S112). For example, the server device 300 estimates the length (height) of the object in a Z direction (height direction) that is perpendicular to the floor surface of the space in which the object is located (e.g., the direction of gravity).

[0098] Then, the information processing system 1 estimates the volume of the object (step S113). For example, the server device 300 estimates the volume of the object based on the estimated X-direction length, Y-direction length, and Z-direction length (height) of the object. For example, the server device 300 estimates the volume of the object to be a value calculated by multiplying the X-direction length, Y-direction length, and Z-direction length (height) of the object.

[0099] Then, the information processing system 1 records the volume of the object (step S114). For example, the server device 300 records the estimated volume of the object in the storage unit 320 in association with information identifying the object (such as an object ID). Furthermore, the server device 300 may record in the storage unit 320 information identifying the object whose volume has been estimated (such as an object ID) in association with information indicating that the volume of the object has been estimated (such as a flag).

[0100] If there is an object whose volume has not been calculated (step S115: Yes), the information processing system 1 returns to step S110 and repeats the process. For example, if there is an object whose volume has not been calculated, the server device 300 returns to step S110 and repeats the process.

[0101] If there is no object whose volume has not been calculated (step S115: No), the information processing system 1 ends the first process.

[0102] As described above, the information processing system 1 uses the lighting device 100 with the camera 21 to capture an image from the ceiling toward the floor surface, and analyzes the image using technology such as AI to detect an object in the image. For example, the information processing system 1 records the feature amount of the object, and if a similar object is found in the next frame, it estimates that the object is the same as the object in the previous frame. In other words, the information processing system 1 tracks the object. By tracking the object, the information processing system 1 can capture an object moving in the image by being carried by hand, on a dolly, by a forklift, or the like, from multiple angles within the angle of view.

[0103] This allows the information processing system 1 to acquire object information similar to that obtained when the same object is photographed by a plurality of cameras, using only one camera 21. Furthermore, since the information processing system 1 photographs the object as an overhead image from the ceiling, it is possible to estimate the angle at which the object was photographed from the position of the object in the image. Therefore, the information processing system 1 can estimate the height direction of the object from this angle information and the size of the object photographed at each angle. This allows the information processing system 1 to estimate the volume of the object using the image from only one camera 21. Therefore, the information processing system 1 can suppress an increase in the cost of estimating the volume of the object.

[0104] In this way, the information processing system 1 can collect information for optimizing flat placement space in the interior space of a factory, warehouse, or the like by estimating the volume of the object.

[0105] Note that the above is merely an example, and the information processing system 1 may estimate the volume of an object using various information. For example, the information processing system 1 may estimate the volume of an object using parameters related to the camera 21. The information processing system 1 may use information related to the camera 21, such as lens information of the camera 21. For example, the information processing system 1 may use the position of the camera 21, the XY coordinates of the corners of the object in the image of the camera 21, the central coordinates of the object in the image, a correspondence table between the coordinates of the image and the angle (angle of view) of the actual position relative to the camera 21, and the like.

[0106] Furthermore, the information processing system 1 may estimate the material of an object. For example, the information processing system 1 may estimate the material of an object based on the feature amount of the object. For example, the information processing system 1 may estimate the material of an object using information on the estimated color of the object. For example, the information processing system 1 may estimate the material of an object using information on the estimated shape of the object. For example, the information processing system 1 may estimate the volume of an object using information on the estimated material of the object.

[0107] Furthermore, for example, the information processing system 1 calculates the distance between the position of the object SB1 in each of the frames FR11 to FR13 and the position of the camera 21 based on the coordinate information of the object SB1 in each of the frames FR11 to FR13. In this case, the information processing system 1 may estimate the length of the object SB1 in a direction (Z direction) perpendicular to the two-dimensional direction based on the distance between the object SB1 and the camera 21 in each of the frames FR11 to FR13 and the size (area) of the object SB1 in the two-dimensional direction (X direction and Y direction). Then, the information processing system 1 may estimate the volume of the object SB1 using the size of the object SB1 in the two-dimensional direction (X direction and Y direction) and the length of the object SB1 in the direction (Z direction) perpendicular to the two-dimensional direction.

[0108] For example, the information processing system 1 may use information indicating the distance from the position of the camera 21 of the lighting device 100 to the floor surface. In this case, the information processing system 1 may store, for example, in the storage unit 320 of the server device 300, position information indicating the distance from the position of the camera 21 of the lighting device 100 to the floor surface in the vertical direction (gravity direction) (also referred to as "floor surface distance"), and use the position information stored in the storage unit 320. The information processing system 1 estimates the distance between the camera 21 of the lighting device 100 and the object SB1 based on an image including the frames FR11 to FR13 and the like.

[0109] The information processing system 1 estimates the vertical length between the camera 21 of the lighting device 100 and the object SB1 (also called "object distance") using a technique related to image recognition technology. The information processing system 1 estimates the vertical length between the camera 21 of the lighting device 100 and the object SB1 (object distance) using images including frames FR11 to FR13, etc., and an arbitrary distance estimation algorithm.

[0110] For example, the information processing system 1 estimates the vertical length (object distance) between the camera 21 of the lighting device 100 and the object SB1 based on the coordinate information of the object SB1 in each of the frames FR11 to FR13 and the distance between the camera 21 of the lighting device 100 and the object SB1 in each of the frames FR11 to FR13. The information processing system 1 may estimate the value obtained by subtracting the object distance from the floor distance as the height of the object SB1. Furthermore, when a person is carrying the object SB1 in his / her hand, the information processing system 1 may estimate the value obtained by subtracting the object distance and a value indicating the person's carrying position (for example, a value corresponding to the person's chest position) from the floor distance as the height of the object.

[0111] [Example of object presence / absence determination process] For example, the information processing system 1 executes a process (also referred to as a "second process") of determining the presence or absence of an object in the shooting range of the camera 21 by using an image captured by the camera 21. Below, problems related to the second process, an example of the second process executed by the information processing system 1, etc. will be described.

[0112] In conventional object detection using cameras, when detecting an object using images (video) captured by the camera, if the object is in the camera's blind spot, it cannot be detected. Particularly in environments with many objects such as factories and warehouses, even if cameras are installed, the object to be detected may be in the shadow of other objects, making it difficult to grasp the exact amount of objects. One solution to this problem is to install cameras to capture the blind spots, but in this case, it is necessary to analyze the video from multiple cameras, and there are other issues such as increased management costs due to the increased cost of installing the cameras.

[0113] Therefore, the information processing system 1 executes a second process for determining the presence or absence of an object. The server device 300 generates tracking data that tracks an object included in an image captured by the camera 21 and a transporting subject that transports the object, and determines the presence or absence of an object within the shooting range of the camera 21 based on the generated tracking data.

[0114] An example of this point will be described with reference to Fig. 9 and Fig. 10. Fig. 9 is a diagram showing an example of photographing by a lighting device having a photographing function. Fig. 10 is a diagram showing an example of an image including a person photographed by the lighting device.

[0115] 9 and 10, a case is shown in which a person HM2, who is the carrier, moves an object SB2, which is a cardboard box, in one direction (for example, from left to right) along the floor surface within the shooting range AR of the camera 21, and only the person HM2 moves while placing the object SB2 in a blind spot from the camera 21. Note that in Figs. 9 and 10, the person HM2 is shown as a person HM2-1, a person HM2-2, a person HM2-3 according to the position of the person HM2, but when the person HM2-1, the person HM2-2, the person HM2-3, etc. are not particularly distinguished from each other, they are described as a person HM2. Also, in Figs. 9 and 10, the object SB2 is shown as an object SB2-1, an object SB2-2, an object SB2-3 according to the position of the object SB2, but when the object SB2-1, the object SB2-2, the object SB2-3, etc. are not particularly distinguished from each other, they are described as an object SB2.

[0116] In Fig. 9, person HM2 changes (moves) his / her position from the position shown by person HM2-1 to the position shown by person HM2-2 to the position shown by person HM2-3 in that order. As shown in Fig. 9, person HM2 is carrying object SB2 at the position shown by person HM2-1 and the position shown by person HM2-2, and is not holding object SB2 at the position shown by person HM2-3.

[0117] 9, the position of the object SB2 is changed (moved) in the order of the position shown by object SB2-1, the position shown by object SB2-2, and the position shown by object SB2-3. As shown in FIG 9, the positions shown by object SB2-1 and object SB2-2 show a state in which the object SB2 is carrying the object SB2 being carried by person HM2. Also, as shown in FIG 9, the position shown by object SB2-3 shows that the object SB2 is not being carried by person HM2 and is placed in a position that is in a blind spot from the camera 21, as shown by object SB2-3 in dotted line.

[0118] Frame FR21 in Fig. 10 corresponds to a frame (still image) captured when object SB2 is being carried by person HM2-1 and is at the position shown by object SB2-1 in Fig. 9. Frame FR22 in Fig. 10 corresponds to a frame taken at a later time than frame FR21, and corresponds to a frame (still image) captured when object SB2 is being carried by person HM2-2 and is at the position shown by object SB2-2 in Fig. 9. Frame FR23 in Fig. 10 corresponds to a frame taken at a later time than frame FR22, and corresponds to a frame (still image) captured when object SB2 is placed in a position that is in the blind spot of camera 21 and is shown by object SB2-3 in Fig. 9, and only person HM2-3 is moving.

[0119] 9 and 10, the server device 300 executes an object detection process to detect an object from an image including frames FR21-FR23, etc. captured by the camera 21. Then, the server device 300 analyzes the object detection result to generate tracking data that tracks the object SB2 included in the image including frames FR21-FR23, etc., and the person HM2 who is the carrier carrying the object SB2.

[0120] For example, the server device 300 generates, by analyzing the object detection result, tracking data indicating that the position of person HM2 has changed from a position corresponding to person HM2-1 to a position corresponding to person HM2-2 to a position corresponding to person HM2-3, in that order, using an image including frames FR21-FR23, etc. For example, the server device 300 generates, by analyzing the object detection result, tracking data indicating that object SB2 has been carried by person HM2, its position has changed from a position corresponding to object SB2-1 to a position corresponding to object SB2-2, and thereafter it is no longer included in the image, using an image including frames FR21-FR23, etc.

[0121] Then, based on the generated tracking data, the server device 300 determines whether or not the object SB2 is present within the shooting range AR of the camera 21. For example, based on information that the object SB2 moves together with the person HM2 from a position corresponding to the object SB2-1 to a position corresponding to the object SB2-2, and then only the person HM2 changes position to a position corresponding to the person HM2-3, the server device 300 determines whether or not the object SB2 is present within the shooting range AR of the camera 21. In this case, the server device 300 determines that the object SB2 is placed in a position that is a blind spot from the camera 21 in the shooting range AR of the camera 21 during the time when the person HM2 moves from a position corresponding to the person HM2-2 to a position corresponding to the person HM2-3, and determines that the object SB2 is present within the shooting range AR of the camera 21.

[0122] From here, a first processing flow in the information processing system 1 will be described. Fig. 11 is a flowchart showing an example of a process for determining the presence or absence of an object. Note that, although the information processing system 1 will be described below as the processing subject, the process shown in Fig. 11 may be performed by any of the devices such as the lighting device 100, the repeater 200, the server device 300, and the terminal device 400 depending on the device configuration included in the information processing system 1. Also, the description of the same points as those described above in Fig. 8 and the like will be omitted as appropriate.

[0123] The information processing system 1 starts object detection analysis of a video (step S201). For example, the server device 300 starts object detection processing using a video (image) captured by the camera 21.

[0124] The information processing system 1 reads an image into the AI ​​(step S202). For example, the server device 300 inputs the image into models M1 to M3, etc. Then, in FIG. 11, the information processing system 1 executes in parallel the processes of steps S203 to S208 corresponding to person detection, steps S209 to S215 corresponding to object detection, and steps S216 to S220 corresponding to forklift detection.

[0125] The information processing system 1 performs a person detection analysis (step S203). For example, the server device 300 executes a person detection process using a video (image) captured by the camera 21. For example, the server device 300 executes the person detection process using a model M2 that is a person detection model that detects a person (person).

[0126] If a person is not detected (step S204: No), the information processing system 1 executes the process of step S221 without executing the processes of steps S205 to S208. For example, if a person is not detected in a target frame (such as a still image), the server device 300 executes the process of step S221.

[0127] When the information processing system 1 detects a person (step S204: Yes), it records the feature amounts of the person, such as the color and shape (step S205). For example, when the server device 300 detects a person in a target frame (still image, etc.), it stores the feature amounts of the person, such as the color and shape, detected in the frame in the storage unit 320.

[0128] If a similar object has not been detected (step S206: No), the information processing system 1 executes the process of step S221 without executing the processes of steps S207 to S208. For example, if a similar person has not been detected in a frame prior to the target frame, the server device 300 executes the process of step S221.

[0129] If a similar object has already been detected (step S206: Yes), the information processing system 1 estimates that it is the same object (step S207). For example, if there is a person whose feature amount is similar to that of the detected person among the people whose feature amount is stored in the storage unit 320, the server device 300 determines that a similar person has already been detected and estimates that it is the same person.

[0130] Then, the information processing system 1 records the person information and the tracking data (step S208). For example, the server device 300 stores in the storage unit 320 person information in which a person detected in a target frame is associated with the same person detected in a frame prior to the target frame. For example, the server device 300 may make the information identifying the person detected in the target frame (carrier ID such as person ID) the same as the information identifying the same person detected in a frame prior to the target frame (carrier ID such as person ID), thereby making it possible to identify the same person in each frame. For example, the person information includes various information used for processing, such as the feature amount of the person detected in the target frame, information indicating the area (coordinate data, vertical length, horizontal Y length, etc.) that the person occupies in the frame (still image). In addition, for example, the tracking data includes various information regarding the movement of the person to be tracked, such as information indicating the transition of the change in the position of the person to be tracked (movement route).

[0131] The information processing system 1 performs object detection analysis (step S209). For example, the server device 300 executes object detection processing using a video (image) captured by the camera 21. For example, the server device 300 executes the object detection processing using a model M1 that is an object detection model that detects an object.

[0132] If an object is not detected (step S210: No), the information processing system 1 executes the process of step S221 without executing the processes of steps S211 to S214. For example, if an object is not detected in a target frame (such as a still image), the server device 300 executes the process of step S221.

[0133] When the information processing system 1 detects an object (step S210: Yes), it records the feature quantities such as the color and shape of the object (step S211). For example, when the server device 300 detects an object in a target frame (still image, etc.), it stores the feature quantities such as the color and shape of the object detected in the frame in the storage unit 320.

[0134] If a similar object has not been detected (step S212: No), the information processing system 1 executes the process of step S221 without executing the processes of steps S213 to S214. For example, if a similar object has not been detected in a frame prior to the target frame, the server device 300 executes the process of step S221.

[0135] If a similar object has already been detected (step S212: Yes), the information processing system 1 estimates that it is the same object (step S213). For example, if there is an object whose feature amount is similar to that of the detected object among the objects whose feature amount is stored in the storage unit 320, the server device 300 determines that a similar object has already been detected and estimates that it is the same object.

[0136] Then, the information processing system 1 records the object information and the tracking data (step S214). For example, the server device 300 stores in the storage unit 320 object information in which an object detected in a target frame is associated with the same object detected in a frame prior to the target frame. For example, the server device 300 may make the same object identifiable in each frame by making the information (object ID) for identifying the object detected in the target frame the same as the information (object ID) for identifying the same object detected in a frame prior to the target frame. For example, the object information includes various information used for processing, such as the feature amount of the object detected in the target frame, information indicating the area (coordinate data, vertical length, horizontal Y length, etc.) that the object occupies in the frame (still image). In addition, for example, the tracking data includes various information related to the movement of the object to be tracked, such as information indicating the transition of the change in the position of the object to be tracked (movement path).

[0137] The information processing system 1 performs a forklift detection analysis (step S215). For example, the server device 300 executes a forklift detection process using a video (image) captured by the camera 21. For example, the server device 300 executes the forklift detection process using a model M3 that is a forklift detection model for detecting a forklift.

[0138] If a forklift is not detected (step S216: No), the information processing system 1 executes the process of step S221 without executing the processes of steps S217 to S214. For example, if a forklift is not detected in a target frame (still image, etc.), the server device 300 executes the process of step S221.

[0139] When the information processing system 1 detects a forklift (step S216: Yes), it records the feature quantities of the forklift, such as the color and shape (step S217). For example, when the server device 300 detects a forklift in a target frame (still image, etc.), it stores the feature quantities of the forklift, such as the color and shape, detected in the frame in the storage unit 320.

[0140] If a similar object has not been detected (step S218: No), the information processing system 1 executes the process of step S221 without executing the processes of steps S219 to S220. For example, if a similar forklift has not been detected in a frame prior to the target frame, the server device 300 executes the process of step S221.

[0141] If a similar object has already been detected (step S218: Yes), the information processing system 1 estimates that it is the same object (step S219). For example, if there is a forklift with a feature similar to that of the detected forklift among the forklifts whose feature values ​​are stored in the storage unit 320, the server device 300 determines that a similar forklift has already been detected and estimates that it is the same forklift.

[0142] Then, the information processing system 1 records the forklift information and the following data (step S220). For example, the server device 300 stores in the storage unit 320 forklift information in which a forklift detected in a target frame is associated with the same forklift detected in a frame prior to the target frame. For example, the server device 300 may make the information identifying the forklift detected in the target frame (transporter ID such as a forklift ID) the same as the information identifying the same forklift detected in a frame prior to the target frame (transporter ID such as a forklift ID), thereby making it possible to identify the same forklift in each frame. For example, the forklift information includes various information used for processing, such as the feature amount of the forklift detected in the target frame, information indicating the area (coordinate data, length in the vertical direction and length in the horizontal Y direction, etc.) that the forklift occupies in the frame (still image). In addition, for example, the following data includes various information related to the movement of the target to be followed, such as information indicating the transition of the change in the position of the forklift to be followed (movement route).

[0143] For example, the information processing system 1 executes the process of step S221 for a target frame after each detection process is completed. In Fig. 11, the information processing system 1 executes the process of step S221 for a target frame when all of the processes of steps S203 to S208, steps S209 to S215, and steps S216 to S220 have reached step S221.

[0144] If there is a next frame (step S221: Yes), the information processing system 1 returns to step S202 and repeats the process. For example, if there is a frame after the target frame, the server device 300 returns to step S202 and repeats the process. Note that, when loading a video including each frame (still image) into the AI ​​all at once, the server device 300 may return to after step S202 and repeat the process from S203 onward.

[0145] If there is no next frame (step S221: No), the information processing system 1 stores the data (step S222). For example, if there is no frame after the target frame, the server device 300 stores the information acquired in the process up to step S220 in the storage unit 320.

[0146] If there is no object detection result (step S223: No), the information processing system 1 ends the second process. If there is an object detection result (step S223: Yes), the information processing system 1 calls up the object information (step S224). For example, if there is an object detection result in the data stored in the storage unit 320, the server device 300 acquires object information of objects whose volumes are unclassified from the object information stored in the storage unit 320.

[0147] Then, the information processing system 1 determines whether or not there is a detection result of a carrier moving simultaneously with the object (step S225). For example, the server device 300 analyzes the data stored in the storage unit 320 and determines whether or not there is a detection result of a carrier (carrier) moving simultaneously with the object.

[0148] When the information processing system 1 determines that there is a detection result of a carrier moving simultaneously with the object (step S225: Yes), it determines whether they moved simultaneously from the start to the end of the detection (step S226). For example, when the server device 300 determines that there is a detection result of a carrier (carrier) moving simultaneously with the object, it analyzes the data stored in the storage unit 320 and determines whether the object and the carrier (carrier) moved simultaneously from the start to the end of the detection.

[0149] When the information processing system 1 determines that the objects and the transporting subject did not move simultaneously from the start to the end of the detection (step S226: No), it determines whether the detection started at the same time (step S227). For example, when the server device 300 determines that the objects and the transporting subject did not move simultaneously from the start to the end of the detection, it analyzes the data stored in the storage unit 320 and determines whether the object and the transporting subject started detection at the same time. Note that the information processing system 1 does not need to perform step S227. In this case, the information processing system 1 may execute the process of step S228 when step S226 is No.

[0150] When it is determined that the detection did not start at the same time (step S227: No), the information processing system 1 determines whether the detection ended at the same time (step S228). For example, when it is determined that the detection did not start at the same time, the server device 300 analyzes the data stored in the storage unit 320 and determines whether the detection of the object and the transporting subject ended at the same time.

[0151] When the information processing system 1 determines that the object and the transporting subject do not end detection at the same time (step S228: No), it records the object as an object in the screen and classifies it as an object in the screen (step S229). For example, when the server device 300 determines that the object and the transporting subject do not end detection at the same time, it determines that the object is within the shooting range AR of the camera 21. Then, the server device 300 records the object as an object in the screen and classifies it as an object in the screen. The information processing system 1 may record information (such as an object ID) that identifies the object classified in step S229 in association with information (such as a flag) indicating that the object has been classified in the storage unit 320. Then, the information processing system 1 executes the process of step S231 after step S229.

[0152] When the information processing system 1 determines that the detection of the object and the transporting subject is completed at the same time (step S228: Yes), it records the object as an object outside the screen and classifies the object as an object outside the screen (step S230). For example, when the server device 300 determines that the detection of the object and the transporting subject is completed at the same time, it determines that the object is outside the shooting range AR of the camera 21. Then, the server device 300 records the object as an object outside the screen and classifies the object as an object outside the screen. The information processing system 1 may record information (such as an object ID) for identifying the object classified in step S230 in association with information (such as a flag) indicating that the object has been classified in the storage unit 320. Then, the information processing system 1 executes the process of step S231 after step S230.

[0153] Furthermore, when the information processing system 1 determines that there is no detection result of a carrier moving simultaneously with the object (step S225: No), it executes the process of step S229. For example, when the server device 300 determines that there is no detection result of a carrier moving simultaneously with the object, it records the object as an object in the screen and classifies the object as an object in the screen.

[0154] When the information processing system 1 determines that the objects move simultaneously from the start to the end of the detection (step S226: Yes), it executes the process of step S230. For example, when the server device 300 determines that the objects move simultaneously from the start to the end of the detection, it records the object as an off-screen object and classifies the object as an off-screen object.

[0155] When the information processing system 1 determines that the detections have started simultaneously (step S227: Yes), it executes the process of step S229. For example, when the server device 300 determines that the detections have started simultaneously, it records the object as an object in the screen and classifies the object as an object in the screen.

[0156] After the process of step S229 or the process of step S230, the information processing system 1 determines whether or not there is unclassified object information (step S231). If there is unclassified object information (step S231: Yes), the information processing system 1 returns to step S224 and repeats the process. For example, if there is unclassified object information, the server device 300 returns to step S224 and repeats the process.

[0157] If there is no unclassified object information (step S231: No), the information processing system 1 ends the second process.

[0158] As described above, when detecting an object in a video, the information processing system 1 tracks the object between multiple frames, and classifies (counts) an object in a blind spot as an object placed inside a factory or warehouse building. This allows the information processing system 1 to grasp the amount of objects in the entire building with a minimum number of cameras. Therefore, the information processing system 1 can suppress an increase in costs related to object detection. For example, the information processing system 1 can analyze an overhead image taken by a camera 21 installed on the ceiling in a factory, warehouse, etc., and automatically count the amount of objects in the image using technology such as AI analysis, thereby improving the accuracy of the counting method.

[0159] For example, when performing image analysis, the information processing system 1 performs person or forklift detection in addition to object detection, and can detect the delivery of luggage by tracking each of them. For example, when an object and a person move simultaneously, and only the object stops moving and the person moves off screen, the information processing system 1 can determine that the luggage has been delivered. Furthermore, when an object is no longer detected on the screen, the information processing system 1 can determine that the luggage has entered a blind spot, and record information on the amount of the object (size, area, volume, etc.) as luggage placed in the blind spot.

[0160] Also, for example, the information processing system 1 can perform a process to reduce the amount of objects in the screen when a person or a forklift without luggage enters the screen and exits the screen together with an object, as if the person or the forklift had entered the screen and the object had been removed, as opposed to the action of carrying in. Similarly, the information processing system 1 can also treat an object that moves together with a person or a forklift that has started to be detected in the screen as an object being removed, even for luggage placed in a blind spot, and can reduce the amount of objects in the image by assuming that the luggage has been removed at the timing when the luggage leaves the screen. In this way, the information processing system 1 can detect the amount of objects other than those shown in the screen, making it possible to grasp the amount of luggage in a space such as a factory or warehouse with a minimum number of cameras.

[0161] As described above, the information processing system 1 captures an overhead image of a factory or warehouse with the camera 21 installed on the ceiling and records the image. The information processing system 1 analyzes the image using AI or the like to detect objects, people, and forklifts in the image. The information processing system 1 records the feature values ​​and coordinates of the objects, people, and forklifts detected for each frame, and determines whether the objects, people, and forklifts are the same based on the information of the objects, people, and forklifts detected in the previous and next frames using AI or an algorithm, and creates tracking data for the objects based on the information if they are considered to be the same. The information processing system 1 performs the above-mentioned processing (second processing) by comparing the tracking data of the objects, people, and forklifts for each frame. The information processing system 1 can determine the loading / unloading and passing of luggage through the above-mentioned processing, and can grasp the amount of objects in the screen with a minimum number of cameras. In this way, the information processing system 1 can appropriately determine the presence or absence of an object within the imaging range.

[0162] [Example of object detection process control] For example, the information processing system 1 executes a process (also referred to as a "third process") for controlling an object detection process depending on a situation related to the shooting range of the camera 21. Below, problems related to the third process, an example of the third process executed by the information processing system 1, and the like will be described.

[0163] Conventionally, in order to detect an object, it is necessary to constantly analyze camera images using AI (models) or the like. However, as shown in FIG. 12, object detection processing takes longer to analyze images than detecting a specific object such as a person, and measures such as improving the specifications of the analysis device or increasing the number of processing devices are necessary. FIG. 12 is a diagram showing an example of the costs related to detection. For example, FIG. 12 shows the environment of processing devices for person detection AI and object detection AI.

[0164] For example, the first cost CS1 in Fig. 12 indicates the cost of human detection processing that detects people using a human detection AI, that is, a human detection model (e.g., model M2, etc.). As shown in the first cost CS1, in human detection processing, for example, the objects to be detected are limited, so that the specifications of the analysis equipment can be average, and even if processing is performed in a distributed manner, the number of processing devices can be small. In this way, the analysis cost of human detection processing is low.

[0165] For example, the second cost CS2 in FIG. 12 indicates the cost of object detection processing that detects an object using an object detection AI, that is, an object detection model (e.g., model M1, etc.). As shown in the second cost CS2, in object detection processing, for example, since there are many objects assumed to be targets to be detected, the specifications of the analysis equipment must be higher than in person detection processing, and even if processing is performed in a distributed manner, the number of processing devices must be greater than in person detection processing. In other words, the analysis cost of object detection processing is higher than that of person detection processing. As such, since the analysis cost required for object detection processing is high, there is a problem that the analysis cost becomes high if object detection processing is continuously executed.

[0166] Therefore, in the information processing system 1, the execution state of the object detection process for detecting an object from an image captured by the camera 21 is controlled depending on whether or not the situation regarding the shooting range of the camera 21 satisfies a predetermined condition. For example, the information processing system 1 may constantly execute the person detection process, which has a low analysis cost, and execute the object detection process when a person is detected by the person detection process. In this case, for example, in the information processing system 1, if a person is not detected by the person detection process, the object detection process is stopped.

[0167] In this way, the information processing system 1 causes the first detection unit 332 to execute the object detection process when the situation regarding the shooting range AR of the camera 21 satisfies a predetermined condition, and causes the first detection unit 332 to stop executing the object detection process when the situation regarding the shooting range AR of the camera 21 does not satisfy the predetermined condition. For example, the information processing system 1 determines that the predetermined condition is satisfied when a carrying subject carrying an object is located in the shooting range AR of the camera 21. For example, the information processing system 1 determines that the predetermined condition is satisfied when a person is detected by the second detection unit 333 that detects a person from an image captured by the camera 21.

[0168] From here, a third process flow in the information processing system 1 will be described. Fig. 13 is a flowchart showing an example of control of object detection process. Note that, in the following, the information processing system 1 will be described as the processing subject, but the process shown in Fig. 8 may be performed by any of the devices such as the lighting device 100, the repeater 200, the server device 300, the terminal device 400, etc., depending on the device configuration included in the information processing system 1.

[0169] 13 shows a process when a person appears in an image, that is, when a person is detected by human detection processing. For example, the information processing system 1 has stopped object detection processing at the start of the process shown in FIG.

[0170] In the information processing system 1, the person detection AI detects a person (step S301). For example, the server device 300 detects a person by a person detection process using a video (image) captured by the camera 21. For example, the server device 300 detects a person by executing the person detection process using the model M2, which is a person detection model that detects a person (person).

[0171] The information processing system 1 starts the object detection AI (step S302). For example, the server device 300 starts an object detection process using a video (image) captured by the camera 21.

[0172] The information processing system 1 analyzes the camera image using an object detection AI (step S303). For example, the server device 300 executes an object detection process using a model M1, which is an object detection model that detects objects, for the image (image) captured by the camera 21, and analyzes the image (image) captured by the camera 21.

[0173] If the person detection is continuing (step S304: Yes), the information processing system 1 repeats the process of step S303. For example, if the server device 300 detects a person by the person detection process, the information processing system 1 continues the object detection process.

[0174] If the person detection is not continuing (step S304: No), the information processing system 1 stops the object detection AI and ends the process. For example, if the person detection process no longer detects a person, the server device 300 stops the object detection process and ends the process shown in FIG.

[0175] In this way, the information processing system 1 analyzes the camera footage by constantly running the person detection AI. When a person appears in the camera footage, the person detection AI of the information processing system 1 detects the person. When the information processing system 1 detects a person, it starts the object detection AI. While person detection continues, the information processing system 1 analyzes the camera footage with the object detection AI. When the person disappears from the camera footage, the information processing system 1 stops the object detection AI.

[0176] As described above, the information processing system 1 constantly executes the person detection process to detect a person, and executes the object detection process only when a person is detected. The person detection process using a person detection AI or the like has a short analysis time, and even if it operates constantly, the analysis cost is smaller than that of the object detection process using an object detection AI or the like. This allows the information processing system 1 to shorten the execution time of the object detection process and suppress an increase in the operating cost due to the execution of the object detection process. Therefore, the information processing system 1 can suppress an increase in the cost related to object detection.

[0177] Note that the above is merely an example, and the information processing system 1 may control the object detection process using various information. For example, the information processing system 1 may determine whether a predetermined trigger condition is satisfied based on sensor information, and execute the object detection process if the condition is satisfied. In this case, the information processing system 1 may determine whether a predetermined trigger condition is satisfied based on sensor information, and stop the object detection process if the condition is not satisfied. For example, the information processing system 1 may execute the object detection process when the predetermined condition is satisfied, with the predetermined condition being that the color information has changed. For example, the information processing system 1 may execute the object detection process when the predetermined condition is satisfied, with the predetermined condition being that the lighting device 100 has been turned on.

[0178] The stop is not limited to the case where the object detection process is not performed, but includes the restriction of the object detection process. For example, the stop and start of the object detection process described above may be the stop and start of an instance corresponding to the object detection process. Furthermore, the stop and start of the object detection process described above may be the stop and start of a virtual machine corresponding to the object detection process. The stop may be to set the processing range to only the moving image in the range where the trigger condition is satisfied. Furthermore, the entire moving image may be set to the processing range only when the trigger condition is satisfied, and the stop may be to limit the processing range to only a part while the trigger condition is not satisfied.

[0179] Up to this point, the first process to the third process have been described as being performed by the information processing system 1 or the server device 300. However, each of the first process to the third process may be performed by any of the devices, such as the server device 300, the lighting device 100, the repeater 200, the terminal device 400, etc., depending on the device configuration included in the information processing system 1.

[0180] Furthermore, the above-described first to third processes may be executed in any combination as long as each process can be combined. For example, the information processing system 1 may execute the first process and the second process in combination. In this case, the information processing system 1 may execute the first process and the second process in parallel. For example, the information processing system 1 may determine the presence or absence of an object while estimating the volume of the object.

[0181] For example, the information processing system 1 may execute the first process and the third process in combination. In this case, the information processing system 1 may execute the first process and estimate the volume of the object using an image captured after starting the object detection process in the third process. Also, for example, the information processing system 1 may execute the second process and the third process in combination. In this case, the information processing system 1 may execute the second process and determine the presence or absence of the object using an image captured after starting the object detection process in the third process. Also, for example, the information processing system 1 may execute all of the first process to the third process in combination. The information processing system 1 may execute the first process and the second process in parallel using an image captured after starting the object detection process in the third process. For example, the information processing system 1 may determine the presence or absence of the object while estimating the volume of the object using an image captured after starting the object detection process in the third process.

[0182] Also, for example, the information processing system 1 may execute only at least one of the first process to the third process. For example, in the above-mentioned example, when the information processing system 1 executes only the first process, the server device 300 may not have the second detection unit 333, the control processing unit 334, and the determination unit 337. Also, for example, in the above-mentioned example, when the information processing system 1 executes only the second process, the server device 300 may not have the second detection unit 333, the control processing unit 334, and the estimation unit 336. Also, for example, in the above-mentioned example, when the information processing system 1 executes only the third process, the server device 300 may not have the estimation unit 336 and the determination unit 337.

[0183] [Providing report information] The information processing system 1 may provide various types of information using information collected based on various processes such as the first process to the third process described above. An example of this point will be described below. In the following, a case where the server device 300, among the devices included in the information processing system 1, provides information will be described as an example.

[0184] For example, the server device 300 provides report information based on the various processes described above to the terminal device 400. The server device 300 provides the report information to the terminal device 400 used by a manager who manages a factory, a warehouse, or the like in which the lighting device 100 is installed. The server device 300 provides report information indicating a detection result based on a detection result of an object by the first detection unit 332 and a detection result of a person by the second detection unit 333.

[0185] For example, the server device 300 provides report information indicating the results (detection results) of various detection processes performed by the first detection unit 332, the second detection unit 333, etc. For example, the server device 300 provides report information indicating the detection results for each time period. For example, the server device 300 provides report information indicating the amount of change or rate of change in (the volume of) an object estimated to be located in an internal space of a factory, warehouse, etc.

[0186] For example, when an object is located in a blind spot, the server device 300 provides report information including information notifying (alerting) that an object is located in the blind spot. For example, the server device 300 provides a report according to the determination result of the trigger condition and the detection result of the moved object. For example, the server device 300 provides report information including the detection result of a transporting entity (moving object) that moves the object and the detection result of the object (moved object).

[0187] For example, the server device 300 may provide, as the report information, information indicating the utilization status of a space for placing objects such as luggage in an internal space of a factory, a warehouse, etc. For example, the server device 300 may provide, as the report information, information indicating the utilization rate of a predetermined area (also referred to as an "estimated area") such as a flat placement space in a factory, a warehouse, etc.

[0188] In this case, for example, the server device 300 calculates the available volume of the estimated area by using values ​​indicating the area and the upper limit of the height direction of a flat space (estimated area) in a factory, warehouse, etc. For example, if the length of one direction (X direction) along the floor surface of the estimated area (two-dimensional direction) is A and the length of the other direction (Y direction) is B, the server device 300 multiplies A and B to calculate the area (available area) of the estimated area as A×B. Then, for example, if the length (e.g., the top end) of the height direction (Z direction) perpendicular to the two-dimensional direction of the estimated area is C, the server device 300 multiplies A and B to calculate the available volume of the estimated area as A×B×C.

[0189] Also, for example, the server device 300 calculates the total value of the volumes estimated for objects located in an estimated area in a factory, warehouse, etc. as the volume actually used in the estimated area (actually used volume). For example, when objects are stacked in the estimated area, the server device 300 multiplies a and b to calculate the area actually used in the estimated area (actually used area) as a×b, assuming that the length from one end to the other end of the stacked objects on one side (X direction) of the estimated area (two-dimensional direction) is a and the length from one end to the other end on the other side (Y direction) is b. Then, for example, when the length (e.g., the top end) of the height direction (Z direction) perpendicular to the two-dimensional direction of the estimated area is c, the server device 300 multiplies a and b and c to calculate the actually used volume of the estimated area as a×b×c.

[0190] In this case, the server device 300 calculates the estimated area in a factory, warehouse, etc. as the utilization rate based on the available volume and the actual utilization volume. For example, the server device 300 calculates the value obtained by dividing the actual utilization volume by the available volume, i.e., the value calculated by "actual utilization volume / available volume", as the utilization rate of the estimated area in a factory, warehouse, etc. Then, the server device 300 provides report information indicating the utilization rate of the estimated area in the factory, warehouse, etc.

[0191] [Other system configuration examples] The above-described configuration of the information processing system 1 is merely an example, and the information processing system 1 may have any device configuration as long as the desired processing is possible. For example, the information processing system 1 may have a configuration without the repeater 200. Furthermore, the various processes performed by the server device 300 may be executed by any device included in the information processing system 1, not limited to the server device 300. For example, at least a part of the various processes performed by the server device 300 (e.g., the first detection unit 332, the second detection unit 333, etc.) may be executed by the repeater 200 or the lighting device 100. In this way, the various processes performed by the server device 300 may be distributed and processed by multiple devices included in the information processing system 1.

[0192] Although an embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as well as the scope and gist of the invention. In addition, these embodiments and their modifications can be appropriately combined as long as the processing contents are not contradictory. [Explanation of symbols]

[0193] 1. Information Processing Systems 100 Lighting Equipment 10. Communications Department 20 Camera unit 30 Lighting Unit 40 Control section 41 Acquisition Department 42 Processing section 43 Transmitter 50 Storage section 200 Repeater 300 Server device (information processing device) 310 Communications Department 320 Storage section 321 AI information storage unit 322 Image information storage unit 323 Processing information storage unit 330 Control Unit 331 Acquisition Department 332 First detection unit 333 Second detection unit 334 Control Processing Section 335 Generation part 336 Estimation Department 337 Judgment section 338 Provision Department 400 Terminal Equipment

Claims

1. an image storage unit that stores the image captured by the image capture unit; a generating unit that generates tracking data that tracks an object included in the video image stored by the image storage unit and a transporting entity that transports the object; a determination unit that determines the presence or absence of the object within a shooting range of the shooting unit based on the tracking data generated by the generation unit; An information processing device having the above configuration.

2. The determination unit is It is determined whether the object, which is a moving target whose position is changed by the transportation by the transportation subject, is located within the shooting range of the shooting unit. The information processing device according to claim 1 .

3. The determination unit is Based on a comparison between the change in the position of the object indicated by the tracking data and the change in the position of the transporting subject, it is determined whether or not the object is located in a position that is a blind spot with respect to the field of view of the imaging unit. The information processing device according to claim 1 .

4. The determination unit is A determination is made as to whether or not the object transported by the transporting entity, which is any one of a person, a moving body that moves in response to a human operation, and a moving body that moves automatically, is located within a photographing range of the photographing unit. The information processing device according to claim 1 .

5. an image storage step of storing the image captured by the image capturing unit; a generating step of generating tracking data that tracks an object included in the video image stored by the image storing step and a transporting entity that transports the object; a determination step of determining the presence or absence of the object within a shooting range of the shooting unit based on the tracking data generated in the generation step; An information processing method comprising:

6. A lighting device having a shooting unit for shooting an image; an image storage unit that stores the image captured by the image capture unit; a generating unit that generates tracking data that tracks an object included in the video image stored by the image storage unit and a transporting entity that transports the object; a determination unit that determines the presence or absence of the object within a shooting range of the shooting unit based on the tracking data generated by the generation unit; An information processing system having the above configuration.

Images