Information processing apparatus, information processing method, and information processing system

Abstract

To reduce complexity of operations performed by an administrator.SOLUTION: An information processing apparatus has: a storage unit that stores images of a predetermined area photographed by a photographing unit; a movable body detection unit that detects a movable body included in the images stored by the storage unit; an estimation unit that estimates the class of the area on the basis of a result of detection performed by the movable body detection unit; and an object detection unit that performs object detection in the area on the basis of the estimated class of the area.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing device, an information processing method, and an information processing system. [Background technology]

[0002] Conventionally, there has been provided a non-contact object detection technology using an image captured by a camera (see, for example, Patent Document 1). For example, such an object detection technology uses an image captured by a camera to detect an object located within the camera's angle of view (capture range). Also, for example, in a factory or warehouse, there are areas where placing objects in the aisles is prohibited, and there is a technology that detects when an object is placed in this area and notifies the administrator. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2022-524787 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the above-mentioned conventional technology, it is necessary to set the area where items are placed in advance, and in an actual manufacturing site, when the layout of parts storage areas is constantly changing or the types of manufactured products change from time to time, the area settings must be changed each time, which is troublesome for the manager.

[0005] An object of the present invention is to provide an information processing device, an information processing method, and an information processing system that can reduce the burden on an administrator. [Means for solving the problem]

[0006] The information processing device of this embodiment has a memory unit that stores video captured by a shooting unit of a specified area, a moving object detection unit that detects moving objects contained in the video stored in the memory unit, an estimation unit that estimates the type of the area based on the detection result of the detection unit, and an object detection unit that detects objects in the area based on the estimated type of the area. Effect of the Invention

[0007] According to the present invention, it is possible to reduce the burden on the administrator. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an information processing system according to an embodiment. [Diagram 2] FIG. 2 is a perspective view illustrating an example of a lighting device according to an embodiment. [Diagram 3] FIG. 3 is a block diagram showing a configuration of a lighting device according to an embodiment. [Figure 4] FIG. 4 is a block diagram showing the configuration of a server device according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of an AI information storage unit according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of the area information storage unit according to the embodiment. [Figure 7] FIG. 7 is a flowchart showing an example of an area type estimation process. [Figure 8] FIG. 8 is a flowchart showing an example of an object detection process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] An information processing device (e.g., server device 300; the same applies below) according to an embodiment described below has a memory unit 320 that stores video of a specific area captured by an imaging unit (e.g., camera 21; the same applies below), a moving object detection unit (second detection unit 333) that detects a moving object included in the video stored by the memory unit 320, an estimation unit 334 that estimates the type of area based on the detection result of the moving object detection unit, and an object detection unit (first detection unit 332) that detects objects in the area based on the estimated type of area.

[0010] The estimation unit 334 according to the embodiment described below estimates a prohibited area, which is a partial area in which the presence of an object is prohibited, as the type of area. The object detection unit detects the presence or absence of an object in the prohibited area.

[0011] The information processing device according to the embodiment described below further includes a processing unit 335 that tags the video stored in the storage unit 320 when the object detection unit detects the presence of an object in the prohibited area.

[0012] The information processing device according to the embodiment described below further includes a processing unit 335 that outputs an instruction signal to an alarm device arranged in the area to issue an alarm when the object detection unit detects the presence of an object in the prohibited area.

[0013] A moving object detection unit according to an embodiment described below detects the position of a moving object in an area. An estimation unit 334 estimates a partial area corresponding to the position of the moving object as a prohibited area.

[0014] A moving object detection unit according to an embodiment described below detects a movement trajectory of a moving object in an area. An estimation unit 334 estimates a partial area corresponding to the movement trajectory of the moving object as a prohibited area.

[0015] The object detection unit according to the embodiment described below determines the type of area to be used for object detection based on the schedule information of the area.

[0016] The information processing method of the embodiment described below is an information processing method including a storage process of storing video of a specified area captured by a capture unit in a storage unit 320, a moving object detection process of detecting a moving object contained in the video stored in the storage unit 320, an estimation process of estimating the type of area based on the detection result of the moving object detection process, and an object detection process of detecting an object in the area based on the estimated type of area.

[0017] The information processing system 1 according to the embodiment described below includes a lighting device 100 having an imaging unit that captures an image, a memory unit 320 that stores an image of a specified area captured by the imaging unit, a moving object detection unit that detects a moving object included in the image stored in the memory unit 320, an estimation unit 334 that estimates the type of the area based on the detection result of the moving object detection unit, and an object detection unit that detects objects in the area based on the estimated type of area.

[0018] [Embodiment] [Information processing system configuration] From here, the device configuration of the information processing system 1 according to the embodiment and the function of each device will be described. First, the device configuration of the information processing system 1 according to the embodiment will be described based on Fig. 1. Fig. 1 is a diagram showing the information processing system according to the embodiment.

[0019] The information processing system 1 shown in Fig. 1 includes a lighting device 100, a repeater 200, a server device 300, and a terminal device 400. The information processing system 1 may include a plurality of lighting devices 100, a plurality of repeaters 200, a plurality of server devices 300, and a plurality of terminal devices 400. The lighting device 100 is connected to the repeater 200 via a LAN (Local Area Network) or the like so as to be able to communicate with each other by wire or wirelessly. The lighting device 100 communicates with the server device 300 via the repeater 200.

[0020] The lighting device 100 is a lighting device with a camera. For example, the lighting device 100 is a device in which a camera configuration is integrated into a lighting fixture. The lighting device 100 is an edge device equipped with a light source 31 and a camera 21. For example, the lighting device 100 is an information processing device (computer) that executes processing related to capturing an image. Note that the image in this application is a concept that includes a moving image (video) and a still image.

[0021] For example, the lighting device 100 is a lighting device that is installed on the ceiling of a structure such as a factory or a warehouse, and illuminates the internal space of the structure when turned on. FIG. 1 shows a case where the lighting device 100 is installed in the internal space of a factory or a warehouse where an object SB such as luggage is managed. Here, the lighting device 100 will be described with reference to FIG. 2. FIG. 2 is a perspective view showing an example of a lighting device according to an embodiment. The lighting device 100 according to the embodiment is a ceiling-mounted lighting device, which is a so-called base light. In the example shown in FIG. 2, the lighting device 100 has a camera unit 20 and a lighting unit 30. Note that the lighting device 100 is not limited to a base light, and may be a ceiling light, a downlight, a spotlight, or the like.

[0022] Camera unit 20 is an imaging device that captures an image of a predetermined area around lighting device 100. In Fig. 1, camera unit 20 captures an image of an imaging range AR that corresponds to the angle of view of camera 21. For example, imaging range AR indicates the imaging range of camera 21. For example, camera unit 20 captures images such as moving images (video) or still images, targeting the imaging range AR.

[0023] For example, when an object SB is located within the shooting range AR, the camera unit 20 captures an image including the object SB. Also, when an entity (also referred to as a "carrying entity") carrying an object SB such as luggage managed in the space is located within the shooting range AR, the camera unit 20 captures an image including the carrying entity (also referred to as a "carrier" or "moving object"). For example, an object SB etc. is a transported target (moved object) in comparison with a moving object (carrying entity). For example, when a person HM, who is an example of a carrying entity (also referred to as a "person"), is located within the shooting range AR, the camera unit 20 captures an image including the person HM.

[0024] The transporting subject is not limited to a person, but may include various equipment (moving bodies) as long as it is capable of transporting the object SB. For example, when a forklift FL, which is an example of a transporting subject, is located within the shooting range AR, the camera unit 20 captures an image including the forklift FL. Note that, although a forklift is illustrated as an example of a transporting subject other than a person in FIG. 1, the transporting subject other than a person may be various equipment (moving bodies) such as an automatic guided vehicle, an autonomous mobile robot, a cart, etc. In other words, the transporting subject is not limited to a person, but may be various equipment as long as it is capable of transporting the object SB, i.e., the position of the object SB can be changed.

[0025] The repeater 200 is a device having a function of performing relay processing. For example, the repeater 200 is a so-called gateway that connects a network within a facility in which the lighting device 100 is installed to an external network such as the Internet.

[0026] The server device 300 is an information processing device that executes various information processes described later. The server device 300 is a device used as a server of a client-server system. For example, the server device 300 is an information processing device (computer) that functions as a so-called cloud server. The server device 300 executes an object detection process that detects an object from an image captured by the camera 21. The server device 300 executes a process that detects a transport subject such as a person from an image captured by the camera 21. The server device 300 controls various detection processes and executes processes based on the results of various detections, which will be described in detail later.

[0027] Server device 300 is connected to relay 200 and terminal device 400 via a predetermined communication network such as the Internet, either wired or wirelessly. Server device 300 may be connected to relay 200 and terminal device 400 in any manner as long as information can be transmitted and received, and may be connected to relay 200 and terminal device 400 via wired or wireless communication. For example, server device 300 may communicate with terminal device 400 via any device (such as a gateway device).

[0028] The terminal device 400 is a device having a function of displaying information. For example, the terminal device 400 is an information processing device (computer) used by a user. For example, the terminal device 400 is a mobile terminal device that can communicate with any server device 300 via a predetermined network. FIG. 1 shows a case where the terminal device 400 is a notebook PC (Personal Computer). Note that the terminal device 400 may be not only a notebook PC, but also a smart device such as a smartphone or a tablet, or a computer such as a desktop PC.

[0029] The terminal device 400 displays information provided by the server device 300. For example, the terminal device 400 transmits information requesting information (request information) to the server device 300, and displays the information received from the server device 300. For example, the terminal device 400 transmits request information specifying information to the server device 300, and displays the information received from the server device 300. For example, the terminal device 400 displays content including report information, etc.

[0030] In the present disclosure, when the information processing system 1 detects an object present in a prohibited area from an image captured by the camera 21 (see FIG. 3), the information processing system 1 automatically estimates the area type, such as the prohibited area, based on the movement of a moving object detected in advance from past images, and performs object detection based on the estimated area type.

[0031] This allows the area type to be set automatically even when the layout of parts storage areas at the manufacturing site changes dynamically, or when the types of manufactured products change each time, etc. Therefore, the administrator does not need to change the area type settings each time, reducing the administrator's hassle.

[0032] [Lighting equipment configuration example] Next, the configuration of the lighting device 100 according to the embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the lighting device according to the embodiment. For example, the lighting device 100 is an information processing device (computer) that functions as a lighting device with a camera. As shown in Fig. 3, the lighting device 100 has a communication unit 10, a camera unit 20, a lighting unit 30, a control unit 40, and a storage unit 50.

[0033] The communication unit 10 transmits and receives information to and from other devices such as the repeater 200. The communication unit 10 is realized, for example, by a predetermined communication circuit or a NIC (Network Interface Card). The communication method by the communication unit 10 may be, for example, wireless communication such as a local area network (LAN). Note that the communication method by the communication unit 10 may be any communication method, for example, wired communication, infrared communication, etc.

[0034] Camera unit 20 has a camera 21, a camera control unit 22, and a storage unit 23. Camera 21 has an imaging element that electronically acquires an image, such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and captures an image corresponding to a predetermined angle of view. For example, camera 21 has a fixed imaging range (imaging range AR in FIG. 1), and captures an image of the fixed imaging range.

[0035] The camera control unit 22 controls the photographing by the camera 21. In addition, the camera control unit 22 outputs the image photographed by the camera 21 to the control unit 40.

[0036] Storage unit 23 stores programs and data for implementing various controls of camera unit 20. Storage unit 23 may store images captured by camera 21 or images (image data) captured by camera 21. Storage unit 23 is implemented by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0037] The lighting unit 30 includes a light source 31, a lighting control unit 32, and a storage unit 33. The light source 31 includes a semiconductor light emitting element such as an LED (Light Emitting Diode). The light source 31 is turned on or off in a manner corresponding to a control signal output from the lighting control unit 32.

[0038] The lighting control unit 32 controls turning on and off of the light sources 31. The lighting control unit 32 turns on the light sources 31 that are to be turned off, or turns off the light sources 31 that are to be turned on. In the case of the lighting unit 30 in which the light source 31 can be dimmed, the lighting control unit 32 may control the dimming degree of the light source 31.

[0039] The storage unit 33 stores programs and data for implementing various controls of the lighting unit 30. The storage unit 33 is implemented by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 33 may be provided in the storage unit 50 described later.

[0040] The storage unit 50 stores various information related to information processing. The storage unit 50 stores programs for implementing various controls of the control unit 40. The storage unit 50 is implemented by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. For example, the storage unit 50 may store information stored in the storage unit 23 or information stored in the storage unit 33.

[0041] The control unit 40 is realized, for example, by a central processing unit (CPU) or a micro processing unit (MPU) executing various programs stored in an internal memory using a RAM as a working area. The control unit 40 can also be realized, for example, by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The control unit 40 has an acquisition unit 41, a processing unit 42, and a transmission unit 43, and realizes or executes the functions and actions of information processing described below. The internal configuration of the control unit 40 is not limited to the configuration shown in FIG. 3, and may be other configurations as long as they perform information processing described below.

[0042] The acquisition unit 41 acquires various information. The acquisition unit 41 acquires various information from the camera unit 20. The acquisition unit 41 acquires images captured by the camera unit 20 from the camera unit 20. The acquisition unit 41 acquires various information from the lighting unit 30. The acquisition unit 41 acquires various information from the memory unit 50. The acquisition unit 41 acquires various information from an external information processing device. The acquisition unit 41 receives various information from an external information processing device via the communication unit 10. The acquisition unit 41 receives information from the repeater 200. The acquisition unit 41 receives information from the server device 300 via the repeater 200.

[0043] The processing unit 42 processes the images captured by the camera unit 20 and executes a predetermined process. The processing unit 42 performs image processing. The processing unit 42 stores various information in the storage unit 50. Here, the processing unit 42 stores the images captured by the camera 21 in a manner according to settings from the terminal device 400. For example, the processing unit 42 may store only one week's worth of images in the storage unit 50 and erase images that have been captured more than one week ago. Furthermore, when a setting is made to upload images to the cloud, the processing unit 42 may upload the captured images to the server device 300 without registering them in the storage unit 50.

[0044] The transmitting unit 43 transmits various information to an external device via the communication unit 10. The transmitting unit 43 transmits various information to the repeater 200. The transmitting unit 43 transmits images captured by the camera unit 20 to an external device. The transmitting unit 43 transmits the images to the server device 300 or the terminal device 400 via the repeater 200.

[0045] In this way, by including the camera unit 20, the lighting device 100 of the present disclosure can automatically set, for example, brightness and color conditions that are most suitable for the imaging conditions of the camera unit 20. Furthermore, as in the present disclosure, the camera unit 20 and the lighting unit 30 can be arranged in a relatively close positional relationship, so that when imaging with the camera unit 20, the reflection of a shadow of an object caused by the light of the lighting unit 30 can be minimized.

[0046] [Server device configuration example] Next, the configuration of the server device 300 according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the server device according to the embodiment. As shown in Fig. 4, the server device 300 has a communication unit 310, a storage unit 320, and a control unit 330.

[0047] The communication unit 310 is realized by, for example, a predetermined communication circuit or a NIC. For example, the communication unit 310 can communicate with external devices such as the repeater 200 and the terminal device 400. For example, the communication unit 310 can communicate with the terminal device 400 used by the administrator of the information processing system 1.

[0048] The storage unit 320 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 320 stores various information related to information processing. The storage unit 320 functions as an image storage unit that stores images such as moving images (video) or still images captured by the camera 21. That is, the storage unit 320 stores images captured by the camera 21. The storage unit 320 stores images captured by the camera 21 with a fixed shooting range. For example, the storage unit 320 according to the embodiment includes an AI information storage unit 321, an image information storage unit 322, a processing information storage unit 323, and an area information storage unit 324, as shown in FIG. 4.

[0049] The storage unit 320 may store various information according to the purpose, not limited to the above. For example, the storage unit 320 stores various information received from the lighting device 100, the repeater 200, etc. in addition to the above.

[0050] The AI ​​information storage unit 321 stores information about AI (artificial intelligence) used in processing. For example, the AI ​​information storage unit 321 stores information about a model learned by machine learning using AI-related technology. For example, the AI ​​information storage unit 321 stores information (model data) about a trained model (model) learned (generated) by a learning process. FIG. 5 is a diagram illustrating an example of an AI information storage unit according to an embodiment. In the example shown in FIG. 5, the AI ​​information storage unit 321 includes items such as "model ID," "purpose," and "model data."

[0051] "Model ID" indicates identification information for identifying a model. "Use" indicates the use of the corresponding model. "Model data" indicates the data of the model. Figure 5 etc. shows an example in which conceptual information such as "MDT1" is stored in "model data", but in reality, various information constituting the model is included, such as information on the model configuration (network configuration) and information on parameters. For example, "model data" includes information including the nodes in each layer of the network, the functions employed by each node, the connection relationships of the nodes, and the connection coefficients set for the connections between the nodes.

[0052] In FIG. 5, the model (model M1) identified by the model ID "M1" indicates that the application is "object detection". In other words, the model M1 indicates that it is a model that outputs information indicating an object contained in an input image. For example, the model M1 outputs, for an input image, information indicating features such as the color and shape of a detected object and an area (segmentation) that the object occupies in the image. For example, the model M1 outputs, for an input image, a label indicating an object based on features such as the color and shape of a detected object (for example, information indicating the type of object) and information indicating an area (segmentation) that the object occupies in the image. In addition, it indicates that the model data of the model M1 is model data MDT1. Note that the model M1 may be a model that detects objects other than a transport subject such as a person or a forklift, or may be a model that detects a transport subject such as a person or a forklift as an object. In the case where model M1 is a model that detects transporting entities such as people and forklifts as objects, server device 300 may use information from the detection results using model M1, excluding detection results corresponding to transporting entities such as people and forklifts, as the result of the object detection process.

[0053] Also, the model (model M2) identified by the model ID "M2" indicates that the application is "person detection". In other words, model M2 indicates that it is a model that outputs information indicating a person (person) included in an input image. For example, model M2 outputs, for an input image, information indicating features such as the color and shape of a detected person and the area (segmentation) that the person occupies in the image. For example, model M2 outputs, for an input image, information indicating a label indicating a person based on features such as the color and shape of a detected person (for example, a transporting entity ID that identifies a transporting entity) and the area (segmentation) that the person occupies in the image. Also, it indicates that the model data of model M2 is model data MDT2.

[0054] Also, the model (model M3) identified by the model ID "M3" indicates that the application is "forklift detection." In other words, model M3 indicates that it is a model that outputs information indicating a forklift contained in an input image. For example, model M3 outputs, for an input image, information indicating features such as the color and shape of the detected forklift, and the area (segmentation) that the forklift occupies in the image. For example, model M3 outputs, for an input image, information indicating a label indicating a forklift based on features such as the color and shape of the detected forklift (for example, a transporting entity ID that identifies the transporting entity), and the area (segmentation) that the forklift occupies in the image. Also, it indicates that the model data of model M3 is model data MDT3.

[0055] 5, a case where a forklift is detected as a transporting subject other than a person is described as an example, but when the transporting subject is a device other than a forklift, such as an automated guided vehicle, a model that detects the moving subject may be used. In this case, the AI ​​information storage unit 321 may store a model (e.g., model M4) that detects an automated guided vehicle.

[0056] The AI ​​information storage unit 321 may store various information according to the purpose, not limited to the above. For example, the AI ​​information storage unit 321 may store a model (e.g., model M5) that detects a transport subject (moving object) including a person, a forklift, an unmanned transport vehicle, etc.

[0057] For example, the AI ​​information storage unit 321 may store data used in learning (learning data) in association with a trained model (model). For example, the learning data includes various information related to the data used in learning the corresponding model, such as data (input information) and answer information (output information) corresponding to the data. In this case, the server device 300 may execute a learning process using the learning data stored in the AI ​​information storage unit 321 to generate a model to be stored in the AI ​​information storage unit 321.

[0058] For example, the AI ​​information storage unit 321 may store information indicating the cost of using each model. In this case, for example, the AI ​​information storage unit 321 may store information indicating that the cost (analysis cost) of using the model M1 that detects an object is high. For example, the AI ​​information storage unit 321 may store information indicating that the cost of using the model M2 that detects a person (figure) is low. For example, the AI ​​information storage unit 321 may store information indicating that the cost of using the model M3 that detects a forklift is low.

[0059] The image information storage unit 322 stores information about various images such as moving images (video) and still images. The image information storage unit 322 stores images such as moving images and still images captured by the lighting device 100. For example, the image information storage unit 322 may store not only the images themselves such as moving images and still images, but also file path names indicating the storage locations of these images. The image information storage unit 322 is not limited to the above, and may store various types of information according to the purpose.

[0060] The processing information storage unit 323 stores information used for various information processes. The processing information storage unit 323 stores information used for generation processing. For example, the processing information storage unit 323 stores information used for generation processing of tracking data. For example, the processing information storage unit 323 stores information used for generation processing of report information to be provided.

[0061] The processing information storage unit 323 stores information used in the estimation process. For example, the processing information storage unit 323 stores information used in the volume estimation process of an object. The processing information storage unit 323 stores information used in the determination process. For example, the processing information storage unit 323 stores information used in the determination process of the presence or absence of an object. Note that the processing information storage unit 323 is not limited to the above, and may store various kinds of information according to the purpose.

[0062] The area information storage unit 324 stores information such as the type of area estimated by the estimation unit 334 described later. Fig. 6 is a diagram showing an example of the area information storage unit 324 according to the embodiment. In the example shown in Fig. 6, the area information storage unit 324 includes items such as "area ID", "location", "type", and "attribute".

[0063] "Area ID" indicates identification information for identifying an area. "Location" indicates the location of a partial area whose type has been estimated within an area. "Type" indicates the type of the partial area. "Attribute" indicates the attribute of the partial area.

[0064] In FIG. 6, the area identified as "A1" has three partial areas located at positions "P1," "P2," and "P3." The partial area "P1" has a type of "prohibited area" and an attribute of "work area." The partial area "P2" has a type of "prohibited area" and an attribute of "passage." The partial area "P3" has a type of "permitted area" and an attribute of "warehouse."

[0065] The control unit 330 is realized, for example, by a CPU, an MPU, or the like, executing various programs stored in an internal memory using a RAM as a working area. The control unit 330 can also be realized, for example, by an integrated circuit such as an ASIC or an FPGA. The control unit 330 has an acquisition unit 331, a first detection unit 332, a second detection unit 333, an estimation unit 334, and a processing unit 335, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of the control unit 330 is not limited to the configuration shown in FIG. 4, and may be other configurations as long as they perform information processing described below.

[0066] The acquisition unit 331 acquires various information. The acquisition unit 331 acquires various information from the storage unit 320. The acquisition unit 331 receives various information from an external device via the communication unit 310. The acquisition unit 331 receives information from the repeater 200. For example, the acquisition unit 331 may receive information collected by the lighting device 100 from the repeater 200. The acquisition unit 331 acquires an image captured by the camera 21.

[0067] The first detection unit 332 functions as an object detection processing unit that executes an object detection process to detect an object from an image acquired by the acquisition unit 331. The first detection unit 332 detects an object from an image stored in the storage unit 320. The first detection unit 332 executes the object detection process using a model M1 that is an object detection model. The first detection unit 332 generates information indicating a detection result of the object detection process.

[0068] Furthermore, the first detection unit 332 performs object detection based on the type of area estimated by the estimation unit 334 described later. Specifically, the first detection unit 332 reads out area information that is the estimation result of the estimation unit 334 from the area information storage unit 324, and determines whether or not a stationary object is present in a prohibited area among the detected objects.

[0069] Furthermore, when the prohibited area is an area exclusively for people, the first detection unit 332 determines whether or not a moving object other than a person (such as a forklift) is present in the prohibited area. When the prohibited area is an area exclusively for forklifts, the first detection unit 332 determines whether or not a moving object other than a forklift (such as a person) is present in the prohibited area.

[0070] Furthermore, the first detection unit 332 determines the type of area to be used for object detection based on the schedule information of the worker in the area. For example, when the worker is on a break (time 9 when production is temporarily suspended), the first detection unit 332 performs object detection by setting the prohibited area of ​​the passage dedicated to forklifts as a permitted area (area where people are permitted to pass). Furthermore, the first detection unit 332 acquires information on the product to be manufactured and information on the work process of the product as schedule information, and sets the passage and work area used in the work process as prohibited areas, and sets the passage and work area not used in the work process as permitted areas.

[0071] The second detection unit 333 functions as a moving object detection unit that detects a moving object from the image acquired by the acquisition unit 331. The second detection unit 333 executes a person detection process (also called a "person detection process") using, for example, a model M2 that is a person detection model. The second detection unit 333 generates information indicating a detection result of the person detection process.

[0072] The second detection unit 333 may execute a transport subject detection process to detect various transport subjects, not limited to a person (person). For example, the second detection unit 333 executes the forklift detection process using a model M3 that is a forklift detection model.

[0073] In addition, the second detection unit 333 detects the movement trajectory of the detected moving object in the area. In addition, when a stationary object (such as a pallet or luggage) is detected by the first detection unit 332 and a moving object such as a person or a forklift is detected near the stationary object, if the stationary object is moving together with the moving object, the second detection unit 333 detects that the moving object is transporting the stationary object.

[0074] The estimation unit 334 estimates the type of the area based on the detection result of the second detection unit 333. Specifically, the estimation unit 334 estimates a partial area corresponding to the position where a moving object exists among the areas shown in the video as a prohibited area where placement of a stationary object is prohibited. Specifically, the estimation unit 334 plots moving objects existing within a predetermined period on the video, and estimates a partial area with a density equal to or greater than a threshold as a prohibited area. In this case, the size of the prohibited area is estimated as the size of the largest quadrangle (which may be a triangle or a pentagon or more) formed with the positions of multiple moving objects existing within the predetermined period as vertices. In addition, the estimation unit 334 estimates as the prohibited area a partial area corresponding to the movement trajectory of the moving object. Specifically, the estimation unit 334 estimates as the prohibited area a partial area where a predetermined number or more of the movement trajectories of moving objects existing within a predetermined period are densely packed. In this case, the size of the prohibited area is estimated as the size of the prohibited area having the width of both ends of the two movement trajectories that are the farthest apart among the multiple movement trajectories. Furthermore, the estimation unit 334 estimates a partial area in which there is a predetermined number of stationary objects or more and no moving objects, as an allowed area.

[0075] The estimation unit 334 also estimates the attribute of the estimated prohibited area. Specifically, when the moving object present in the prohibited area is only a person and the stay time is a predetermined time or more, the estimation unit 334 estimates that the prohibited area is a work area for people. When the moving object present in the prohibited area is only a forklift and the stay time is a predetermined time or more, the estimation unit 334 estimates that the prohibited area is a work area for forklifts. When the moving object of the movement trajectory passing through the prohibited area is only a person, the estimation unit 334 estimates that the prohibited area is a passage for people. When the moving object of the movement trajectory passing through the prohibited area is only a forklift, the estimation unit 334 estimates that the prohibited area is a passage for forklifts. The estimation unit 334 generates the estimation result as area information and stores it in the area information storage unit 324.

[0076] In addition, when the passages and work areas in an area are fixed (unchanging depending on the type of product to be manufactured, etc.), the estimation unit 334 may set partial areas such as the passages and work areas as prohibited areas in advance, and estimate the types of partial areas other than the prohibited areas. This makes it possible to omit the estimation process for partial areas whose types do not need to be estimated, thereby reducing the processing load.

[0077] The processing unit 335 performs various processes based on the detection result of the first detection unit 332. For example, when an object prohibited from being present in the prohibited area is present, the processing unit 335 tags the image in which the object is captured among the images stored in the image information storage unit 322. At this time, the processing unit 335 tags the type of the object that was present (person, forklift, stationary object, etc.) and the position of the prohibited area in which the object was present. This allows the administrator to easily find the corresponding image by searching for the tagged information when searching for an image of an intrusion into the prohibited area later. In addition, when the processing unit 335 directly acquires an image from the camera 21 and performs object detection, when an object prohibited from being present in the prohibited area is present, it outputs an instruction signal to an alarm device (not shown) arranged in the prohibited area to issue an alarm. As a result, the alarm device issues an alarm according to the instruction signal, so that a worker in the vicinity of the area can easily understand that an object is present in the prohibited area.

[0078] From here, the flow of the area type estimation process in the information processing system 1 will be described. Fig. 7 is a flowchart showing an example of the area type estimation process. Fig. 8 is a flowchart showing an example of the object detection process. Note that, although the information processing system 1 will be described below as the processing subject, the processes shown in Figs. 7 and 8 may be performed by any of the devices such as the lighting device 100, the repeater 200, the server device 300, the terminal device 400, etc., depending on the device configuration included in the information processing system 1.

[0079] First, the area type estimation process will be described with reference to Fig. 7. As shown in Fig. 7, the information processing system 1 reads out the image stored in the image information storage unit 322 (step S101). Note that the image is not limited to being read out and acquired from the image information storage unit 322, and the image captured by the camera 21 may be acquired directly from the lighting device 100 in real time.

[0080] Next, the information processing system 1 analyzes the read video and determines whether or not a moving object has been detected from the video (step S102). Specifically, the information processing system 1 detects a moving object (a person, a forklift, a mobile robot, etc.) using the model data for object detection, human detection, and forklift detection stored in the AI ​​information storage unit 321.

[0081] When the information processing system 1 detects a moving object (step S102: Yes), it analyzes the video and detects the movement trajectory of the moving object in the video (step S103). Specifically, the information processing system 1 attaches a tag to each detected moving object, and detects tracking data (information on position change) that tracks the moving object for each tag as the movement trajectory.

[0082] Next, the information processing system 1 estimates the type of the area based on the position and movement trajectory of the detected moving object (step S104). Specifically, the information processing system 1 estimates the partial area corresponding to the position of the moving object in the entire area and the region of the movement trajectory as a prohibited area where the placement of stationary objects is prohibited, and estimates the partial area other than the prohibited area as an allowed area where the placement of stationary objects is permitted. In addition, the attribute of the prohibited area is estimated from the type of moving object present in the prohibited area. Specifically, the information processing system 1 estimates a prohibited area where a person stays for a certain period of time as a work area where the person performs work, and estimates a prohibited area where a person or a forklift passes within a certain period of time as an aisle.

[0083] Next, the information processing system 1 stores the estimated area type, area attribute, and location information as area information in the area information storage unit 324 (step S105), and ends the process.

[0084] Next, the object detection process will be described with reference to Fig. 8. As shown in Fig. 8, the information processing system 1 acquires an image captured by the camera 21 of the lighting device 100 (step S201). The image may be acquired directly from the lighting device 100 in real time, or an image stored in the image information storage unit 322 may be acquired.

[0085] Next, the information processing system 1 reads out the area information stored in the area information storage unit 324 (step S202).

[0086] Next, the information processing system 1 analyzes the video to detect objects present in the area (step S203). Specifically, the information processing system 1 detects moving objects (people, forklifts, mobile robots, etc.) and stationary objects using the model data for object detection, human detection, and forklift detection stored in the AI ​​information storage unit 321.

[0087] When an object is detected (step S203: Yes), the information processing system 1 determines whether or not a stationary object is present in the prohibited area based on the area information (step S204). Note that in step S204, it is determined whether or not a stationary object is present in the prohibited area, but it may further determine, for example, whether or not a person is present in an aisle reserved for forklifts, whether or not a forklift is present in an aisle reserved for people, whether or not a person is present in a work area reserved for forklifts, or whether or not a forklift is present in a work area reserved for people.

[0088] Next, if a stationary object is present in the prohibited area (step S204: Yes), the information processing system 1 performs tagging on the target video (step S205). Specifically, the information processing system 1 performs tagging indicating that a stationary object is present in the prohibited area. At this time, if there are multiple prohibited areas, the position (or identification information) of the prohibited area in which the stationary object is present may also be tagged.

[0089] Next, when the information processing system 1 is acquiring the video directly from the lighting device 1 in real time, it outputs an instruction signal to the notification device arranged in the area to make a notification (step S206). As a result, the notification device makes a notification, so that the worker can recognize that a stationary object has been erroneously placed in the prohibited area.

[0090] As described above, the information processing system 1 includes a memory unit 320 that stores video captured by a shooting unit (camera 21) of a specified area, a moving object detection unit (second detection unit 333) that detects a moving object contained in the video stored by the memory unit 320, an estimation unit 334 that estimates the type of area based on the detection result of the moving object detection unit, and an object detection unit (first detection unit 332) that detects objects in the area based on the estimated type of area.

[0091] This allows the information processing system 1 to automatically set the area type even when the layout of parts storage areas at the manufacturing site changes dynamically, or when the types of manufactured products change each time, etc. Therefore, the administrator does not need to change the area type setting each time, which reduces the administrator's troublesomeness.

[0092] [Other system configuration examples] The above-described configuration of the information processing system 1 is merely an example, and the information processing system 1 may have any device configuration as long as the desired processing is possible. For example, the information processing system 1 may have a configuration without the repeater 200. Furthermore, the various processes performed by the server device 300 may be executed by any device included in the information processing system 1, not limited to the server device 300. For example, at least a part of the various processes performed by the server device 300 (e.g., the first detection unit 332, the second detection unit 333, etc.) may be executed by the repeater 200 or the lighting device 100. In this way, the various processes performed by the server device 300 may be distributed and processed by multiple devices included in the information processing system 1.

[0093] Although an embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as well as the scope and gist of the invention. In addition, these embodiments and their modifications can be appropriately combined as long as the processing contents are not contradictory. [Explanation of symbols]

[0094] 1. Information Processing Systems 100 Lighting Equipment 10. Communications Department 20 Camera unit 30 Lighting Unit 40 Control section 41 Acquisition Department 42 Processing section 43 Transmitter 50 Storage section 200 Repeater 300 Server device (information processing device) 310 Communications Department 320 Storage section 321 AI information storage unit 322 Image information storage unit 323 Processing information storage unit 324 Area information storage unit 330 Control Unit 331 Acquisition Department 332 First detection unit 333 Second detection unit 334 Estimation Department 335 Processing section 400 Terminal Equipment

Claims

1. a storage unit that stores an image of a predetermined area captured by the imaging unit; a moving object detection unit that detects a moving object included in the video stored by the storage unit; an estimation unit that estimates a type of the area based on a detection result of the moving object detection unit; an object detection unit that detects an object in the area based on the estimated type of the area; An information processing device having the above configuration.

2. The estimation unit is A prohibited area, which is a partial area within the area in which the presence of an object is prohibited, is estimated as the type of the area; The object detection unit is Detect the presence or absence of the object in the prohibited area The information processing device according to claim 1 .

3. The video processing device further includes a processing unit that, when the object detection unit detects the presence of the object in the prohibited area, tags the video stored in the storage unit. The information processing device according to claim 2 .

4. a processing unit that outputs, when the object detection unit detects the presence of the object in the prohibited area, an instruction signal for instructing an alarm device disposed in the area to issue an alarm. The information processing device according to claim 2 .

5. The moving object detection unit is Detecting the position of the moving object in the area; The estimation unit is A partial area corresponding to the position of the moving object is estimated as the prohibited area. The information processing device according to claim 2 .

6. The moving object detection unit is Detecting a movement trajectory of the moving object in the area; The estimation unit is A partial area corresponding to the movement trajectory of the moving object is estimated as the prohibited area. The information processing device according to claim 2 .

7. The object detection unit is A type of the area to be used for the object detection is determined based on schedule information of the area. The information processing device according to claim 1 .

8. a storage step of storing in a storage unit an image of a predetermined area captured by the image capturing unit; a moving object detection step of detecting a moving object included in the video stored by the storage unit; an estimation step of estimating the type of the area based on a detection result of the moving object detection step; an object detection step of detecting an object in the area based on the estimated type of the area; An information processing method comprising:

9. A lighting device having a shooting unit for shooting an image; a storage unit that stores an image of a predetermined area captured by the imaging unit; a moving object detection unit that detects a moving object included in the video stored by the storage unit; an estimation unit that estimates a type of the area based on a detection result of the moving object detection unit; an object detection unit that detects an object in the area based on the estimated type of the area; An information processing system having the above configuration.

Images