Information processing apparatus and information processing method

Abstract

To provide an information processing apparatus and an information processing method that optimize processing related to image analysis.SOLUTION: In an information processing system in which an information processing apparatus, a manager terminal, and a plurality of cameras are connected with each other via a predetermined network, the information processing apparatus 1 is provided with: a receiving unit 351 that receives selection of a desired question sentence from a plurality of question sentences related to whether an object to be detected is in a predetermined state; a determination unit 353 that determines whether the object to be detected captured in an image photographed by a camera is in a state corresponding to the question sentence, by using a learned model trained to estimate an answer to the question sentence; and an output unit 354 that outputs a result of determination made by the determination unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] Conventionally, there are techniques for performing image analysis based on questions in natural language. For example, one such technique is a technique for detecting people from an image by image analysis and asking each of the detected people a question. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-71675 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology has room for improvement in optimizing the image analysis process. Specifically, since image analysis is performed for each person, if there are many people detected in the image, this may cause processing delays.

[0005] The present invention has been made in view of the above, and has an object to provide an information processing apparatus and an information processing method capable of optimizing processing relating to image analysis. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the objective, the information processing device of the present invention includes a reception unit that receives a selection of a desired question from a plurality of questions regarding whether or not the detection target is in a predetermined state, a judgment unit that uses a trained model trained to estimate an answer to the question to judge whether or not the detection target shown in an image captured by a camera is in a state corresponding to the question, and an output unit that outputs the judgment result by the judgment unit. Effect of the Invention

[0007] According to the present invention, the processing relating to image analysis can be optimized. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of the information processing device. [Diagram 3] FIG. 3 is a diagram illustrating an example of question information. [Figure 4] FIG. 4 is an explanatory diagram of the person score. [Diagram 5] FIG. 5 is a diagram illustrating an example of an output result by the output unit. [Figure 6] FIG. 6 is a flowchart showing a processing procedure executed by the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The information processing device 1 described below includes a receiving unit 351 that receives a selection of a desired question sentence from among multiple questions regarding whether or not the detection target is in a specified state, a judgment unit 353 that judges whether or not the detection target shown in an image captured by a camera is in a state corresponding to the question sentence using a trained model trained to estimate an answer to the question sentence, and an output unit 354 that outputs the judgment result by the judgment unit 353.

[0010] The information processing device 1 described below includes a detection unit 352 that detects detection targets from an image, and a judgment unit 353 judges whether or not a detection target with a high priority among the detection targets detected by the detection unit 352 is in a state where it is the target.

[0011] In the information processing device 1 described below, when the number of detection targets detected from an image exceeds a threshold value, the determination unit 353 determines whether or not the detection target is in a high priority state.

[0012] In the information processing device 1 described below, the receiving unit 351 receives a priority setting for each question sentence, and the determining unit 353 executes processing based on the question sentence of the priority.

[0013] In the information processing device 1 described below, the determination unit 353 executes the process corresponding to the question sentence with a high priority in real time, and processes the process corresponding to the question sentence with a low priority later.

[0014] The information processing device 1 described below includes a detection unit 352 that detects detection targets defined by a question sentence from an image, a determination unit 353 that determines whether the detection targets detected by the detection unit 352 are in a state defined by the question sentence using a trained model trained to estimate an answer to the question sentence, and an output unit 354 that outputs the determination result by the determination unit 353. When the number of detection targets detected by the detection unit 352 exceeds a predetermined number, the determination unit 353 executes processing on detection targets with high priority.

[0015] The information processing method described below includes a receiving step of receiving a selection of a desired question from among a plurality of questions regarding whether or not the detection target is in a predetermined state, a determination step of determining whether or not the detection target shown in an image captured by a camera is in a state corresponding to the question using a trained model trained to estimate an answer to the question, and an output step of outputting the determination result obtained by the determination step.

[0016] The information processing method described below includes a detection process for detecting a detection target defined by a question sentence from an image, a determination process for determining whether or not the detection target detected by the detection process is in a state defined by the question sentence using a trained model trained to estimate an answer to the question sentence, and an output process for outputting the determination result by the determination process, in which when the number of detection targets detected by the detection process exceeds a predetermined number, the determination process performs processing on detection targets with high priority.

[0017] <Embodiment> Hereinafter, a detailed description will be given of an embodiment (hereinafter, referred to as an "embodiment") for carrying out an information processing device and an information processing method according to the present application with reference to the drawings. Note that the information processing device and the information processing method according to the present application are not limited to the embodiment.

[0018] First, an overview of an information processing system according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing a configuration example of a lighting control system according to an embodiment. The information processing system S shown in Fig. 1 is a system that identifies, for example, a person not following a safety manual or a person in a dangerous state based on images captured by the system, for example, in a facility such as a manufacturing site or a factory.

[0019] 1, an information processing system S includes an information processing device 1, an administrator terminal 100, and a plurality of cameras C1 to Cn (n is a natural number). The information processing device 1, the administrator terminal 100, and the plurality of cameras C1 to Cn are connected to each other via a predetermined network N. In the following, when there is no need to distinguish between the cameras C1 to Cn, they will simply be referred to as "camera C."

[0020] The information processing device 1 detects, for example, a person not complying with a safety manual, a person in a dangerous state, etc., from an image captured by a camera C installed in a facility such as a manufacturing site or a factory.

[0021] As described later, the information processing device 1 performs image analysis using a trained model trained to estimate an answer to a question sentence. In this embodiment, the trained model is VQA (Visual Question Answering).

[0022] The information processing device 1 detects a detection target from an image captured by the camera C, and inputs the image of the detection target and the question text into the trained model to determine whether the detection target is in a state corresponding to the question text. In this embodiment, the detection target is a person (worker), but the detection target may be a work vehicle such as a forklift, a dolly, etc.

[0023] The administrator terminal 100 is a terminal owned by a manager of a facility such as a manufacturing site or a factory. For example, the administrator terminal 100 notifies the manager of information transmitted from the information processing device 1. For example, when the information processing device 1 detects that a worker has collapsed in the facility, it notifies the manager of an alert via the administrator terminal 100.

[0024] The camera C is installed in a facility such as a manufacturing site or a factory. For example, the camera C is installed on the ceiling of the facility so as to capture an image of the floor from the ceiling. The camera C captures images at a predetermined period and transmits each captured image to the information processing device 1 via the network N.

[0025] Here, in image analysis using the above-mentioned trained model, the target person (worker) is detected from the image captured by camera C, and the image and question text for each detected person are input into the trained model.

[0026] However, for example, if many people are detected from an image captured by camera C, the number of images and questions to be input into the trained model will increase, which may result in processing delays, etc.

[0027] For this reason, in the present disclosure, the information processing device 1 sets priorities for questions and people, and performs processing with a higher priority. In other words, the information processing device 1 does not apply image processing for all questions to all people, but limits the processing to questions with high importance and people with high importance, thereby preventing processing delays and the like while performing necessary image processing.

[0028] In addition, the information processing device 1 performs image processing that omits processing during a series of these processes during time periods when no person is detected from the image, etc., thereby leveling out the processing. In other words, the information processing device 1 according to the embodiment can optimize the processing related to image analysis.

[0029] Next, a configuration example of the information processing device 1 according to the embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of a functional configuration of the information processing device according to the embodiment. As shown in Fig. 2, the information processing device 1 includes a communication unit 31, a storage unit 34, and a control unit 35.

[0030] The communication unit 31 is implemented by a predetermined communication circuit such as a NIC (Network Interface Card), and performs data communication with various devices via a communication network such as Ethernet (registered trademark) or a LAN.

[0031] The storage unit 34 is implemented by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. In the example shown in Fig. 2, the storage unit 34 includes a question information storage unit 34a, a person score storage unit 34b, a person detection model storage unit 34c, and a VQA model storage unit 34d.

[0032] The question information storage unit 34a stores question information. Here, the question information will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of question information. As shown in Fig. 3, the question information is information in which information items such as "camera ID", "target question sentence", and "priority" are associated with each other.

[0033] The "camera ID" is an identifier for identifying the camera C, and the "target question" is a question to be executed for an image captured by the camera C identified by the corresponding camera ID. The question is a sentence related to the so-called fire ant hat, and an example is "Is it lying down?"

[0034] As described later, the target question sentence can be appropriately set by the administrator. "Priority" is the priority of the corresponding target question sentence. For example, a priority level is assigned to each target question sentence, from level 1 to level 5. In the following, level 5 is considered to have the highest priority.

[0035] Returning to the explanation of Fig. 2, the person score storage unit 34b will be explained. The person score storage unit 34b stores the person score. Here, the person score will be explained with reference to Fig. 4. Fig. 4 is an explanatory diagram of the person score.

[0036] 4 shows a schematic diagram of an image captured by a camera C. As shown in FIG 4, the image is divided into a plurality of regions Ar1 to Ar3, and a different person score is assigned to each region.

[0037] More specifically, area Ar1 is an area located in the center of the image, area Ar2 is assigned to the outer periphery of area Ar1, and area Ar3 is assigned to the outer periphery of area Ar2.

[0038] Here, if the camera C has a wide-angle lens such as a fisheye lens, the image becomes distorted from the inside to the outside of the image. Therefore, for example, the image resolution is poorer in areas Ar2 and Ar3 than in area Ar1, and the accuracy of image processing may decrease.

[0039] That is, the accuracy of image processing for areas Ar2 and Ar3 tends to be lower than that for area Ar1. Therefore, in the present disclosure, the person score for area Ar1 is set high, and the person scores for areas Ar2 and Ar3 are set lower.

[0040] As described later, by prioritizing processing related to a person with a high person score over processing related to a person with a low person score, it is possible to optimize processing related to image analysis. Note that, although Fig. 4 shows the object detection score in a schematic manner, in reality, the person detection score is calculated by a person detection model.

[0041] Returning to the description of Fig. 2, the human detection model storage unit 34c will be described. The human detection model storage unit 34c stores a human detection model. The human detection model is a model that has been trained to detect a human, which is a detection target, from an image.

[0042] The VQA model storage unit 34d stores a VQA model. The VQA model is a trained model trained to estimate an answer to a question. More specifically, when a question about whether a detection target is in a predetermined state and an image of the detection target are input, the VQA model estimates whether the detection target shown in the image is in the predetermined state defined by the question.

[0043] The control unit 35 is a controller that controls the entire information processing system S. For example, the control unit 35 can be implemented by a processor such as a central processing unit (CPU) or a micro processing unit (MPU). Alternatively, the control unit 35 may be implemented by an integrated circuit such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0044] In the example of FIG. 2, the control unit 35 includes a receiving unit 351, a detecting unit 352, a determining unit 353, and an output unit 354.

[0045] The receiving unit 351 receives a selection of a desired question sentence from among a plurality of questions regarding whether or not the detection target is in a predetermined state. For example, the receiving unit 351 receives a selection of a question sentence desired by the administrator from among the plurality of questions via the administrator terminal 100.

[0046] For example, the reception unit 351 presents all questions to the administrator and accepts the selection of a question desired by the administrator from among them. At this time, the reception unit 351 may present a package consisting of a plurality of questions to the administrator, and the administrator may select a package.

[0047] Furthermore, the reception unit 351 receives a priority setting for each question from the administrator. The reception unit 351 allows the administrator to select a priority level from level 1 to level 5 for each question. Note that this priority level may be set in advance on the information processing device 1 side.

[0048] The detection unit 352 detects a person, who is a detection target, from an image captured by the camera C. For example, the detection unit 352 detects a person from the image by inputting the image captured by the camera C to a person detection model stored in the person detection model storage unit 34c.

[0049] Then, the detection unit 352 scores the person detected from the image captured by the camera C based on the person score stored in the person score storage unit 34b, and passes the target range of the detected person to the determination unit 353.

[0050] The determination unit 353 uses a trained model (VQA model) that has been trained to estimate an answer to a question to determine whether or not the detection object appearing in an image captured by the camera is in a state that corresponds to the question.

[0051] As preprocessing for state determination, the determination unit 353 first counts the number of people detected by the detection unit 352. For example, when the number of people is smaller than a threshold, the determination unit 353 inputs images of all people and all question sentences to the VQA model.

[0052] On the other hand, when the number of people detected by the detection unit 352 is equal to or greater than the threshold, the determination unit 353 inputs only people with high person scores and questions with high priorities into the VQA model. In addition, for example, for free time in which no people are detected from the image, the determination unit 353 performs image analysis that omits processing in relation to the person scores and the priorities of the questions.

[0053] That is, the determination unit 353 executes a process corresponding to a question sentence with a high priority in real time, and processes a process corresponding to the question sentence with a low priority later. Then, when the score output from the VQA model is greater than a threshold, the determination unit 353 determines that the person in the input image is in a state defined by the question sentence.

[0054] The output unit 354 outputs the determination result by the determination unit 353. For example, the output unit 354 outputs the determination result by the determination unit 353 to the administrator terminal 100.

[0055] Here, an example of the output result by the output unit 354 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the output result by the output unit 354. As shown in Fig. 5, the output result by the output unit 354 includes items such as "date and time", "link", "camera name", "image", "question text", and "detection result".

[0056] "Date and time" indicates the date and time when the action to be output was performed. "Link" indicates the link (e.g., URL) connecting to the camera C to be output. "Camera name" indicates the name of the camera C that captured the action to be output.

[0057] "Image" indicates an image when the behavior to be output is performed. "Question" indicates the risky behavior to be output. "Detection result" indicates the detection result of the risky behavior to be output.

[0058] 5, the question is "Are you lying down?" and the image shows a person lying down. Therefore, the determining unit 353 determines that the question is "You are lying down" based on the output result of the VQA model.

[0059] Then, the output unit 354 outputs the output result regarding the event that the person has fallen as "Unsafe". Note that the output result shown in FIG. 5 is an example and is not limited to this. For example, when the "question sentence" is "Fallen", since a person who has fallen is shown in the image, the determination unit 353 may determine "Fallen" based on the output result of the VQA model, and the output result by the output unit 354 may be output as "Yes". In this way, the question sentence may not be an interrogative sentence such as "Are you doing AA?" or "Do you have BB?" as in "5W1H", but may be a sentence that simply describes a situation or action.

[0060] Next, a process procedure executed by the information processing device 1 according to the embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the process procedure executed by the information processing device 1. As shown in Fig. 6, the information processing device 1 first acquires a target image that is an image to be subjected to image analysis (step S101).

[0061] Next, the information processing device 1 detects people from the target image (step S102), and then determines whether the number of people detected from the target image is greater than a threshold value (step S103).

[0062] When it is determined that the number of people is greater than the threshold value (step S103; Yes), the information processing device 1 extracts people with high person detection scores (step S104) and selects questions with high priority (step S105).

[0063] Furthermore, when it is determined in step S103 that the number of people is equal to or less than the threshold value (step S103; No), the information processing device 1 selects all questions targeting all detected people (step S107).

[0064] Then, the information processing device 1 performs image analysis using the VQA model based on the processing results up to step S105 or step S107 (step S106), and ends the processing.

[0065] As described above, the information processing device 1 according to the embodiment includes a reception unit 351 that receives a selection of a desired question from among a plurality of questions regarding whether or not the detection target is in a predetermined state, a judgment unit 353 that judges whether or not the detection target shown in an image captured by the camera is in a state corresponding to the question using a trained model trained to estimate an answer to the question, and an output unit 354 that outputs the judgment result by the judgment unit 353.

[0066] Moreover, the information processing device 1 according to the embodiment includes a detection unit 352 that detects a detection target defined by a question sentence from an image, a determination unit 353 that determines whether or not the detection target detected by the detection unit 352 is in a state defined by the question sentence using a trained model trained to estimate an answer to the question sentence, and an output unit 354 that outputs the determination result by the determination unit 353. When the number of detection targets detected by the detection unit 352 exceeds a predetermined number, the determination unit 353 executes processing on detection targets with a high priority.

[0067] Therefore, in the information processing device 1 according to the embodiment, since it is possible to preferentially execute processing related to a selected question or a question with a high priority, it is possible to optimize processing related to image analysis.

[0068] Although an embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope of the invention and its equivalents described in the claims, as well as in the scope and gist of the invention. [Explanation of symbols]

[0069] 1. Information processing device 31 Communications Department 34 Storage section 34a Question information storage unit 34b Person score memory section 34c Person detection model storage unit 35 Control section 351 Reception 352 Detection unit 353 Judgment section 354 Output section C Camera

Claims

1. a reception unit that receives a selection of a desired question sentence from among a plurality of questions regarding whether the detection target is in a predetermined state; a determination unit that determines whether or not the detection target shown in an image captured by a camera is in a state corresponding to the question, using a trained model that has been trained to estimate an answer to the question; an output unit that outputs a determination result by the determination unit; An information processing device comprising:

2. a detection unit that detects the detection target from the image; Equipped with The determination unit is Among the detection targets detected by the detection unit, a detection target having a high priority is determined to be in the above-mentioned state. The information processing device according to claim 1 .

3. The determination unit is When the number of detection targets detected from the image exceeds a threshold, it is determined whether the detection target having the high priority is in the state. The information processing device according to claim 2 .

4. The reception unit is Accepting a priority setting for each of the questions; The determination unit is Execute a process from the question sentence of the priority The information processing device according to claim 1 .

5. The determination unit is A process corresponding to the question sentence having a high priority is executed in real time, and a process corresponding to the question sentence having a low priority is executed later. The information processing device according to claim 4.

6. a detection unit that detects a detection target defined by a query from an image; a determination unit that determines whether or not the detection target detected by the detection unit is in a state defined by the question sentence, using a trained model that has been trained to estimate an answer to the question sentence; an output unit that outputs a determination result by the determination unit; Equipped with The determination unit is When the number of the detection targets detected by the detection unit exceeds a predetermined number, a process is executed for the detection targets having a high priority. Information processing device.

7. a receiving step of receiving a selection of a desired question sentence from among a plurality of questions regarding whether the detection target is in a predetermined state; a determination step of determining whether or not the detection target shown in an image captured by a camera is in a state corresponding to the question sentence, using a trained model trained to estimate an answer to the question sentence; an output step of outputting a determination result obtained by the determination step; An information processing method comprising:

8. A detection step of detecting a detection target defined by the query from an image; a determination step of determining whether or not the detection target detected by the detection step is in a state defined by the question sentence, using a trained model trained to estimate an answer to the question sentence; an output step of outputting a determination result obtained by the determination step; Including, The determination step includes: When the number of the detection targets detected by the detection step exceeds a predetermined number, a process is executed for the detection targets having a high priority. Information processing methods.

Images