Information processing system, information processing device, and information processing method

Abstract

To suppress calculation costs and power costs in making inferences.SOLUTION: An information processing system includes a determination device and a switching device. The determination device includes: a first inference section for causing a first model to process processing object data; a first determination section for determining whether or not a processing result by the first model satisfies a first condition; a second inference section for causing a second model that processes at higher accuracy than the first model, to process the processing object data; a second determination section for determining whether or not a processing result by the second model satisfies a second condition; and a transmission section for transmitting a determination result to the switching device. The switching device includes a switching section which performs switching so that processing of the processing object data is executed by the second model, when the first determination section determines that the first condition is satisfied, and which performs switching so that processing of the processing object data is executed by the first model, when the second determination section determines that the second condition is satisfied.SELECTED DRAWING: Figure 5

Description

Technical Field

[0004] , , ,

[0001] The present invention relates to an information processing system, an information processing apparatus, and an information processing method.

Background Art

[0002] An event-driven two-layer inference system has been proposed in which a first-layer model for detecting light events is introduced before a second-layer model for performing production inference, and the inference of the second-layer model is performed only when it is determined to be necessary by referring to the detection result (evaluation value) of the event by the lightweight first-layer model, thereby reducing the frequency of the second-layer inference and improving the performance and power efficiency of the entire system.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional event-driven two-layer inference system, there are cases where the calculation cost and power cost cannot be suppressed during inference. For example, in the prior art, since both the first-layer model and the second-layer model perform inference while an event is occurring, there is a problem that the calculation cost and power cost are incurred.

[0005] The present invention has been made in view of the above, and has an object to provide an information processing system, an information processing device, and an information processing method that can reduce the cost of inference. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the information processing system of the present invention is an information processing system having a judgment device that judges the processing result and a switching device that switches the model that will process the data to be processed depending on the judgment by the judgment device, wherein the judgment device has a first inference unit that causes the data to be processed to be processed by a first model, a first judgment unit that judges whether the processing result by the first model satisfies a first condition, a second inference unit that causes the data to be processed to be processed by a second model that processes the data to be processed with higher accuracy than the first model, a second judgment unit that judges whether the processing result by the second model satisfies a second condition, and a transmission unit that transmits the result of the judgment to the switching device, and the switching device has a switching unit that, if it is judged that the first condition is satisfied, switches to have the second model process the data to be processed, and if it is judged by the second judgment unit that the second condition is satisfied, switches to have the first model process the data to be processed. [Effects of the Invention]

[0007] According to the present invention, it is possible to reduce the calculation cost and power cost during inference. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram for explaining the problems of the conventional technology. [Figure 2] FIG. 2 is a diagram for explaining the problems with the conventional technology. [Figure 3] FIG. 3 is a diagram for explaining an outline of the processing performed by the information processing system according to the embodiment. [Figure 4]FIG. 4 is a diagram illustrating an example of DNN1 and DNN2. [Figure 5] FIG. 5 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of the configuration of a determination device according to an embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of the configuration of a switching device according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of the switching process performed by the information processing system according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of the switching process performed by the information processing system according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of the switching process performed by the information processing system according to the embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. [Figure 12] FIG. 12 is a flowchart showing the flow of processing performed by the information processing system according to the embodiment. [Figure 13] FIG. 13 is a diagram showing a modified example of the information processing system according to the embodiment. [Figure 14] FIG. 14 is a diagram showing a modified example of the information processing system according to the embodiment. [Figure 15] FIG. 15 is a diagram showing a modified example of the information processing system according to the embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of a computer that implements an information processing system by executing a program. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, with reference to the drawings, an information processing system, an information processing device, and an information processing method according to the present application will be described in detail. Note that the present invention is not limited to these embodiments. In addition, in the description of the drawings, the same parts are denoted by the same reference numerals, and duplicated explanations will be omitted.

[0010] [Outline of processing by the information processing system] First, an overview of processing by an information processing system will be described with reference to Fig. 1 to Fig. 3. Fig. 1 and Fig. 2 are diagrams for explaining problems with the conventional technology. Fig. 3 is a diagram for explaining an overview of processing by an information processing system according to an embodiment.

[0011] An event-driven two-tier inference system exists as a conventional technology. An event-driven two-tier inference system is a system that performs the following processing. For example, as shown in FIG. 1, first, a first-layer model for event detection performs inference using processing target data such as video data (image data) as input. Next, an event determination unit refers to the inference result (event detection result) such as the evaluation value (or confidence level) by the first-layer model, and determines, for example, whether the evaluation value exceeds a threshold value.

[0012] If the event determination unit determines that the conditions are not met, the second-layer model terminates processing without performing actual inference, whereas if the event determination unit determines that the conditions are met, the second-layer model performs actual inference.By performing the above processing, conventional event-driven two-layer inference systems reduce the frequency of inference by the second-layer model, improving the performance and power efficiency of the entire system.

[0013] However, conventional event-driven two-tiered inference systems have the problem that they cannot reduce the computational and power costs for inference. For example, when an event occurs, such as the appearance of a person or an abnormal object in an image, it is expected that events will continue to occur continuously for a while. However, in conventional technologies, inference is performed using the first-layer model regardless of whether an event occurs or not, as shown in Figure 2(1), which shows the inference cost of an event-driven two-tiered inference system.

[0014] More specifically, in an event-driven two-tiered inference system, inference by the first-layer model is performed only during periods when no events are occurring (left side of Figure 2(1)), and inference by the second-layer model is performed in addition to inference by the first-layer model during periods when events are occurring (right side of Figure 2(1)). In other words, in conventional event-driven two-tiered inference systems, inference by the first-layer model is performed regardless of whether an event is occurring, and this always incurs the corresponding computational costs and power costs.

[0015] Therefore, as shown in Figure 2(2), which shows the inference cost of an ideal system, it is desirable that inference be performed not only by the second-layer inference model but also by the first-layer model only when necessary, so that inference is performed only by the first-layer model during periods when no events are occurring (left side of Figure 2(2)), and only by the second-layer model during periods when events are occurring (right side of Figure 2(2)).

[0016] Therefore, the information processing system switches the model that processes the data to be processed in accordance with the event determination result, as shown in Fig. 3. For example, the information processing system first causes the first model (first-layer model) to process the data to be processed. Then, the information processing system determines whether the inference result of the first model satisfies a predetermined condition. That is, the information processing system performs event detection using first-layer inference, as shown in the upper loop of Fig. 3, and determines the event detection result.

[0017] If the inference result of the first model is determined not to satisfy the predetermined condition, the information processing system terminates the process. On the other hand, if the inference result of the first model is determined to satisfy the predetermined condition, the information processing system switches the model that will execute subsequent processing of the processing target data from the first model to the second model (second-layer model).

[0018] Thereafter, the information processing system causes the second model to process the processing target data, and determines whether the inference result of the second model satisfies a predetermined condition different from the condition for determining the inference result of the first model. That is, the information processing system performs event detection using second-layer inference, as shown in the lower loop of Figure 3, and determines the event detection result.

[0019] Here, when it is determined that the inference result of the second model does not satisfy a predetermined condition different from the condition for determining the inference result of the first model, the information processing system causes the second model to process the processing target data without performing a switching process.On the other hand, when it is determined that the inference result of the second model satisfies a predetermined condition different from the condition for determining the inference result of the first model, the information processing system switches the model that executes the processing of the processing target data from the second model to the first model.

[0020] In this way, the information processing system can reduce the computational and power costs of the entire system by, for example, switching so that only the first-layer model performs inference when there is no object to be detected, and only the second-layer model performs inference when there is an object to be detected.

[0021] [Lightweight and high-precision models] Next, an example will be described in which a deep neural network (DNN) is used as a model for inference processing. The first model is a lightweight model (e.g., DNN1) that is fast and low-accuracy, and the second model is a high-accuracy model (e.g., DNN2) that is slow and high-accuracy. Note that in the information processing system of the embodiment, a neural network other than a DNN may be used, and low-computation-amount signal processing and high-computation-amount signal processing may be used instead of a trained model.

[0022] Figure 4 is a diagram illustrating an example of DNN1 and DNN2. A DNN has an input layer that receives data, one or more intermediate layers that transform the data input from the input layer, and an output layer that outputs inference results such as probabilities and likelihoods. The output values ​​output from each layer may be lossy if the input data needs to remain anonymous.

[0023] As shown in Figure 4, the information processing system may use independent DNN1a and DNN2a. For example, after DNN2a is trained using a known method, DNN1a is trained to determine which model, DNN1a or DNN2a, will perform inference based on the desired benefit of the user.

[0024] The user's requests include, for example, higher accuracy of inference results, reduction in data communication volume, faster calculation processing, resource optimization, etc. The evaluation value is a value that evaluates whether the target data should be processed using the first model or the second model while satisfying the user's requests. The more difficult it is to process the target data, the more the evaluation value tends to fall within a range where the second model should be used for evaluation.

[0025] DNN1a and DNN2a are trained to output values ​​related to the evaluation value. DNN1a outputs intermediate output values, which are output values ​​of the intermediate layer of DNN1a, as values ​​related to the evaluation value. DNN2a outputs intermediate output values, which are output values ​​of the intermediate layer of DNN2a, as values ​​related to the evaluation value. The evaluation value may be a value calculated based on the intermediate output values, or it may be the intermediate output value itself. The intermediate output value may be the result of applying a cost function, such as correlating the intermediate output value with the likelihood, that enables learning to better meet the user's needs, to a predetermined intermediate layer, or it may be the output of any intermediate layer of a trained model designed with only the problem to be solved as the cost function.

[0026] This is because, for example, when using a network (such as a CNN) that tends to reflect features that determine the input data in the higher-order intermediate layers, the output values ​​of the higher-order intermediate layers contain useful features that can be used in the problem to be solved. Note that DNN1a and DNN2a may be assigned the same task with different accuracy and performance, or different tasks.

[0027] Furthermore, the evaluation value is not limited to the intermediate output value output from DNN1a or DNN2a. For example, the evaluation value may be an inference error output from DNN1a or DNN2a, or a value based on the inference error. For example, the evaluation value may be a value indicating the degree of certainty that the result of processing performed by the first model or the second model is correct. Furthermore, the evaluation value may be a value determined based on the time required to obtain the processing result of the processing target data, the deadline for obtaining the processing result of the processing target data, the resource usage status when determining whether the processing target data should be processed by the first model or the second model, or whether the processing target data is data in which an event has occurred compared to other data.

[0028] The resource usage status may be the CPU or memory usage or power consumption of a single device that performs inference using the first or second model. The event may be, for example, a change in the target frame compared to the previous frame that exceeds a desired magnitude, or the occurrence of an object that requires detailed estimation. Data indicating the objects and results calculated by the first model may be transmitted to a server device, and the second model may be designed to calculate only objects that have not been calculated by the first model. Specifically, the bounding box coordinate values, the class classification results, and their reliability may be sent together, and only objects that do not satisfy the reliability may be calculated by the second model.

[0029] [Configuration of information processing system] Next, the configuration of an information processing system according to an embodiment will be described with reference to Fig. 5. The information processing system 1 is a system that processes data to be processed using a first model, determines whether the processing result by the first model satisfies a first condition, processes the data to be processed using a second model that processes the data to be processed with higher accuracy than the first model, determines whether the processing result by the second model satisfies a second condition, and if it is determined that the first condition is satisfied, switches the system so that the second model processes the data to be processed, and if it is determined that the second condition is satisfied, switches the system so that the first model processes the data to be processed.

[0030] 5, the information processing system 1 includes a determination device 100 and a switching device 200. The determination device 100 and the switching device 200 are connected by wire or wirelessly. For example, the determination device 100 and the switching device 200 are connected by an APN (All Photonics Network). Each of these devices will be described below.

[0031] The determination device 100 is an example of a computer that inputs processing target data into a first model or a second model, causes the model to perform inference processing, and determines the processing results of each model to determine an event. For example, the determination device 100 determines an event that a person has been detected from the result of the inference processing.

[0032] The switching device 200 is an example of a computer that switches a model that executes data to be processed according to a determination result. For example, when a determination is made that an event has occurred in which a person has been detected in the processing result of a first model, the switching device 200 switches the model that executes data to be processed to a second model. Furthermore, when a determination is made that an event has occurred in which a person has not been detected in the processing result of the second model, the switching device 200 switches the model that executes data to be processed to the first model.

[0033] [Configuration of the Determination Device] Next, the configuration of the determination device 100 will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the configuration of the determination device 100 according to the embodiment. As shown in Fig. 6, the determination device 100 includes a communication unit 110, a control unit 120, and a storage unit 130. Note that these units may be held in a distributed manner by a plurality of devices. The processing of these units will be described below.

[0034] The communication unit 110 is realized by a NIC (Network Interface Card) or the like, and enables communication between an external device and the control unit 120 via a telecommunication line such as a LAN (Local Area Network) or the Internet. For example, the communication unit 110 enables communication between the external device and the control unit 120.

[0035] The storage unit 130 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. Examples of information stored in the storage unit 130 include data to be processed (image data, etc.), inference results, judgment results, a model to input the data to be processed, data related to machine learning algorithms, trained models, and other information necessary for judgment processing. Note that the information stored in the storage unit 130 is not limited to the above.

[0036] The control unit 120 is realized using a CPU (Central Processing Unit), an NP (Network Processor), an FPGA (Field Programmable Gate Array), or the like, and executes a processing program stored in memory. As shown in Fig. 6, the control unit 120 has a first inference unit 121, a first determination unit 122, a second inference unit 123, a second determination unit 124, and a transmission unit 125. Each unit of the control unit 120 will be described below.

[0037] The first inference unit 121 processes the data to be processed using a first model. For example, the first inference unit 121 inputs inference data (data to be processed) into the DNN1 and causes it to perform inference. More specifically, the first inference unit 121 inputs image data transmitted from the switching unit 222 (described later) into the DNN1 and causes it to infer evaluation values ​​for the classification of objects (people, abnormal objects, etc.) and situations (falls, etc.) shown in the image. As described above, the evaluation values ​​include confidence levels, intermediate output values, inference errors, etc.

[0038] The first determination unit 122 determines whether the processing result by the first model satisfies a first condition. For example, the first determination unit 122 determines whether the evaluation value, which is the processing result by the first model, and a threshold have a predetermined relationship. For example, the first determination unit 122 determines whether the certainty, which is the processing result by the first model, is equal to or greater than a threshold. More specifically, in the case where the purpose is to detect a person, the first determination unit 122 determines that the certainty that a person appears in the processing result image by the first model is equal to or greater than the threshold when the certainty is 2 and the threshold is 1.

[0039] The first determination unit 122 can perform a determination not only on the processing result of the first model when one image frame is input, but also on the processing result of the first model when an image consisting of a plurality of consecutive frames is input. For example, the first determination unit 122 determines whether or not the processing result of the first model when a number of images consisting of a plurality of consecutive frames received from the user are input satisfies a first condition.

[0040] The second inference unit 123 processes the processing target data using a second model that processes the processing target data with higher accuracy than the first model. For example, the second inference unit 123 inputs the inference data (processing target data) into the DNN2 to obtain an inference result. More specifically, the second inference unit 123 inputs the image data transmitted from the switching unit 222 into the DNN2, and causes the DNN2 to infer an evaluation value regarding the classification of objects (people, abnormal objects, etc.) and situations (falls, etc.) shown in the image.

[0041] The second determination unit 124 determines whether the processing result by the second model satisfies a second condition. For example, the second determination unit 124 determines whether the evaluation value, which is the processing result by the second model, and a threshold have a predetermined relationship. For example, the second determination unit 124 determines whether the certainty, which is the processing result by the second model, is equal to or less than a threshold. More specifically, in the case of person detection, when the certainty that a person appears in the image as a processing result by the second model is 1 and the threshold is 2, the second determination unit 124 determines that the certainty is equal to or less than the threshold.

[0042] The second determination unit 124 can perform determination not only on the processing result of the second model when one image frame is input, but also on the processing result of the second model when an image consisting of a plurality of consecutive frames is input. For example, the second determination unit 124 determines whether the processing result of the second model when a number of images consisting of a plurality of consecutive frames received from the user are input satisfies the second condition.

[0043] The transmitting unit 125 transmits the determination result to the switching device 200. For example, the transmitting unit 125 transmits the determination result by the first determining unit 122 and the processing result by the second determining unit 124 to the switching device 200.

[0044] [Configuration of switching device] Next, the configuration of the switching device 200 will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the configuration of the switching device 200 according to the embodiment. As shown in Fig. 7, the switching device 200 includes a communication unit 210, a control unit 220, and a storage unit 230. Note that these units may be held in a distributed manner by a plurality of devices. The processing of these units will be described below.

[0045] The communication unit 210 is realized by a NIC (Network Interface Card) or the like, and enables communication between the control unit 220 and an external device via an electric communication line such as a LAN (Local Area Network) or the Internet. For example, the communication unit 210 enables communication between the determination device 100 and the control unit 220.

[0046] The storage unit 230 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. Examples of information stored in the storage unit 230 include data to be processed (image data, etc.), inference results, judgment results, a model for inputting data to be processed, and other information necessary for switching processing. Note that the information stored in the storage unit 230 is not limited to the above.

[0047] The control unit 220 is realized using a CPU (Central Processing Unit), an NP (Network Processor), an FPGA (Field Programmable Gate Array), etc., and executes a processing program stored in a memory. As shown in Fig. 7, the control unit 220 has an acquisition unit 221, a switching unit 222, and a deletion unit 223. Each unit of the control unit 220 will be described below.

[0048] The acquisition unit 221 acquires data to be processed. For example, the acquisition unit 221 acquires image data as the data to be processed. Here, the image data acquired by the acquisition unit 221 may be made up of multiple consecutive frames, and each frame may include an identifier such as a video number or a frame number.

[0049] When the first determination unit 122 determines that the first condition is satisfied, the switching unit 222 switches to have the second model execute the processing of the data to be processed, and when the second determination unit 124 determines that the second condition is satisfied, the switching unit 222 switches to have the first model execute the processing of the data to be processed. For example, when the switching unit 222 receives a determination result from the transmission unit 125 of the determination device 100 that the processing result by the first determination unit 122 satisfies the first condition, the switching unit 222 changes the input destination of the data to be processed to the second model and transmits the data to be processed to the second model.

[0050] On the other hand, when the switching unit 222 receives a judgment result from the transmission unit 125 of the judgment device 100 that the processing result by the second judgment unit 124 satisfies the second condition, it changes the input destination of the data to be processed to the first model and transmits the data to be processed to the first model.

[0051] That is, for example, when a person is not detected, the switching unit 222 transmits the processing target data to the first model, and when a person is detected, the switching unit 222 transmits the processing target data to the second model.

[0052] When switching has been performed so that the second model is to process the data to be processed, switching unit 222 causes the second model to process the data to be processed related to the processing result determined to satisfy the first condition by first determination unit 122. For example, when switching has been performed so that the second model is to process the data to be processed, switching unit 222 transmits the data to be processed related to the processing result determined to satisfy the first condition by first determination unit 122 to the second model.

[0053] In other words, when the model for performing data processing is switched from the first model to the second model, the data to be processed that triggered the switching process is input into the second model, making it possible to process the data to be processed using the second model with higher accuracy.

[0054] When switching unit 222 switches to have the second model execute the processing of the data to be processed, deletion unit 223 deletes the processing result by the first model immediately before the switching. For example, when switching unit 222 switches to have the second model execute the processing of the data to be processed, deletion unit 223 deletes the evaluation value output by the first model immediately before the switching from storage unit 230. In other words, when processing results for the same image data are generated by both the first model and the second model, deletion unit 223 ensures data consistency at the time of switching by deleting the processing result by the first model.

[0055] [Switching process] Next, an example of the switching process performed by the information processing system 1 according to the embodiment will be described with reference to Fig. 8 to Fig. 10. Fig. 8 to Fig. 10 are diagrams for explaining an example of the switching process performed by the information processing system 1 according to the embodiment.

[0056] For example, when the purpose is to detect a person, the information processing system 1 performs switching processing according to the flow shown in Fig. 8. First, the first inference unit 121 inputs image data to the first model (lightweight object detection / moving object detection) and causes it to infer whether a person appears in the image. That is, the first inference unit 121 causes the first model to infer an evaluation value (certainty factor) for the class classification of the object (person) appearing in the image. Next, the first determination unit 122 determines whether the evaluation value (certainty factor), which is the processing result by the first model, is equal to or greater than a threshold value.

[0057] Here, if the first determination unit 122 determines that the evaluation value (certainty) is not equal to or greater than the threshold, it is determined that no person appears in the image, and the information processing system 1 terminates the processing. On the other hand, if the first determination unit 122 determines that the evaluation value (certainty) is equal to or greater than the threshold, it is determined that a person appears in the image, and the switching unit 222 changes the input destination of the image data to the second model, and switches so that the second model processes the image data.

[0058] Subsequently, since the input destination of the image data has been changed to the second model by the switching unit 222, the second inference unit 123 inputs the image data to the second model (actual object detection). The second model infers whether a person appears in the image and the position of the person appearing in the image. That is, the second inference unit 123 causes the second model to infer an evaluation value (certainty) for the class classification of the object (person) appearing in the image and the detailed coordinate position of the person.

[0059] Next, the second determination unit 124 determines whether the evaluation value (certainty) that is the processing result of the second model is equal to or less than a threshold. If the second determination unit 124 determines that the evaluation value (certainty) is not equal to or less than the threshold, it is determined that a person appears in the image, and the second inference unit 123 outputs detailed coordinate information of the person inferred by the second model. On the other hand, if the second determination unit 124 determines that the evaluation value (certainty) is equal to or less than the threshold, it is determined that a person does not appear in the image, and the switching unit 222 changes the input destination of the image data to the first model and switches so that the first model processes the image data.

[0060] Although Figure 8 describes an example of switching processing due to the detection of a person, the information processing system 1 can perform switching processing in response to the detection results of various events, such as when switching processing is performed due to the detection of an abnormal object as shown in Figure 9, or when switching processing is performed due to the detection of a fall as shown in Figure 10.

[0061] Next, an example will be described in which the information processing system 1 allocates data to be processed and aggregates inference results during switching processing. Fig. 11 is a diagram showing an example of the configuration of the information processing system 1 according to the embodiment. Fig. 11 shows an image allocation and inference result aggregation mechanism (Fig. 11(1)) corresponding to the acquisition unit 221, the switching unit 222, the deletion unit 223, and the storage unit 230, a lightweight object detection pipeline (Fig. 11(2)) corresponding to the first inference unit 121, the first determination unit 122, the transmission unit 125, and the storage unit 130, and a production object detection pipeline (Fig. 11(3)) corresponding to the second inference unit 123, the second determination unit 124, the transmission unit 125, and the storage unit 130.

[0062] The image distribution and inference result aggregation mechanism in Figure 11(1) has an image area that stores images (including meta-information) captured by an imaging device such as a camera, an inference result area that stores inference results, an event area that stores events, an inference pipeline endpoint that stores information on whether the lightweight object detection pipeline or the actual object detection pipeline is to be the endpoint, and a switching decision and destination setting that switches the pipeline to which images are sent for each frame depending on the event stored in the event area.

[0063] The lightweight object detection pipeline in Figure 11(2) receives images, detects lightweight objects, and sends events (e.g., whether a person is present or not). The production object detection pipeline in Figure 11(3) receives images, detects production objects, and sends events (e.g., whether a person is present or not) and bounding boxes. The image sorting and inference result aggregation mechanism, lightweight object detection pipeline, and production object detection pipeline may have a buffer pool. Furthermore, existing technologies such as DMA (Direct Memory Access) and RDMA (Remote Direct Memory Access) are used for sending and receiving data.

[0064] An example of switching from a lightweight object detection pipeline to a production object detection pipeline is described below. When the lightweight object detection pipeline in Figure 11(2) determines that a person is present, information about the event indicating that a person is present is sent to the image sorting and inference result aggregation mechanism in Figure 11(1). Then, the switching determination and destination setting of the image sorting and inference result aggregation mechanism in Figure 11(1) switches the image transmission (input) destination to the production object detection pipeline according to the event indicating that a person is present stored in the event area.

[0065] Here, the event detection results used to determine whether to switch from the lightweight motion detection pipeline to the actual object detection pipeline are not limited to the event detection results for one frame. For example, the switching determination and destination setting of the image sorting and inference result aggregation mechanism in Figure 11(1) can switch pipelines according to the event detection results for multiple consecutive frames. This takes into account the accuracy issues of lightweight models and prevents overhead caused by switching inference models due to false positives.

[0066] In addition, the image sorting and inference result aggregation mechanism in Figure 11(1) buffers images from the frame where an event is detected to the frame where the event is no longer detected, and when switching occurs, it sends the buffered consecutive frames to the production object detection pipeline, and when switching does not occur, it deletes the buffered consecutive frames. This prevents image frames from not being processed by the production object detection pipeline due to the time lag between event detection and when switching is determined.

[0067] On the other hand, when switching from the lightweight object detection pipeline to the production object detection pipeline occurs, the image sorting and inference result aggregation mechanism does not aggregate the inference results of the lightweight object detection pipeline. Alternatively, the image sorting and inference result aggregation mechanism deletes the inference results of the lightweight object detection pipeline. In other words, when inference results for the same image data are output from both pipelines, the image sorting and inference result aggregation mechanism aggregates only the inference results from the production object detection pipeline. This makes it possible to maintain consistency before and after the pipeline switch.

[0068] Next, we will explain an example of switching from the production object detection pipeline to the lightweight object detection pipeline. When the production object detection pipeline in Figure 11(3) determines that there is no person, information about the event indicating that there is no person is sent to the image sorting and inference result aggregation mechanism in Figure 11(1). Then, the switching judgment and destination setting of the image sorting and inference result aggregation mechanism in Figure 11(1) switches the image destination to the lightweight object detection pipeline according to the event indicating that there is no person stored in the event area.

[0069] When switching from the production object detection pipeline to the lightweight motion detection pipeline, the switching decision and destination setting of the image sorting and inference result aggregation mechanism in Figure 11(1) can perform the switching process immediately upon detecting an event indicating no person. In other words, by taking into account the difference in accuracy between the metric model and the production model and quickly switching by relying on the event detection by the highly accurate production model, it is possible to reduce calculation and power costs.

[0070] In addition, when receiving images captured by multiple cameras, the image sorting and inference result aggregation mechanism in Figure 11(1) may batch frame images from multiple videos to improve the efficiency of GPU processing used for inference, etc.

[0071] 〔flowchart〕 Next, the flow of processing by the information processing system 1 will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of the flow of processing by the information processing system 1 according to this embodiment.

[0072] First, the acquisition unit 221 acquires processing target data (step S101). For example, the acquisition unit 221 acquires an image captured by an imaging device as the processing target data. Next, the switching unit 222 determines whether the entity that executes the processing of the processing target data is the second model (step S102). Here, if the switching unit 222 determines that the entity that executes the processing of the processing target data is not the second model (step S102: No), the switching unit 222 transmits the processing target data to the first model (step S103).

[0073] Next, the first inference unit 121 executes the first model on the data to be processed (step S104). For example, the first inference unit 121 inputs image data into the DNN1 and causes it to infer evaluation values ​​for the class classification of objects and situations shown in the image.

[0074] Next, the first determination unit 122 determines whether the processing result by the first model satisfies the first condition (step S105). For example, the first determination unit 122 determines whether the evaluation value (certainty factor) that is the processing result by the first model is equal to or greater than a threshold. Here, if the first determination unit 122 determines that the first condition is not satisfied (step S105: No), the information processing system 1 ends the processing.

[0075] On the other hand, if the first determination unit 122 determines that the first condition is satisfied (step S105: Yes), the transmission unit 125 transmits the determination result to the switching device 200, and the switching unit 222 switches the data to be processed so that the second model processes the data to be processed (step S106). For example, the switching unit 222 sets the destination of the data to be processed to the second model. Then, the processing of step S102 is performed again.

[0076] In step S102, if the switching unit 222 determines that the entity that executes the processing of the processing target data is the second model (step S102: Yes), the switching unit 222 transmits the processing target data to the second model (step S107).

[0077] Next, the second inference unit 123 executes the second model on the data to be processed (step S108). For example, the second inference unit 123 inputs image data into the DNN2 and causes it to infer evaluation values ​​for the class classification of objects and situations shown in the image.

[0078] Next, the second determination unit 124 determines whether the processing result by the second model satisfies a second condition (step S109). For example, the second determination unit 124 determines whether the evaluation value (certainty factor) that is the processing result by the second model is less than a threshold value.

[0079] If the second determination unit 124 determines that the second condition is satisfied (step S109: Yes), the transmission unit 125 transmits the determination result to the switching device 200, and the switching unit 222 switches so that the first model performs the processing (step S110). For example, the switching unit 222 sets the destination of the data to be processed to the first model. Then, the processing of step S101 is performed again.

[0080] On the other hand, if the second determination unit 124 determines that the second condition is not satisfied (step S109: No), the process of step S101 is performed again.

[0081] 〔effect〕 Thus, the information processing system 1 according to the embodiment includes a determination device 100 that determines the processing result, and a switching device 200 that switches the destination of the data to be processed in accordance with the determination by the determination device 100. The determination device 100 includes a first inference unit 121 that causes the data to be processed to be processed by a first model, a first determination unit 122 that determines whether the processing result by the first model satisfies a first condition, a second inference unit 123 that causes the data to be processed to be processed by a second model that processes the data with higher accuracy than the first model, a second determination unit 124 that determines whether the processing result by the second model satisfies a second condition, and a transmission unit 125 that transmits the determination result to the switching device 200. The switching device 200 includes a switching unit 222 that, if the first determination unit 122 determines that the first condition is satisfied, switches the processing of the data to be processed to be performed by the second model, and, if the second determination unit 124 determines that the second condition is satisfied, switches the processing of the data to be processed to be performed by the first model.

[0082] As a result, the information processing system 1 according to the embodiment can reduce computational costs and power costs during inference by, for example, switching so that when there is no object to be detected, only the first layer model performs inference, and when there is an object to be detected, only the second layer model performs inference.

[0083] That is, the information processing system 1 according to the embodiment can improve the performance and power efficiency of the entire system by switching models so that only one inference model is always in operation.

[0084] When switching is performed so that the second model processes the data to be processed, the switching unit 222 of the information processing system 1 according to the embodiment causes the second model to process the data to be processed related to the processing result determined to satisfy the first condition by the first determination unit 122. As a result, when the model for data processing is switched from the first model to the second model, the information processing system 1 inputs the data to be processed that triggered the switching process into the second model, thereby enabling the data to be processed to be processed using the second model with higher accuracy and maintaining consistency before and after the switching.

[0085] The switching device 200 of the information processing system 1 according to the embodiment further includes a deletion unit 223 that deletes the processing result by the first inference unit 121 immediately before the switching, when the switching unit 222 switches to have the second inference unit 123 execute processing of the data to be processed. This allows the information processing system 1 to store only the processing result by the second model, which has higher accuracy, even when processing results are output from both the first model and the second model using the same data to be processed as input.

[0086] The first determination unit 122 of the information processing system 1 according to the embodiment determines whether the processing result of an image consisting of multiple consecutive frames according to the first model satisfies a first condition, and the second determination unit 124 determines whether the processing result of an image consisting of multiple consecutive frames according to the second model satisfies a second condition. This allows the information processing system 1 according to the embodiment to prevent overhead caused by switching the inference model that performs processing due to erroneous detection.

[0087] [Modification] In Figure 11 above, we have explained the switching process that is performed by the cooperation of the image sorting and inference result aggregation mechanism of Figure 11(1), which corresponds to the acquisition unit 221, the switching unit 222, the deletion unit 223, and the memory unit 230; the lightweight object detection pipeline of Figure 11(2), which corresponds to the first inference unit 121, the first determination unit 122, the transmission unit 125, and the memory unit 130; and the production object detection pipeline of Figure 11(3), which corresponds to the second inference unit 123, the second determination unit 124, the transmission unit 125, and the memory unit 130. However, various variations in the arrangement of each of these components are possible.

[0088] For example, as shown in Fig. 11, the image sorting and inference result aggregation mechanism, lightweight object detection pipeline, and production object detection pipeline may all exist within a server terminal. Also, as shown in Fig. 13, the image sorting and inference result aggregation mechanism may exist in an edge device, and the lightweight object detection pipeline and production object detection pipeline may exist within a server device.

[0089] In addition, there may be cases where the image sorting and inference result aggregation mechanism and the lightweight object detection pipeline exist in an edge device, and the actual object detection pipeline exists in a server device, as shown in Fig. 14. In addition, there may be cases where the image sorting and inference result aggregation mechanism exists in an edge device, and the lightweight object detection pipeline and the actual object detection pipeline exist in different server devices, as shown in Fig. 15.

[0090] The components of each device illustrated above are conceptual functional units and do not necessarily have to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to that illustrated, and all or part of each device can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0091] 〔program〕 It is also possible to create a program written in a computer-executable language that executes the processes executed by the information processing system 1 described in the above embodiment. In this case, the same effects as those of the above embodiment can be achieved by having a computer execute the program. Furthermore, such a program may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read and executed by a computer to achieve the same processes as those of the above embodiment.

[0092] Fig. 16 is a diagram showing an example of a computer that executes an information processing program. As shown in Fig. 16, a computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0093] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0094] 16, the hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. The tables described in the above embodiments are stored in the hard disk drive 1090 or the memory 1010, for example.

[0095] The control program is stored in the hard disk drive 1090 as a program module in which instructions to be executed by the computer 1000 are written. Specifically, the hard disk drive 1090 stores a program module 1093 in which each process executed by the controller 2000 described in the above embodiment is written.

[0096] Furthermore, data used for information processing by the control program is stored as program data, for example, in the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 and program data 1094 stored in the hard disk drive 1090 into the RAM 1012 as necessary, and executes each of the above-described procedures.

[0097] The program module 1093 and program data 1094 related to the control program are not limited to being stored in the hard disk drive 1090, but may be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100, etc. Alternatively, the program module 1093 and program data 1094 related to the control program may be stored in another computer connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.

[0098] 〔others〕 Although various embodiments have been described in detail herein with reference to the drawings, these embodiments are merely examples and are not intended to limit the present invention. The features described herein can be realized in various ways, including various modifications and improvements based on the knowledge of those skilled in the art.

[0099] Furthermore, the above-mentioned "module (-er suffix, -or suffix)" can be read as a unit, means, circuit, etc. For example, a communication module, a control module, and a storage module can be read as a communication unit, a control unit, and a storage unit, respectively. [Explanation of symbols]

[0100] 1. Information Processing Systems 100 Judgment device 110 Communications Department 120 control section 121 1st Reasoning Part 122 1st Judgment Department 123 Second Reasoning Part 124 Second Judgment Section 125 Transmitter 130 Storage section 200 Switching Device 221 Acquisition Department 222 Switching section 223 Deleted section 230 Storage section

Claims

1. An information processing system having a determination device that determines a processing result, and a switching device that switches a model that executes processing of processing target data in accordance with the determination by the determination device, The determination device a first inference unit that causes a first model to process the processing target data; a first determination unit that determines whether a processing result based on the first model satisfies a first condition; a second inference unit that processes the processing target data using a second model that processes the processing target data with higher accuracy than the first model; a second determination unit that determines whether a processing result based on the second model satisfies a second condition; a transmitting unit that transmits the result of the determination to a switching device; and The switching device a switching unit that switches the processing of the processing target data to be executed by the second model when the first determination unit determines that the first condition is satisfied, and that switches the processing of the processing target data to be executed by the first model when the second determination unit determines that the second condition is satisfied; An information processing system comprising:

2. The switching unit is When the process is switched to have the second model execute the processing of the processing target data, the second model executes the processing of the processing target data related to the processing result determined by the first determination unit to satisfy the first condition.

2. The information processing system according to claim 1, wherein:

3. The switching device a deletion unit that deletes the processing result by the first model immediately before the switching is performed when the switching unit switches to have the second model execute the processing of the processing target data.

3. The information processing system according to claim 2, further comprising:

4. the first determination unit determines whether or not a processing result of an image made up of a plurality of consecutive frames using the first model satisfies the first condition; The second determination unit determines whether or not a processing result of an image made up of a plurality of consecutive frames using the second model satisfies the second condition.

2. The information processing system according to claim 1, wherein:

5. a first inference unit that causes a first model to process the processing target data; a first determination unit that determines whether a processing result based on the first model satisfies a first condition; a second inference unit that processes the processing target data using a second model that processes the processing target data with higher accuracy than the first model; a second determination unit that determines whether a processing result based on the second model satisfies a second condition; a switching unit that switches the processing of the processing target data to be executed by the second model when the first determination unit determines that the first condition is satisfied, and that switches the processing of the processing target data to be executed by the first model when the second determination unit determines that the second condition is satisfied; An information processing device comprising:

6. An information processing method executed in an information processing system having a determination device that determines a processing result, and a switching device that switches a model that executes processing of processing target data in accordance with the determination by the determination device, a first inference step in which the determination device processes the processing target data using a first model; a first determination step in which the determination device determines whether or not a processing result based on the first model satisfies a first condition; a second inference step in which the determination device processes the processing target data using a second model that processes the processing target data with higher accuracy than the first model; a second determination step in which the determination device determines whether or not a processing result based on the second model satisfies a second condition; a transmission step in which the determination device transmits a result of the determination to a switching device; a switching step of the switching device switching the processing of the processing target data to be executed by the second model when it is determined in the first determination step that the first condition is satisfied, and switching the processing of the processing target data to be executed by the first model when it is determined in the second determination step that the second condition is satisfied; An information processing method comprising:

Images