Edge device and object detection system

Abstract

Even when detecting moving objects within the sensing range of an object detection sensor, this makes it possible to further reduce the amount of data transmitted from the edge device to the server device and the processing load on the server device. [Solution] The system includes an edge-side inference unit 102 connected to an object detection sensor 20 that is installed with a fixed sensing range, which detects moving objects using a first inference model from the sensing results sequentially sensed by the object detection sensor 20; a difference detection unit 103 that detects the presence of a moving object based on the difference between the sensing results; and a server transmission determination unit 104 that transmits the sensing results to a server device 30 if there is a discrepancy in whether or not a moving object is detected between the edge-side inference unit 102 and the difference detection unit 103.

Description

Technical Field

[0001] The present disclosure relates to an edge device and an object detection system.

Background Art

[0002] Edge computing that processes data collected by sensors is also known in edge devices installed at the ends of networks. For example, Patent Document 1 discloses a processing system that performs inference using a lightweight model in an edge device while performing inference using a high-precision model in a server device. The lightweight model and the high-precision model take an image as input and infer the probability for each class of objects depicted in the image. In the technology of Patent Document 1, when the confidence level of inference of an input image in the edge device is less than a threshold value, the image to be processed is output to the server device, and inference is executed in the server device. The inference result in the server device is used for re-learning the lightweight model of the edge device. In the technology of Patent Document 1, re-learning of the lightweight model of the edge device is triggered by an increase in the processing rate in the server device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the technology disclosed in Patent Document 1 has a problem when a moving object within the imaging range of the imaging device is used as the object for object detection. The details are as follows: When a moving object is used as the object for object detection, situations frequently occur where the moving object is not located within the imaging range. Therefore, many images that do not include the moving object are obtained. In the technology disclosed in Patent Document 1, the confidence level of the input image inference at the edge device falls below the threshold for images that do not include the object for detection. Consequently, an excessive number of images that do not include the object for detection are output to the server device, increasing the amount of data transmitted to the server device and the processing load on the server device.

[0005] One objective of this disclosure is to provide an edge device and object detection system that enables the amount of data transmitted from the edge device to the server device and the processing load on the server device to be reduced, even when the object detection target is a moving object that is moving within the sensing range of the sensor for object detection. [Means for solving the problem]

[0006] The above objectives are achieved by a combination of features described in the independent claims, and the subordinate claims provide further advantageous specific examples of the disclosure. The reference numerals in parentheses in the claims indicate correspondences with specific means described in the embodiments described later as one aspect, and do not limit the technical scope of this disclosure.

[0007] To achieve the above objective, the edge device of this disclosure is an edge device installed at the end of a network and connected to server devices (30, 30a, 30b, 30c) via the network, and is connected to an object detection sensor (20) installed with a fixed sensing range, and includes a sensing result acquisition unit (101) that acquires sensing results sequentially sensed by the sensor, an edge-side inference unit (102, 102a, 102b, 102c) that detects a moving object using a first inference model, which is a machine learning-prepared inference model for detecting a moving object to be detected, from the sensing results acquired by the sensing result acquisition unit, and sequentially sensed by the sensing result acquisition unit The system includes a difference detection unit (103) that detects the presence of a moving object based on the difference between the next acquired sensing results, and a server transmission determination unit (104) that determines whether or not to transmit the sensing results acquired by the sensing result acquisition unit to a server device that detects moving objects using a second inference model, which is a machine learning-trained inference model for detecting moving objects and has higher detection accuracy than the first inference model, based on the discrepancy between whether or not a moving object is detected between the edge-side inference unit and the difference detection unit. The server transmission determination unit transmits the sensing results used for detecting moving objects in the edge-side inference unit to the server device if there is a discrepancy between whether or not a moving object is detected between the edge-side inference unit and the difference detection unit.

[0008] Furthermore, in order to achieve the above objective, the object detection system of this disclosure is an object detection system that includes server devices (30, 30a, 30b, 30c) and edge devices (10, 10a, 10b, 10c) provided at the end of the network and connected to the server devices via the network, wherein the edge devices are connected to an object detection sensor (20) installed with a fixed sensing range, and include a sensing result acquisition unit (101) that acquires sensing results sequentially sensed by the sensor, an edge-side inference unit (102, 102a, 102b, 102c) that detects a moving object using a first inference model, which is a machine learning-trained inference model for detecting a moving object to be detected, from the sensing results acquired by the sensing result acquisition unit, and sequentially from the sensing result acquisition unit The system includes a difference detection unit (103) that detects the presence of a moving object based on the difference between the acquired sensing results, and a server transmission determination unit (104) that determines whether or not to transmit the sensing results acquired by the sensing result acquisition unit to the server device based on the discrepancy between the presence or absence of detection of a moving object between the edge-side inference unit and the difference detection unit. The server transmission determination unit transmits the sensing results used for detecting the moving object in the edge-side inference unit to the server device if there is a discrepancy between the presence or absence of detection of a moving object between the edge-side inference unit and the difference detection unit. The server device includes a server-side inference unit (301, 301a) that detects the moving object from the sensing results transmitted from the edge device using a second inference model, which is a machine learning-prepared inference model for detecting moving objects and has higher detection accuracy than the first inference model.

[0009] With the above configuration, since the sensing range of the object detection sensor is fixed, the difference detection unit can more easily detect the presence of a moving object based on the difference between the sensing results. Also, since the difference detection unit can detect the presence of a moving object, if there is a discrepancy between the detection of a moving object between the edge-side inference unit and the difference detection unit, there is a possibility of false detection in the first inference model. By limiting the transmission of the sensing results used to detect the moving object to the server device only in cases where there is a possibility of false detection in the first inference model, it is possible to further reduce the amount of data transmitted from the edge device to the server device. In addition, this reduces the amount that must be processed on the server side, and thus reduces the processing load on the server device. As a result, even when a moving object moving within the sensing range of the object detection sensor is the target of object detection, it is possible to further reduce the amount of data transmitted from the edge device to the server device and the processing load on the server device. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of a schematic configuration of an object detection system. [Figure 2] This figure shows an example of a schematic configuration of the edge device in Embodiment 1. [Figure 3] This figure shows an example of a schematic configuration of the server device in Embodiment 1. [Figure 4] This figure shows an example of a schematic configuration of the edge device in Embodiment 2. [Figure 5] This figure shows an example of a schematic configuration of the server device in Embodiment 2. [Figure 6] This figure shows an example of a schematic configuration of the edge device in Embodiment 3. [Figure 7] This figure shows an example of a schematic configuration of the server device in Embodiment 3. [Figure 8] This figure shows an example of a schematic configuration of the edge device in Embodiment 4. [Figure 9]This figure shows an example of a schematic configuration of the server device in Embodiment 4. [Modes for carrying out the invention]

[0011] Multiple embodiments for disclosure will be described with reference to the drawings. For the sake of clarity, in some embodiments, parts having the same function as those shown in the drawings used in previous descriptions will be denoted by the same reference numerals, and their descriptions may be omitted. For parts denoted by the same reference numerals, refer to the descriptions in other embodiments.

[0012] (Embodiment 1) <Outline configuration of object detection system 1> Embodiment 1 of this disclosure will be described below with reference to the drawings. The object detection system 1 shown in Figure 1 is a system that uses edge computing to detect moving objects. The object detection system 1 is an infrastructure system that detects moving objects in the vicinity of, for example, an intersection with poor visibility. Examples of moving objects to be detected include vehicles such as automobiles, motorcycles, and bicycles. Other examples of moving objects to be detected include pedestrians. As shown in Figure 1, the object detection system 1 includes an edge device 10, an object detection sensor 20, and a server device 30.

[0013] The object detection sensor 20 is a sensor for detecting objects. The object detection sensor 20 is installed with a fixed sensing range. The object detection sensor 20 is connected to the edge device 10. The connection between the object detection sensor 20 and the edge device 10 may be wired or wireless. The object detection sensor 20 outputs the sensing results to the edge device 10 sequentially. In other words, the sensing results are output sequentially from the object detection sensor 20 to the edge device 10 in a time-series manner. For example, the sensing results sensed every 100 msec may be output sequentially. Examples of objects for the object detection sensor 20 include cameras and LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), which are sensors that obtain images within a sensing range. Images obtained by a camera are captured images. Images obtained by LiDAR are point cloud images. In this embodiment, the following explanation will continue using the case where the object detection sensor 20 is a camera as an example. Note that the same processing may be performed when using point cloud images instead of captured images. Both the captured image and the point cloud image should be represented by coordinates in at least two dimensions.

[0014] The server device 30 is connected to the edge device 10 via a network NW. The server device 30 may be a collection of multiple servers. The server device 30 can also be referred to as a cloud. The server device 30 is mainly composed of a computer equipped with, for example, a processor, volatile memory, non-volatile memory, I / O, and a bus to connect them. Details of the server device 30 will be described later.

[0015] The edge device 10 is located at the end of the network NW and is connected to the server device 30 via the network NW. The configuration may also use a server as the edge device 10. The edge device 10 is primarily composed of a computer, for example, equipped with a processor, volatile memory, non-volatile memory, I / O, and a bus to connect these components. Details of the edge device 10 will be described later.

[0016] <Schematic Configuration of Edge Device 10> Next, the schematic configuration of the edge device 10 will be described with reference to FIG. 2. As shown in FIG. 2, the edge device 10 includes a sensing result acquisition unit 101, an edge-side inference unit 102, a difference detection unit 103, and a server transmission determination unit 104 as functional blocks. Note that part or all of the functions executed by the edge device 10 may be configured hardware-wise by one or more ICs or the like. Also, part or all of the functional blocks included in the edge device 10 may be realized by a combination of software execution by a processor and hardware members.

[0017] The sensing result acquisition unit 101 acquires the sensing results sequentially sensed by the object detection sensor. That is, the sensing result acquisition unit 101 sequentially acquires the sensing results along the time series. In the example of the present embodiment, it is assumed that the sensing result acquisition unit 101 acquires the captured images sequentially captured by the camera.

[0018] The edge-side inference unit 102 detects a moving object from the sensing results acquired by the sensing result acquisition unit 101 using the first inference model. The first inference model is a machine-learned inference model for detecting the moving object to be detected. As the inference model, for example, an inference model using a deep neural network (hereinafter, DNN) may be used. What kind of moving object is to be detected can be set in advance and the machine learning of the first inference model can be performed. For example, vehicles and pedestrians may be set as the moving objects to be detected. The machine learning of the first inference model may be performed using the sensing results including the moving object to be detected. The edge-side inference unit 102 detects the moving object to be detected when the moving object to be detected can be recognized from the sensing results using the first inference model. On the other hand, the edge-side inference unit 102 does not detect the moving object to be detected when the moving object to be detected cannot be recognized from the sensing results using the first inference model. In the example of the present embodiment, the moving object to be detected is detected from the captured image using the first inference model.

[0019] The difference detection unit 103 detects the presence of a moving object based on the difference between sensing results sequentially acquired by the sensing result acquisition unit 101. Since the object detection sensor 20 is installed with a fixed sensing range, the area of ​​stationary objects in the sensing results does not change between sensing results. On the other hand, the area of ​​moving objects in the sensing results does change between sensing results. Therefore, the presence of a moving object can be detected based on the difference between sensing results. This detection of the presence of a moving object based on the difference between sensing results has a lower processing load compared to detection using a machine learning inference model. In addition, detection of the presence of a moving object based on the difference between sensing results is less prone to false detections where a moving object exists but is not detected. In other words, the difference detection unit 103 can detect the presence of a moving object more easily and with greater accuracy. The difference detection unit 103 only needs to detect the presence of a moving object when a difference occurs between sensing results over a certain area or larger. This certain area or larger can be set according to the estimated size within the sensing results when the sensing results include the moving object to be detected. In this embodiment, the difference detection unit 103 detects the presence of a moving object based on the difference in pixels between captured images.

[0020] The server transmission determination unit 104 determines whether or not to transmit the sensing results acquired by the sensing result acquisition unit 101 to the server device 30. This determination will be referred to as the transmission determination below. The server transmission determination unit 104 makes the transmission determination based on the difference in whether or not a moving object is detected by the edge-side inference unit 102 and the difference detection unit 103. The server transmission determination unit 104 only needs to make the transmission determination based on the difference in whether or not a moving object is detected by the edge-side inference unit 102 and the difference detection unit 103 using the same sensing results. The difference in whether or not a moving object is detected by the edge-side inference unit 102 and the difference detection unit 103 will be referred to as the detection difference below. A detection difference occurs when the difference detection unit 103 detects the presence of a moving object, but the edge-side inference unit 102 does not. A detection difference also occurs when the difference detection unit 103 does not detect the presence of a moving object, but the edge-side inference unit 102 does.

[0021] The server transmission determination unit 104 then transmits the sensing result used for detecting the moving object by the edge-side inference unit 102 to the server device 30 if there is a detection discrepancy. If there is a discrepancy in whether or not a moving object is detected between the edge-side inference unit 102 and the difference detection unit 103, there is a possibility of false detection in the first inference model. By limiting the transmission of the sensing result used for detecting the moving object to the server device 30 to cases where there is a possibility of false detection in the first inference model, it becomes possible to further reduce the amount of data transmitted from the edge device 10 to the server device 30. In addition, this reduces the amount that must be processed on the server device 30 side, and further reduces the processing load on the server device 30. As a result, even when a moving object moving within the sensing range of the object detection sensor 20 is the target of object detection, it becomes possible to further reduce the amount of data transmitted from the edge device 10 to the server device 30 and the processing load on the server device 30. Furthermore, if there is no detection misalignment, the server transmission determination unit 104 does not need to transmit the sensing result used for detecting the moving object by the edge-side inference unit 102 to the server device 30.

[0022] The server transmission determination unit 104 preferably transmits the sensing result used by the edge-side inference unit 102 to the server device 30 when the difference detection unit 103 detects the presence of a moving object, but the edge-side inference unit 102 does not detect it. As mentioned above, the difference detection unit 103 is less prone to false detections where a moving object is present but not detected. Therefore, when the difference detection unit 103 detects the presence of a moving object, but the edge-side inference unit 102 does not detect it, there is a high possibility of a false detection in the first inference model. With the above configuration, it becomes possible to transmit the sensing result used to detect the moving object to the server device 30 when there is a high possibility of a false detection in the first inference model.

[0023] If the server transmission determination unit 104 determines that it will transmit the sensing results to the server device 30, it is preferable that the sensing results for a predetermined number of samples thereafter be handled as follows: Regardless of the determination result of whether or not to transmit to the server device 30, it is preferable that the server transmission determination unit 104 continues to transmit the sensing results to the server device 30. The predetermined number of samples can be any value that can be set arbitrarily, for example, 100 samples.

[0024] <Outline configuration of server device 30> Next, the general configuration of the server device 30 will be explained using Figure 3. As shown in Figure 3, the server device 30 includes a server-side inference unit 301, an edge learning data generation unit 302, an edge learning database (hereinafter referred to as DB) 303, an edge retraining unit 304, and an edge inference model DB 305 as functional blocks. Some or all of the functions performed by the server device 30 may be configured hardware-wise using one or more ICs, etc. Also, some or all of the functional blocks provided by the server device 30 may be realized by a combination of software execution by a processor and hardware components.

[0025] The server-side inference unit 301 acquires sensing results transmitted from the edge device 10. The server-side inference unit 301 detects moving objects using a second inference model based on the acquired sensing results. In other words, the server-side inference unit 301 detects moving objects using the second inference model if the detection result of the moving object by the edge-side inference unit 102 may be a false positive. The second inference model is a machine-learned inference model for detecting target moving objects. For example, an inference model using a DNN may be used. The moving objects targeted for detection by the second inference model are the same as those targeted by the first inference model. The second inference model should be an inference model with higher detection accuracy than the second inference model. Specifically, if both the first and second inference models are DNN-based inference models, the second inference model should have more layers than the first inference model. The machine learning of the second inference model should also be performed using sensing results that include the target moving object. The server-side inference unit 301 detects the target moving object if the second inference model can recognize the target moving object from the sensing results. On the other hand, the server-side inference unit 301 does not detect the target moving object if it cannot recognize the target moving object using the second inference model based on the sensing results. In this embodiment, the target moving object is detected using the second inference model from the captured image transmitted from the edge device 10.

[0026] The edge learning data generation unit 302 generates learning data (hereinafter referred to as edge learning data) for retraining the first inference model from the detection results of the moving object by the server-side inference unit 301. When the edge learning data generation unit 302 fails to detect the target moving object and the sensing results are sent to the server device 30, it can, for example, do the following: The edge learning data generation unit 302 should use the information of the region in which the server-side inference unit 301 was able to detect the target moving object and the type of the detected moving object as edge learning data. The region information can, for example, be the coordinates of the four corners of the region. In the retraining described later, supervised learning can be performed on the region in which the target moving object was detected, using the type of the detected moving object as the correct answer. When the edge learning data generation unit 302 detects the target moving object by the edge-side inference unit 102 and the sensing results are sent to the server device 30, it can, for example, do the following: The edge training data generation unit 302 should use the region from the sensing results where the edge-side inference unit 102 detected the target moving object but the server-side inference unit 301 did not, as the edge training data. In the retraining described later, supervised learning should be performed on this region where the target moving object was not detected, treating it as correct if it is not a moving object.

[0027] The edge learning data generation unit 302 stores the generated edge learning data in the edge learning DB 303. For example, non-volatile memory can be used as the edge learning DB 303. The edge learning DB 303 may be configured to be installed in the server device 30 or to be installed outside the server device 30. The edge learning DB 303 corresponds to the edge learning data storage unit.

[0028] The edge retraining unit 304 retrains the first inference model. The edge retraining unit 304 includes a retraining determination unit 341 and a retraining execution unit 342 as sub-functional blocks. The retraining determination unit 341 determines whether or not to retrain the first inference model. The retraining determination unit 341 should determine whether to retrain the first inference model if the edge training data in the edge training DB 303 exceeds a specified amount. On the other hand, the retraining determination unit 341 should determine whether to retrain the first inference model if the edge training data in the edge training DB 303 does not exceed a specified amount. The specified amount can be any amount that is presumed to be effective for retraining.

[0029] The retraining execution unit 342 performs retraining of the first inference model when the retraining determination unit 341 determines that retraining of the first inference model should be performed. In other words, the retraining execution unit 342 performs retraining of the first inference model based on edge training data of a specified amount or more stored in the edge training DB 303. Retraining can be performed as described above. The retraining execution unit 342 stores the retrained first inference model (hereinafter referred to as the retrained first inference model) in the edge inference model DB 305. The retraining execution unit 342 also causes the edge-side inference unit 102 to replace the first inference model used by the edge inference unit 102 with the retrained first inference model. The retraining execution unit 342 can perform this replacement by reading the retrained first inference model stored in the edge inference model DB 305 and transmitting it to the edge device 10. For example, non-volatile memory can be used as the edge inference model DB 305. The edge inference model DB305 may be configured to be installed in the server device 30, or it may be configured to be installed outside the server device 30. The edge inference model DB305 corresponds to the edge inference model storage unit.

[0030] With the above configuration, the data used for retraining the first inference model on the server device 30 can be narrowed down to data containing sensing results where there is a possibility of false detection in the first inference model. Therefore, the processing load for retraining on the server device 30 can be further reduced.

[0031] The edge retraining unit 304 preferably divides the sensing results, which are transmitted continuously for a predetermined number of samples, into training data and evaluation data in a predetermined ratio. The predetermined ratio is such that the proportion of training data is higher than the proportion of evaluation data, and can be set arbitrarily. For example, the data can be divided so that the ratio of training data to evaluation data is 4:1. The edge retraining unit 304 uses the training data to retrain the first inference model. On the other hand, the edge retraining unit 304 uses the evaluation data to evaluate the detection rate of moving objects in the retrained first inference model after the retraining. Since the sensing results for a predetermined number of samples are temporally continuous data, the inclusion of the moving objects to be detected in the data can be considered to be approximately the same. Therefore, the sensing results for a predetermined number of samples can be used not only as training data but also as evaluation data. If the detection rate does not meet the predetermined value in the evaluation using the evaluation data, the edge retraining unit 304 should do the following. The edge retraining unit 304 only needs to avoid replacing the first inference model used by the edge-side inference unit 102 with the retrained first inference model. On the other hand, the edge retraining unit 304 only needs to perform the replacement if its detection rate meets a specified value. This makes it possible to replace the first inference model with the retrained first inference model only when an improvement in detection accuracy can be expected with the retrained first inference model. As a result, it becomes possible to suppress the deterioration of detection accuracy caused by replacing the first inference model with the retrained first inference model.

[0032] (Embodiment 2) The configuration is not limited to the embodiment described above, but may also be that of the following embodiment 2. Below, an example of the configuration of embodiment 2 will be explained with reference to a diagram. The object detection system 1 of embodiment 2 is the same as the object detection system 1 of embodiment 1, except that it includes an edge device 10a and a server device 30a instead of the edge device 10 and server device 30.

[0033] <Outline configuration of edge device 10a> Next, the schematic configuration of the edge device 10a will be explained using Figure 4. As shown in Figure 4, the edge device 10a includes a sensing result acquisition unit 101, an edge-side inference unit 102a, a difference detection unit 103, and a server transmission determination unit 104 as functional blocks. The edge device 10a is the same as the edge device 10 of Embodiment 1, except that it includes an edge-side inference unit 102a instead of an edge-side inference unit 102.

[0034] The edge-side inference unit 102a is the same as the edge-side inference unit 102 of Embodiment 1, except for a few differences. These differences will be explained below. The edge-side inference unit 102a requires the use of an inference model using a DNN as the first inference model.

[0035] <Outline configuration of server device 30a> Next, the general configuration of the server device 30a will be explained using Figure 5. As shown in Figure 5, the server device 30a includes a server-side inference unit 301a, an edge learning data generation unit 302, an edge learning DB 303, an edge retraining unit 304a, and an edge inference model DB 305 as functional blocks. The server device 30a includes a server-side inference unit 301a instead of a server-side inference unit 301. The server device 30a includes an edge retraining unit 304a instead of an edge retraining unit 304. Except for these points, the server device 30a is the same as the server device 30 of Embodiment 1.

[0036] The server-side inference unit 301a is the same as the server-side inference unit 301 of Embodiment 1, except for a few differences. These differences will be explained below. The server-side inference unit 301a is required to use an inference model using a DNN as the second inference model.

[0037] The edge relearning unit 304a includes a relearning determination unit 341 and a relearning execution unit 342a as sub-functional blocks. The edge relearning unit 304a is the same as the edge relearning unit 304 of Embodiment 1, except that it includes a relearning execution unit 342a instead of a relearning execution unit 342. The relearning execution unit 342a is the same as the relearning execution unit 342 of Embodiment 1, except that some processing is different. These differences will be explained below.

[0038] The retraining execution unit 342a obtains the weights of the first inference model currently used by the edge-side inference unit 102a as initial weights for retraining. These weights are parameters that indicate how strongly the neurons between layers of the DNN are connected. The retraining execution unit 342a performs retraining with a setting that prioritizes and significantly changes the weights of layers closer to the output among the multiple layers of the DNN. For example, the range of weight changes can be increased as the layer gets closer to the output. Alternatively, the weights can be changed only for a certain number of layers that are closer to the output among the multiple layers of the DNN. The certain number of layers can be 1 to several layers and can be set to an arbitrary value.

[0039] Rebuilding the entire DNN each time it is retrained may actually decrease detection accuracy. In contrast, the above configuration allows for the reuse of much of the pre-trained DNN, making it easier to improve detection accuracy even with a small amount of data for retraining.

[0040] (Embodiment 3) The configuration is not limited to the embodiment described above, but may also be that of the following embodiment 3. Below, an example of the configuration of embodiment 3 will be explained with reference to a diagram. The object detection system 1 of embodiment 3 is the same as the object detection system 1 of embodiment 1, except that it includes an edge device 10b and a server device 30b instead of the edge device 10 and server device 30.

[0041] <Outline configuration of edge device 10b> Next, the schematic configuration of the edge device 10b will be explained using Figure 6. As shown in Figure 6, the edge device 10b includes a sensing result acquisition unit 101, an edge-side inference unit 102b, a difference detection unit 103, and a server transmission determination unit 104 as functional blocks. The edge device 10a is the same as the edge device 10 of Embodiment 1, except that it includes an edge-side inference unit 102b instead of an edge-side inference unit 102.

[0042] The edge-side inference unit 102b is the same as the edge-side inference unit 102 of Embodiment 1, except for a few differences. These differences will be explained below. The edge-side inference unit 102b transmits the number of detected objects, which is the number of moving objects detected, to the server device 30b at predetermined intervals. The predetermined interval is an arbitrarily set value, for example, one minute. For objects that are identified as the same moving object using, for example, optical flow, even if they are included in multiple sensing results, they should be counted as one. Alternatively, the number of detected objects may be the number of moving objects detected from the sensing results obtained within the predetermined interval, without distinguishing whether they are the same moving object or not.

[0043] <Outline configuration of server device 30b> Next, the schematic configuration of the server device 30b will be explained using Figure 7. As shown in Figure 7, the server device 30b includes a server-side inference unit 301, an edge learning data generation unit 302, an edge learning DB 303, an edge retraining unit 304b, an edge inference model DB 305, a detection count statistics processing unit 306, and a detection count statistics DB 307 as functional blocks. The server device 30b includes an edge retraining unit 304b instead of an edge retraining unit 304. The server device 30b includes a detection count statistics processing unit 306 and a detection count statistics DB 307. Except for these points, the server device 30b is the same as the server device 30 of Embodiment 1.

[0044] The detection count statistics processing unit 306 calculates the total number of detected objects transmitted from the edge-side inference unit 102b at regular intervals (hereinafter referred to as the total number of detections per hour). The regular interval referred to here can be any period longer than the predetermined interval mentioned above, and is a value that can be set arbitrarily. For example, the regular interval can be set to 15 minutes. If the predetermined interval is 1 minute and the regular interval is 15 minutes, the detection count statistics processing unit 306 will calculate the total number of detected objects per hour in 15-minute increments. The detection count statistics processing unit 306 stores the calculated total number of detections per hour in the detection count statistics DB 307. For example, non-volatile memory can be used as the detection count statistics DB 307. The detection count statistics DB 307 may be configured to be provided in the server device 30b or to be provided outside the server device 30b. The detection count statistics DB 307 corresponds to the detection count statistics storage unit.

[0045] The edge relearning unit 304b includes a relearning determination unit 341 and a relearning execution unit 342b as sub-functional blocks. The edge relearning unit 304b is the same as the edge relearning unit 304 of Embodiment 1, except that it includes a relearning execution unit 342b instead of a relearning execution unit 342. The relearning execution unit 342b is the same as the relearning execution unit 342 of Embodiment 1, except that some processing differs. These differences will be explained below.

[0046] The retraining execution unit 342b identifies the time period during the day when the number of detected moving objects is lowest, based on the total number of detections per hour stored in the detection statistics DB 307. The retraining execution unit 342b only needs to identify the time period with the lowest total number of detections per hour among the total number of detections per hour for a given period of time. At the identified time period, the retraining execution unit 342b replaces the first inference model used by the edge-side inference unit 102b with the retrained first inference model.

[0047] When replacing the first inference model with a retrained first inference model, it is necessary to stop the function of detecting moving objects on the edge device 10. With the above configuration, it becomes possible to replace the first inference model with a retrained first inference model at a timing that has less impact on stopping the function of detecting moving objects on the edge device 10. If the configuration of Embodiment 3 is not applied, the function of detecting moving objects on the edge device 10 can be stopped at a desired time and the first inference model can be replaced with a retrained first inference model.

[0048] (Embodiment 4) The configuration is not limited to the embodiment described above, but may also be that of the following embodiment 4. Below, an example of the configuration of embodiment 4 will be explained with reference to a diagram. The object detection system 1 of embodiment 4 is the same as the object detection system 1 of embodiment 1, except that it includes an edge device 10c and a server device 30c instead of the edge device 10 and server device 30.

[0049] <Outline configuration of edge device 10c> Next, the schematic configuration of the edge device 10c will be explained using Figure 8. As shown in Figure 8, the edge device 10c includes a sensing result acquisition unit 101, an edge-side inference unit 102c, a difference detection unit 103, and a server transmission determination unit 104 as functional blocks. The edge device 10c is the same as the edge device 10 of Embodiment 1, except that it includes an edge-side inference unit 102c instead of an edge-side inference unit 102.

[0050] The edge-side inference unit 102c is the same as the edge-side inference unit 102 of Embodiment 1, except for a few differences. These differences will be explained below. When the edge-side inference unit 102c receives region information from the variable background identification unit 313 (described later), it is preferable to exclude the region indicated by the region information from the target region for detecting moving objects. Details will be described later.

[0051] <Outline configuration of server device 30c> Next, the general configuration of the server device 30c will be explained using Figure 9. As shown in Figure 9, the server device 30c includes the following functional blocks: server-side inference unit 301, edge learning data generation unit 302, edge learning DB 303, edge retraining unit 304, edge inference model DB 305, undetected unit 308, server learning data generation unit 309, server learning DB 310, server retraining unit 311, server inference model DB 312, and variable background identification unit 313. The server device 30c is the same as the server device 30 of Embodiment 1, except that it includes the undetected unit 308, server learning data generation unit 309, server learning DB 310, server retraining unit 311, server inference model DB 312, and variable background identification unit 313.

[0052] The undetected area determination unit 308 determines an undetected area in which the target moving object is not detected by the server-side inference unit 301. The undetected area is an area within the sensing result. For example, the undetected area may be represented by the coordinates of the four corners. The undetected area is an area in which the presence of the target moving object is detected by the difference detection unit 103, but the target moving object is not detected by both the edge-side inference unit 102c and the server-side inference unit 301. In other words, it is an area in the sensing result in which a false detection may have occurred by the server-side inference unit 301. The undetected area determination unit 308 can obtain information from the edge device 10c regarding whether or not the difference detection unit 103 detected the presence of the target moving object, and whether or not the edge-side inference unit 102c detected the target moving object.

[0053] The server training data generation unit 309 generates training data (hereinafter referred to as server training data) for retraining the second inference model for the undetected regions determined by the undetection determination unit 308. The edge training data generation unit 302 should use the regions in the sensing results where the presence of a moving object was detected by the difference detection unit 103 as server training data. The type of moving object that should have been detected can be accepted, for example, by manual input. This input can be made by an operator who has visually confirmed the captured image or point cloud image via the operation input unit. The region containing the moving object in the sensing results can also be accepted, for example, by manual input. The region information can be, for example, the coordinates of the four corners of the region. In the retraining described later, supervised learning should be performed on the region containing the target moving object in the sensing results, using the type of moving object to be detected as the correct answer.

[0054] The server learning data generation unit 309 stores the generated server learning data in the server learning DB 310. For example, non-volatile memory can be used as the server learning DB 310. The server learning DB 310 may be configured to be located within the server device 30c, or it may be located outside the server device 30c. The server learning DB 310 corresponds to the server learning data storage unit.

[0055] The server relearning unit 311 retrains the second inference model. The server relearning unit 311 retrains the second inference model based on the server training data stored in the server training DB 310. Similar to the retraining of the first inference model, the retraining of the second inference model may also be performed when a specified amount of server training data has been accumulated. The server relearning unit 311 stores the retrained second inference model (hereinafter referred to as the retrained second inference model) in the server inference model DB 312. The server relearning unit 311 also causes the server-side inference unit 301 to replace the second inference model it uses with the retrained second inference model. The server relearning unit 311 can perform this replacement by reading the retrained second inference model stored in the server inference model DB 312 and sending it to the server-side inference unit 301. For example, non-volatile memory can be used as the server inference model DB 312. The server inference model DB312 may be configured to be installed in the server device 30c, or it may be configured to be installed outside the server device 30c. The server inference model DB312 corresponds to the server inference model storage unit.

[0056] With the above configuration, it becomes possible to retrain the second inference model on the server device 30c to improve detection accuracy.

[0057] The variable background identification unit 313 identifies the presence or absence of false detection areas from the sensing results when the undetection determination unit 308 determines that an undetected area exists. False detection areas are areas that are not the moving object being detected, but which change over time. Examples of false detection areas include electronic display boards with changing displays and the branches of street trees swaying in the wind. The variable background identification unit 313 can identify the presence or absence of false detection areas from the sensing results group, which includes the sensing results in which the undetected area was determined by the undetection determination unit 308, or the sensing results before and after that. For example, the variable background identification unit 313 can be configured to identify the presence or absence of false detection areas in response to input indicating the presence or absence of false detection areas, which is manually entered. This input can be made by an operator who has visually confirmed the captured image or point cloud image via an operation input unit. Alternatively, the variable background identification unit 313 may identify the presence or absence of false detection areas based on the difference between sensing result groups. In this case, a learning model that has been trained to recognize the differences between sensing result groups that are characteristic of false detection areas can be used. When the variable background identification unit 313 identifies a false detection area, it transmits area information indicating that false detection area to the edge-side inference unit 102c. The area information may be, for example, the coordinates of the four corners of the area.

[0058] As mentioned above, the edge-side inference unit 102c, upon receiving the region information, excludes the region indicated by the region information from the target region for detecting moving objects. As a result, the edge-side inference unit 102c will no longer detect erroneously detected regions as moving objects. Therefore, even if there are electronic billboards with changing displays, branches of street trees swaying in the wind, etc., within a fixed area of ​​the sensing range of the object detection sensor 20, these will not be erroneously detected as moving objects.

[0059] This disclosure is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of this disclosure. Furthermore, the control unit and method described in this disclosure may be implemented by a dedicated computer comprising a processor programmed to execute one or more functions embodied by a computer program. Alternatively, the apparatus and method described in this disclosure may be implemented by a dedicated hardware logic circuit. Alternatively, the apparatus and method described in this disclosure may be implemented by one or more dedicated computers comprising a combination of a processor that executes a computer program and one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by the computer on a computer-readable non-transitional tangible recording medium. [Explanation of symbols]

[0060] 1 Object detection system, 10, 10a, 10b, 10c Edge device, 20 Object detection sensor (sensor), 30, 30a, 30b, 30c Server device, 101 Sensing result acquisition unit, 102, 102a, 102b Edge-side inference unit, 103 Difference detection unit, 104 Server transmission determination unit, 301 Server-side inference unit, 302 Edge learning data generation unit, 303 Edge learning DB (Edge learning data storage unit), 304, 304a, 304b Edge retraining unit, 305 Edge inference model DB (Edge inference model storage unit), 306 Detection count statistics processing unit, 307 Detection count statistics DB (Detection count statistics storage unit), 308 Non-detection determination unit, 309 Server learning data generation unit, 310 Server learning DB (Server learning data storage unit), 311 Server retraining unit, 312 Server inference model DB (server inference model storage unit), 313 Variable background identification unit

Claims

1. An edge device located at the end of a network and connected to server devices (30, 30a, 30b, 30c) via the network, It is connected to an object detection sensor (20) that is installed with a fixed sensing range, A sensing result acquisition unit (101) acquires sensing results that are sequentially sensed by the aforementioned sensor, An edge-side inference unit (102, 102a, 102b, 102c) detects a moving object from the sensing results acquired by the sensing result acquisition unit using a first inference model, which is a machine learning-prepared inference model for detecting a moving object to be detected, A difference detection unit (103) detects the presence of a moving object based on the difference between the sensing results acquired sequentially by the sensing result acquisition unit, The system includes a server transmission determination unit (104) that determines whether or not to transmit the sensing result acquired by the sensing result acquisition unit to the server device that detects the moving object using a second inference model, which is a machine learning-trained inference model for detecting the moving object and has a higher detection accuracy than the first inference model, based on the discrepancy between the presence or absence of detection of the moving object between the edge-side inference unit and the difference detection unit. The server transmission determination unit is an edge device that, when there is a discrepancy between the detection of the moving object by the edge-side inference unit and the difference detection unit, transmits the sensing result used for detecting the moving object by the edge-side inference unit to the server device.

2. An edge device according to claim 1, The server transmission determination unit transmits the sensing result used by the edge-side inference unit to the server device when the difference detection unit detects the presence of the moving object, but the edge-side inference unit does not detect the moving object.

3. An object detection system including server devices (30, 30a, 30b, 30c) and edge devices (10, 10a, 10b, 10c) provided at the end of the network and connected to the server devices via the network, The edge device is It is connected to an object detection sensor (20) that is installed with a fixed sensing range, A sensing result acquisition unit (101) acquires sensing results that are sequentially sensed by the aforementioned sensor, An edge-side inference unit (102, 102a, 102b, 102c) detects a moving object from the sensing results acquired by the sensing result acquisition unit using a first inference model, which is a machine learning-prepared inference model for detecting a moving object to be detected, A difference detection unit (103) detects the presence of a moving object based on the difference between the sensing results acquired sequentially by the sensing result acquisition unit, The system includes a server transmission determination unit (104) that determines whether or not to transmit the sensing result acquired by the sensing result acquisition unit to the server device, based on the discrepancy between the detection status of the moving object between the edge-side inference unit and the difference detection unit. The server transmission determination unit, when there is a discrepancy between the detection of the moving object by the edge-side inference unit and the difference detection unit, transmits the sensing result used for detecting the moving object by the edge-side inference unit to the server device. The server device is An object detection system comprising a server-side inference unit (301, 301a) that detects the moving object from the sensing results transmitted from the edge device using a second inference model, which is a machine-learned inference model for detecting the moving object and has higher detection accuracy than the first inference model.

4. The object detection system according to claim 3, The server device is An edge learning data generation unit (302) generates edge learning data, which is learning data for retraining the first inference model, from the detection results of the moving object in the server-side inference unit, and stores it in the edge learning data storage unit (303), An object detection system comprising edge retraining units (304, 304a, 304b) that retrain the first inference model based on a specified amount or more of the edge training data stored in the edge training data storage unit, store the retrained first inference model, which is the retrained first inference model, in the edge inference model storage unit (305), and replace the first inference model used in the edge-side inference unit with the retrained first inference model.

5. The object detection system according to claim 4, The first inference model used in the edge-side inference unit (102a) is an inference model using a deep neural network, The edge retraining unit (304a) acquires the weights of the first inference model currently used in the edge-side inference unit as initial weights for retraining, and performs retraining with a setting that prioritizes and significantly changes the weights of the layers closer to the output among the multiple layers of the deep neural network.

6. The object detection system according to claim 4, If the server transmission determination unit (104) determines that the sensing results should be transmitted to the server device, it will continue to transmit the sensing results to the server device for a predetermined number of subsequent samples, regardless of the determination result regarding whether or not to transmit them to the server device. The edge retraining unit (304) divides the sensing results, which are continuously transmitted for a predetermined number of samples, into training data and evaluation data at a predetermined ratio, the training data is used to retrain the first inference model, and the evaluation data is used to evaluate the detection rate of the moving object in the retrained first inference model after the retraining, and in the evaluation using the evaluation data, if the detection rate does not meet a predetermined value, the first inference model used in the edge-side inference unit is not replaced with the retrained first inference model.

7. The object detection system according to claim 4, The edge-side inference unit (102b) transmits the number of detected objects, which is the number of moving objects detected, to the server device at predetermined intervals. The server device (30b) is The system includes a detection count statistics processing unit (306) that calculates the total number of detected objects transmitted from the edge-side inference unit for each set of time periods longer than the predetermined interval, and stores the calculated total number of detected objects for each set of time in a detection count statistics storage unit (307). The edge retraining unit (304b) is an object detection system that, based on the total number of detections by time stored in the detection count statistics storage unit, identifies the time period during the day when the number of detected objects of the moving object is lowest, and replaces the first inference model used in the edge-side inference unit with the retrained first inference model during that identified time period.

8. The object detection system according to claim 4, The server device (30c) is The difference detection unit has detected the presence of the moving object, but the server-side inference unit (301) has not detected it. The undetected determination unit (308) determines the undetected region, which is the region of the sensing result, A server learning data generation unit (309) generates server learning data, which is learning data for retraining the second inference model, for the undetected region determined by the undetected region determined by the undetected region, and stores it in a server learning data storage unit (310), An object detection system comprising: a server retraining unit (311) that retrains the second inference model based on the server training data stored in the server training data storage unit, stores the retrained second inference model in the server inference model storage unit (312), and replaces the second inference model used in the server-side inference unit with the retrained second inference model.

9. The object detection system according to claim 8, The server device is When the undetection determination unit determines the undetected area, it identifies from the sensing results whether there is a false detection area, which is an area that is not the moving object to be detected and changes over time, and if it identifies that there is a false detection area, it transmits area information indicating the false detection area to the edge-side inference unit. The edge-side inference unit, if it has received the region information from the variable background identification unit, excludes the region indicated by the region information from the target region for detecting the moving object.

Images