Servers and Information Processing Devices

Abstract

This enables automatic detection of changes in the target of recognition due to environmental changes. [Solution] The server (20) includes a storage unit (21) that stores an AI model for detecting recognition targets from captured images, and a control unit (22) that outputs information indicating that the recognition target has changed when the point at which a recognition target that could be recognized in the past can no longer be recognized is above a first threshold.

Description

Technical Field

[0001] The present disclosure relates to a server and an information processing apparatus.

Background Art

[0002] Conventionally, a technique for selecting learning data that contributes to improving the inference result of a model with low load is known. For example, in Patent Document 1, environmental information of a storage area that affects the recognition difficulty of an article included in an image obtained by imaging a storage area for storing an article, fineness information regarding the resolution of the image, and / or inference information based on the output of a model for recognizing the article are used as meta-information, and an information providing unit that provides the image with the meta-information, and an image selection unit that selects an image stored in a storage unit in which a plurality of images with the meta-information are stored according to the meta-information are disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The technique described in Patent Document 1 selects learning data according to the recognition difficulty of an article, but it is premised on the fact that the article can be recognized, and the case where an article that could be recognized cannot be recognized is not considered. For example, if the article to be stored is in a box and the design of the box changes, the article cannot be recognized. When such a design change or the like occurs, the operator has to visually confirm that the design has changed, collect image data of the new design to create learning data, and update (re-learn) the model.

[0005] In view of such circumstances, an object of the present disclosure is to enable automatic detection of changes in a recognition target accompanying environmental changes. [Means for solving the problem]

[0006] A server according to one embodiment of this disclosure includes a storage unit that stores an AI model for detecting a recognition target from a captured image, and a control unit that outputs information indicating that the recognition target has changed when the point at which a recognition target that could be recognized in the past can no longer be recognized satisfies the condition that the point is above a first threshold.

[0007] An information processing apparatus according to one embodiment of the present disclosure includes: a storage unit that stores an AI model received from a server; a control unit that inputs a captured image taken by an in-vehicle camera to the AI ​​model, performs recognition of a recognition target, and determines whether the shooting conditions of the captured image are the same as those used when the recognition target was previously recognized if the recognition target could not be recognized from the captured image; and a communication unit that, if the recognition target could not be recognized from the captured image and the shooting conditions are the same as those used when the recognition target was previously recognized, transmits the shooting conditions, the captured image in which the recognition target could not be recognized, and information notifying the server that the recognition target can no longer be recognized under the shooting conditions under which it was previously recognized. [Effects of the Invention]

[0008] According to one embodiment of this disclosure, it becomes possible to automatically detect changes in the recognition target in response to environmental changes. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example configuration of an image recognition system according to one embodiment. [Figure 2] This flowchart shows an example of a vehicle processing procedure according to one embodiment. [Figure 3] This flowchart shows an example of the server processing procedure according to one embodiment. [Modes for carrying out the invention]

[0010] <Configuration of the image recognition system> Referring to Figure 1, the configuration of an image recognition system according to one embodiment will be described. The image recognition system 1 comprises a plurality of vehicles 10 and a server 20. For convenience, only one vehicle is shown in Figure 1. The plurality of vehicles 10 and the server 20 are connected to each other so as to be able to communicate via a network 30 including the Internet. The server 20 may be cloud-based or on-premise. The vehicle 10 comprises an information processing device 11, an in-vehicle camera 12, and a sensor 13.

[0011] The on-board camera 12 is a camera mounted on the vehicle 10, and multiple cameras may be mounted. The on-board camera 12 is connected to the information processing device 11. The on-board camera 12 may be integrated with the information processing device 11. The on-board camera 12 has an image sensor and captures images of the area around the vehicle 10 while the vehicle 10 is in motion.

[0012] Sensor 13 includes a satellite positioning sensor that receives satellite signals from GNSS (Global Navigation Satellite System) such as GPS (Global Positioning System) to measure the position, orientation, direction, and current time of the vehicle 10.

[0013] The information processing device 11 is a device mounted on the vehicle 10 and communicates wirelessly with the server 20. The information processing device 11 comprises an input unit 111, a storage unit 112, a control unit 113, an output unit 114, and a communication unit 115.

[0014] The input unit 111 includes at least one input interface. The input interface may be, for example, a physical key, a capacitive key, a pointing device, a touchscreen integrated with a display, or a microphone. The input unit 111 also includes an interface with the in-vehicle camera 12 and receives images captured by the in-vehicle camera 12 (hereinafter referred to as "captured images").

[0015] The storage unit 112 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, RAM, ROM, or flash memory. The storage unit 112 stores the shooting conditions and the captured images. Further, the storage unit 112 stores an AI (Artificial Intelligence) model that detects a recognition target from the captured images. The AI model may be a machine learning model or a deep learning model.

[0016] The output unit 114 includes at least one output interface. The output interface is, for example, a display that outputs information as video or a speaker that outputs information as sound. The display is, for example, an LCD or an organic EL display.

[0017] The communication unit 115 includes a wireless communication interface for performing wireless communication with the server 20. The communication unit 115 receives an AI model from the server 20. Further, the communication unit 115 transmits the shooting conditions and the captured images to the server 20.

[0018] The control unit 113 includes a processor (a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing), a programmable circuit (e.g., FPGA), a dedicated circuit (e.g., ASIC), or any combination thereof. The control unit 113 executes processes related to the operation of the information processing apparatus 11 while controlling each part of the information processing apparatus 11.

[0019] The server 20 is a server that performs wireless communication with the information processing apparatus 11, and includes a storage unit 21, a control unit 22, and a communication unit 23.

[0020] The storage unit 21 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The storage unit 21 stores a machine learning model that detects a recognition target from the captured images. Further, the storage unit 21 stores the shooting conditions and the captured images.

[0021] The control unit 22 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. While controlling each part of the server 20, the control unit 22 executes processes related to the operation of the server 20.

[0022] The communication unit 23 includes a wireless communication interface for performing wireless communication with the information processing device 11. The communication unit 23 transmits the AI model to the information processing device 11. Also, the communication unit 23 receives the shooting conditions and the shot image from the information processing device 11.

[0023] In the present embodiment, as an assumed case, "the recognition target is the signboard of a gas station, and the image recognition system 1 detects the gasoline price from an image in which the signboard of the gas station is shot, and provides a map showing the location of the gas station and the gasoline price." is adopted for the explanation. In order to realize this service, the server 20 stores an AI model trained with an image including the signboard of the gas station as teacher data, and updates the AI model according to changes in the environment as described later.

[0024] The control unit 113 of the information processing device 11 inputs the shot image into the AI model and detects the signboard of the gas station. Then, the control unit 113 recognizes a three-digit numerical value as the gasoline price from the shot image including the signboard of the gas station using an existing recognition technology such as OCR (Optical Character Recognition / Reader). In order to extract a three-digit numerical value from the image in which the signboard of the gas station appears, the possibility that a numerical value other than the price of the gas station is misrecognized is low. However, the number of digits of the numerical value to be detected may be changed according to the market price situation (price range) of the gasoline price or the market price situation of the area where image recognition is performed. The control unit 22 of the server 20 maps the gasoline price recognized by the information processing device 11 on the map together with the image or icon of the signboard of the gas station and stores it in the storage unit 21. Note that the control unit 22 of the server 20 may detect the gasoline price from the shot image received from the vehicle 10.

[0025] <Example of operation> Next, an example of the processing procedure for vehicle 10 will be explained with reference to Figure 2. Figure 2 is a flowchart showing the operation of the information processing device 11. The procedure shown in Figure 2 is executed by the information processing device 11 at the timing of power-up for vehicle 10. Specifically, the control unit 113 controls the input unit 111, storage unit 112, output unit 114, and communication unit 115 to realize each process.

[0026] In step S11, the information processing device 11 receives (downloads) the latest AI model from the server 20 and stores it in the storage unit 112. For example, the information processing device 11 sends a request signal to the server 20 requesting the transmission of the latest AI model (the AI ​​model that recognizes the object to be recognized), receives the AI ​​model transmitted from the server 20 in response to the request signal, and stores it in the storage unit 112. Note that it is not necessary to receive (acquire) the AI ​​model every time the vehicle 10 starts up; the latest AI model can be acquired at an appropriate time, such as every week, so as not to interfere with the operation of the AI ​​model.

[0027] In step S12, the information processing device 11 acquires images from the in-vehicle camera 12. At the time of step S12, the information processing device 11 has set the in-vehicle camera 12 to an operational state that captures images of the outside of the vehicle while the vehicle 10 is in motion (in-vehicle camera 12 activated state).

[0028] In step S12, the information processing device 11 acquires (identifies) the shooting conditions at the time the in-vehicle camera 12 takes a picture and stores them in the storage unit 112. The shooting conditions include information acquired from the sensor 13, including the position of the vehicle 10 (shooting position) and the orientation of the vehicle 10 (shooting direction). Also, depending on the weather at the time of shooting, the object to be recognized may or may not be recognized. Therefore, the shooting conditions may include information indicating the weather. Information indicating the weather may be acquired from an external device such as a weather data server, or it may be recognized from the captured image. Also, depending on the time of shooting (brightness and sunlight), the object to be recognized may or may not be recognized. Therefore, the shooting conditions may include information indicating the current time acquired from the sensor 13. Also, the shooting conditions may include information indicating the direction of travel of the vehicle 10 acquired from the sensor 13.

[0029] In step S13, the information processing device 11 inputs the captured image into the AI ​​model and performs recognition of the object to be recognized.

[0030] In step S14, the information processing device 11 determines whether or not it was able to recognize the object to be recognized from the captured image. If it was recognized, the process proceeds to step S15; otherwise, the process proceeds to step S16.

[0031] In step S15, the information processing device 11 transmits (uploads) the shooting conditions and the captured image in which the recognition target was recognized (hereinafter referred to as the "recognition OK image") to the server 20. The communication unit 115 may transmit only the shooting location information from the shooting conditions to the server 20. The information processing device 11 also stores the shooting conditions in the storage unit 112.

[0032] In step S16, the information processing device 11 compares the current shooting conditions with the shooting conditions used when the object to be recognized was previously recognized, based on the past shooting conditions stored in the memory unit 112. If the two shooting conditions match, the process proceeds to step S17. If the two shooting conditions do not match, the process proceeds to step S18. For example, if there are no images taken under the same shooting conditions in the past, or if the object to be recognized was not recognized even when shooting under the same conditions in the past, the process proceeds to step S18.

[0033] In step S17, the information processing device 11 sends (uploads) the shooting conditions, the captured image in which the object to be recognized could not be recognized (hereinafter referred to as the "recognition NG image"), and information notifying the server 20 that the object to be recognized can no longer be recognized using the shooting conditions under which it was previously recognized (hereinafter referred to as the "recognition NG information"). The information processing device 11 may also send only the shooting location information from the shooting conditions to the server 20.

[0034] In step S18, the information processing device 11 terminates the process if the vehicle 10 is parked, and returns to step S12 if the vehicle 10 is running, and repeats the above process. For example, parking of the vehicle 10 is detected when the engine of the vehicle 10 is turned off.

[0035] Next, an example of the server 20's processing procedure will be explained with reference to Figure 3. Figure 3 is a flowchart showing the processing performed by the server 20. The processing shown in Figure 3 is performed at appropriate intervals for AI model qualification and updating, for example, at 1-day intervals. In other words, if the situation changes, such as when a new recognition target is created or when the characteristics of the recognition target change, the AI ​​model's recognition performance deteriorates. Therefore, this processing is performed at a time when a change in circumstances occurs that makes the deterioration of the AI ​​model's recognition performance unacceptable. Specifically, the control unit 22 controls the storage unit 21 and the communication unit 23 to implement each of these processes.

[0036] In step S21, the server 20 reads from the storage unit 21 the shooting conditions, recognized OK images, recognized NG images, and recognized NG information that were received from each of the multiple information processing devices 11 and stored in the storage unit 21. More specifically, it reads from the storage unit 21 the data received from the multiple information processing devices 11 since the previous AI model retraining process. The server 20 also constantly receives this data transmitted from each information processing device 11 and stores it in the storage unit 21. The control unit 22 receives information indicating the price of gasoline from the information processing device 11 or detects the price of gasoline from the recognized OK images, identifies the location of the gas station from the shooting position and shooting direction, and creates a map that maps the image or icon of the gas station sign, the price of gasoline, and other data.

[0037] In step S22, the server 20 determines, based on the data read from the storage unit 21 in step S21, whether the condition is met that the recognition target, which could previously be recognized under the same shooting conditions, can no longer be recognized at a number of locations (multiple locations) exceeding the first threshold Th1. In other words, it determines whether the inability to recognize the recognition target has occurred in many locations. Specifically, the control unit 22 counts the number of locations at gas stations where the sign can no longer be recognized, based on the shooting locations of the "recognition NG images" in the data read from the storage unit 21, and determines whether this count exceeds the first threshold Th1. The first threshold Th1 can be set to an appropriate value based on experiments (simulations), etc. If the condition is met, the process proceeds to step S25; otherwise, the process proceeds to step S23.

[0038] In step S23, the server 20 determines from the data read from the storage unit 21 in step S21 whether the condition is met that at locations below the second threshold Th2 (Th2 ≤ Th1), the recognition target that could previously be recognized under the same shooting conditions is no longer recognized. If this condition is met (i.e., the recognition target can no longer be recognized at only some (a small number) locations), the process proceeds to step S24; otherwise, the process returns to step S21. The processing in steps S22 and S23 is modified based on the technical idea that if the recognition target cannot be recognized at many locations, it is highly likely that the AI ​​model needs to be modified due to a change in the design of the recognition target (i.e., the AI ​​model's performance has deteriorated), and if the recognition target cannot be recognized at only a few locations, it is highly likely that there has been a change in the presence or absence of the recognition target.

[0039] In step S24, the server 20 deletes the mapping data (data mapped in step S21) for that location from the map. In other words, at a location where the object to be recognized can no longer be recognized, the server assumes that the store or other facility (gas station) related to the object to be recognized has closed down or changed, and deletes the store or other facility related to the object to be recognized at that location from the map.

[0040] In step S25, the server 20 determines that the object to be recognized has changed at the point where it can no longer be recognized. The control unit 22 then notifies the operator (administrator of the server 20) by outputting information indicating that the object to be recognized has changed, along with the image of the object that failed to be recognized. A change in the object to be recognized could be, for example, a change in the logo (design or text) of the object to be recognized. Upon receiving this notification, the operator notifies the person updating the AI ​​model of the situation. The operator and the person updating the AI ​​model may be the same person. Furthermore, the AI ​​model can be updated automatically or manually.

[0041] In step S26, the server 20 collects the recognition NG images received from the information processing device 11. The recognition NG images are images that include the recognition target after the change. The server 20 then uses the collected recognition NG images as training data. That is, the control unit 22 creates training data by annotating the collected recognition NG images (i.e., assigning correct labels to the images). Alternatively, the operator may identify the new recognition target and create the training data, or an annotation service of the cloud platform may be used.

[0042] In step S27, the server 20 retrains (updates) the AI ​​model using the training data created in step S26 and training data previously used for training (learning).

[0043] In step S28, the server 20 sends the updated new AI model to the information processing device 11. The information processing device 11 receives this new AI model, stores it in the memory unit 112, and uses it for recognition processing of the recognition target. Through the above process, the server 20 can automatically detect changes in the recognition target. In addition, the server 20 can update its AI model, which has become outdated over time, by retraining it with learning data using the image of the recognition target after the change, so that it can recognize the AI ​​model after the change.

[0044] Furthermore, it is also possible to use a computer capable of executing programs to function as the information processing device 11 and server 20 described above. The program can be stored on a non-temporary computer-readable medium. [Explanation of symbols]

[0045] 1 image recognition system, 10 vehicles, 20 servers, 30 networks, 11 Information processing device, 12 In-vehicle camera, 13 Sensor, 21,112 Storage unit, 22,113 Control unit, 23,115 Communication unit, 111 Input unit, 114 Output unit

Claims

1. A memory unit that stores an AI model for detecting recognition targets from captured images, A control unit that outputs information indicating that the recognized object has changed when the point at which the previously recognized object can no longer be recognized is above a first threshold, A server equipped with the following features.

2. The server according to claim 1, wherein the control unit, when the above conditions are met, collects an image containing the changed recognition target as training data and updates the AI ​​model.

3. A map is created that maps the location of the object to be recognized. The server according to claim 1 or 2, wherein the control unit deletes mapping data at a location from the map if the location where a previously recognized object can no longer be recognized satisfies the condition that the location is below a second threshold, and the second threshold is below the first threshold.

4. A memory unit that stores the AI ​​model received from the server, The AI ​​model receives images captured by an in-vehicle camera as input, performs recognition of the object to be recognized, and, if the object to be recognized cannot be recognized from the captured image, determines whether the shooting conditions of the captured image are the same as when the object to be recognized was previously recognized. A communication unit transmits to the server the shooting conditions, the shooting conditions, the shooting image in which the target could not be recognized, and information notifying that the target could no longer be recognized under the shooting conditions under which it could have been recognized in the past, if the target cannot be recognized from the captured image and the shooting conditions are the same as when the target could have been recognized in the past. An information processing device equipped with the following features.

5. The information processing apparatus according to claim 4, wherein the communication unit transmits to the server the shooting conditions of the captured image and the captured image in which the recognition target was recognized when the recognition target can be recognized from the captured image.

6. The information processing device according to claim 5, wherein the shooting conditions include information such as the position of the vehicle on which the in-vehicle camera is mounted and the orientation of the vehicle.

Images