Information processing system and program

Abstract

To realize a highly versatile method capable of executing various tasks even when a memory capacity is small.SOLUTION: An information processing system includes: a processor that executes a specific process for estimating a state of a monitoring target based on a captured image of the monitoring target; and a memory that stores instructions and data used for executing the specific process. As a process of individually executing division processes in which the specific process is divided into a plurality of process units, the processor releases a use area of the memory used in a previous division process, and performs a sequential process of executing the current division process using a free area of the memory after the release.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system and a program. [Background technology]

[0002] Techniques have been proposed to reduce memory consumption when executing tasks. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-123074 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-described conventional techniques do not necessarily realize a highly versatile method that can execute various tasks even when memory capacity is small.

[0005] An object of the present invention is to provide an information processing system and a program that can realize a highly versatile method that can execute various tasks even when memory capacity is small. [Means for solving the problem]

[0006] An information processing system according to one embodiment of the present invention comprises a processor that executes a specific process for estimating the state of a monitored object based on an image of the monitored object, and a memory in which instructions and data used in executing the specific process are stored, and the processor performs sequential processing in which the specific process is divided into a plurality of processing units and each divided process is individually executed, by freeing up the memory area used in the previous divided process and then using the freed up memory area to execute the current divided process.

[0007] Furthermore, one form of information processing system according to the present invention comprises a processor that executes a specific process for estimating the state of a monitored object based on an image of the monitored object, and a memory in which instructions and data used to execute the specific process are stored, and the processor performs sequential processing by dividing the captured image into a plurality of images and individually executing the specific process for each divided image, thereby freeing up the memory area used in the previous specific process and executing the current specific process using the freed up memory area. [Effects of the Invention]

[0008] According to the present invention, a highly versatile method can be realized that is capable of executing various tasks even when memory capacity is small. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram showing an outline of information processing according to the present invention. [Figure 3] FIG. 3 is a diagram illustrating an overall image of the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating the procedure of information processing according to the first embodiment. [Figure 5] FIG. 5 is a diagram illustrating an overall image of the second embodiment. [Figure 6] FIG. 6 is a diagram illustrating the procedure of information processing according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0011] One or more embodiments (including examples, modifications, and application examples) described below can be implemented independently. However, at least a portion of the embodiments described below may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from each other. Therefore, these embodiments may contribute to solving different purposes or problems and may produce different effects from each other.

[0012] (Embodiment) 1. Introduction Power for base stations is supplied by generators. While the generators are running, it is impossible to know when the generator's fuel will run out, and the only option is to manually calculate when to refuel. This has resulted in situations where, when the remaining fuel meter is checked on-site, there is sufficient fuel and no need to refuel, or conversely, the amount of fuel prepared is not enough.

[0013] Therefore, in order to remotely check the remaining fuel in a generator without purchasing expensive equipment, the inventor of the present invention installed a general-purpose IoT device (e.g., a smart camera) at a base station and implemented a program that calculates the remaining fuel from a captured image of the remaining fuel meter.

[0014] However, general-purpose IoT devices have a limited non-volatile memory capacity (for example, about 1.5 MB), which means that there is an overwhelming lack of memory to execute tasks. For this reason, there has been a demand for a highly versatile method that can execute various tasks even when memory capacity is limited.

[0015] In view of the above-mentioned problems, the present invention provides a program and system that can cause a processor to execute specific processing (tasks) for estimating the state of a monitored object based on captured images obtained by photographing the monitored object, by individually and sequentially executing specific processing for each divided element into which a certain element is divided into predetermined units.

[0016] In the proposed technology of the present invention, since data is repeatedly read from and discarded into memory while specific processes are executed individually and sequentially, memory utilization rate decreases each time a specific process ends. As a result, the next specific process can be executed in a state where memory utilization rate has decreased, making it possible to execute each specific process even with a small memory capacity.

[0017] In the following embodiments, the monitored object is described as "remaining fuel in a generator" and the state of the monitored object is described as "the numerical value of remaining fuel," but the monitored object and the state of the monitored object to which the proposed technology of the present invention can be applied are not limited to these examples. For example, the proposed technology of the present invention can be widely applied to use cases such as calculating the state of an object related to an object (e.g., predetermined parameters such as speed, temperature, humidity, etc.) from information about the object detected from a captured image. In other words, the proposed technology of the present invention is thought to be useful for various applications other than calculating the remaining fuel in a generator, and can be said to be a highly versatile method that can perform various tasks even when memory capacity is small.

[0018] In addition, as will be described later, in the proposed technology of the present invention, the state of the monitored object is estimated from the results of image analysis of the captured image, and an object detection model is used for the image analysis. That is, the captured image is input to an object detection model (AI) to perform processing to detect a specific object in the captured image.

[0019] The task of the object detection model is to simultaneously estimate the coordinates (localization) and type (classification) of objects contained in the input image, and then perform post-processing to output a list of bounding box information (e.g., the x-coordinate of the center of the box, the y-coordinate of the center of the box, the width of the box, the height of the box) and a reliability score (probability) of the detection result as the detection result. This task corresponds to the "specific processing for estimating the state of the monitored object."

[0020] [2. System Configuration] An example of the configuration of a system 1 according to an embodiment is shown in Fig. 1. First, the terms used below, including those in Fig. 1, will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] For example, the various programs include a program (program according to the present invention) in which instructions are written to cause a processor to execute the above-mentioned specific processing (task) and an estimation processing for estimating the state of a monitored object (information processing according to the present invention), and the program is executed using the RAM as a working area. Therefore, the RAM has a memory area in which information necessary for the information processing according to the present invention is stored. Therefore, the RAM corresponds to the "memory" according to the present invention. The RAM may be a non-volatile RAM.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] 1, the system 1 includes a smart device 10 and a server device (cloud) 30. The smart device 10 is an example of an IoT device, and in this embodiment, is a smart camera with an imaging function.

[0027] The smart device 10 includes a computer 12, a reception device 14, an output device 16, an imaging device 18, and a communication I / F 20. The computer 12 includes a processor 22, a RAM 24, and a storage 26. The processor 22, the RAM 24, and the storage 26 are connected to a bus 28. The reception device 14, the output device 16, and the imaging device 18 are also connected to the bus 28. The communication I / F 20 is connected to a network N. Examples of the network N include a wide area network (WAN) and / or a local area network (LAN). For example, the communication I / F 20 can transmit an estimation result of the state of the monitored object to a server device 30 via the network N.

[0028] The reception device 14 includes a touch panel 14A, a microphone 14B, and the like, and receives user input. The touch panel 14A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the pointer. The microphone 14B detects the user's voice to receive user input by voice. For example, the communication I / F 20 can transmit data indicating the user input received by the touch panel 14A and the microphone 14B to the server device 30 via the network N.

[0029] The output device 16 includes a display 16A and a speaker 16B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., audio and / or text). The display 16A displays visual information such as text and images in accordance with instructions from the processor 22. The speaker 16B outputs audio in accordance with instructions from the processor 22.

[0030] The computer 12 includes at least a processor 22 and a RAM 24, and corresponds to the "information processing system" according to the present invention.

[0031] The imaging device 18 is a small digital camera equipped with an optical system including a lens, an aperture, a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] 3. Overview of Information Processing According to the Present Invention Fig. 2 is a diagram showing an overview of information processing according to the present invention. Fig. 2 shows a communication base station BS and a generator GN that supplies power to the communication base station BS. The generator GT has a remaining fuel meter MT, and a smart device 10 is installed in a position where it can photograph the remaining fuel meter MT. The remaining fuel meter MT may be an analog meter with a needle that indicates the remaining fuel, or a digital meter that displays the remaining fuel on a screen.

[0033] The smart device 10 (specifically, the processor 22) performs information processing according to the present invention by executing a program in the RAM 24. For example, the smart device 10 performs information processing according to the present invention at predetermined intervals (for example, every 15 minutes).

[0034] The information processing according to the present invention may be a set of photographing by the imaging device 18, a detection process (inference) for detecting a specified object from the captured image using an object detection model read from RAM 24, and an estimation process for estimating and calculating the value of remaining fuel from the detection results.

[0035] More specifically, the smart device 10 captures an image of the remaining fuel meter MT.

[0036] The smart device 10 executes a detection process using the captured image as an input image for the object detection model to cause the object detection model to detect a predetermined object from the captured image. This process is a task of the object detection model and corresponds to the identification process of the present invention. The identification process uses a memory area of ​​the RAM 24. For example, the memory area is used for the captured image, the object detection model, and image analysis using the object detection model.

[0037] However, since the smart device 10 is an inexpensive general-purpose IoT device, its memory capacity is small. Therefore, if a specific process is simply executed, the memory capacity may be insufficient, causing the processor 22 to stop operating. To avoid this situation and enable the specific process to be executed even when the memory capacity is small, the information processing according to the present invention is executed. Specifically, as shown in FIG. 2, the processor 22 executes information processing in which a certain element is divided into predetermined units and the specific process is executed individually and sequentially for each divided element.

[0038] Information processing according to the present invention can be divided into two methods. One is a method in which a single detection process using a single high-performance object detection model capable of detecting various types of objects is divided into detection processes using multiple lightweight object detection models prepared according to the types of objects to be detected from a captured image, each of which can detect only specific types of objects. These divided processes are executed individually and sequentially, and data is repeatedly read from and discarded from the RAM 24 during the process. This method will be described below as a first embodiment.

[0039] The other is a method in which a captured image is divided into a plurality of images, and a detection process for each divided image is performed individually and sequentially to detect a predetermined object from the divided images using an object detection model, and during this process, data is repeatedly read into the RAM 24 and discarded from the RAM 24. This method will be described below as a second embodiment.

[0040] (First embodiment) [1. Overall picture] Fig. 3 is a diagram illustrating an overall image of the first embodiment. Fig. 3 shows a scene in which information processing according to the first embodiment is performed in an estimation process in which the remaining fuel value is read from a captured image IM of an analog remaining fuel meter MT.

[0041] First, a description will be given of division according to the first embodiment. Fig. 3 shows an object detection model Mx as an example of one high-performance object detection model capable of detecting various types of objects.

[0042] The remaining fuel meter MT shown in FIG. 3 has a scale configuration that includes the character "empty" indicating the state of remaining fuel, the character "full" indicating the state of remaining fuel, and a "needle" indicating the scale between the characters "empty" and "full." In the case of such a scale configuration, to read the remaining fuel value, it is necessary to infer the positions of the character "empty," the character "full," and the object "needle" in the captured image IM using the object detection model Mx. More specifically, as shown in FIG. 3, it is necessary to infer the positions of the bounding boxes surrounding "empty" (hereinafter abbreviated as "box position"), the box surrounding "full," and the box surrounding the "needle" using the object detection model Mx.

[0043] Here, it is assumed that a high-performance object detection model Mx capable of detecting, for example, three types of objects, K0 to K2, is used. As shown in Fig. 3, for example, object type K0 is the object "pointer," object type K1 is the character "empty," and object type K2 is the character "full." When a captured image IM is input, the object detection model Mx outputs, as a detection result of detecting the three types of objects, K0 to K2, position information (center x coordinate, center y coordinate) of a box enclosing the object "pointer," position information (center x coordinate, center y coordinate) of a box enclosing the character "empty," and position information (center x coordinate, center y coordinate) of a box enclosing the character "full."

[0044] However, in the smart device 10, the memory capacity of the memory area of ​​the RAM 24 is small. Therefore, if the processor 22 attempts to use the highly functional object detection model Mx to infer three types of objects, K0 to K2, at once, the memory capacity may be insufficient, causing the operation of the processor 22 to stop.

[0045] Therefore, in the first embodiment, a detection process using one high-performance model called object detection model Mx is divided into separate processes for each of a plurality of different models, and these processes are executed individually and sequentially. Specifically, in the example of Fig. 3, an object detection model M0 capable of detecting only objects (pointers) of type K0, an object detection model M1 capable of detecting only objects (sky) of type K1, and an object detection model M2 capable of detecting only objects (full) of type K2 are used instead of object detection model Mx.

[0046] Because object detection model M0, object detection model M1, and object detection model M2 are lightweight models specialized for detecting one type of object, the memory capacity required for each detection process is smaller than that required for object detection model Mx. For this reason, in the example of Fig. 3, processor 22 causes detection process using object detection model M0, detection process using object detection model M1, and detection process using object detection model M2 to be executed individually and sequentially.

[0047] 3, the detection process by object detection model Mx is divided into units of a plurality of different object detection models My (object detection model M0, object detection model M1, object detection model M2) prepared to separately detect three types of objects, K0 to K2. Therefore, the detection process by object detection model M0, the detection process by object detection model M1, and the detection process by object detection model M2 can each be said to be a divided process obtained by dividing the detection process by object detection model Mx.

[0048] The flow of information processing according to the first embodiment will now be described using the example of Fig. 3. Fig. 3 shows a scene in which a first detection process Y1, which is a first stage of detection processing using object detection model M1, is performed, followed by a second detection process Y2, which is a second stage of detection processing using object detection model M0, and finally a third detection process Y3, which is a third stage of detection processing using object detection model M2. Note that the order in which the detection processes are performed among object detection model M1, object detection model M2, and object detection model M3 is not limited.

[0049] According to the example of FIG. 3, the processor 22 first retrieves the object detection model M1 from the storage 26 and loads the object detection model M1 into the RAM 24. Then, the processor 22 reads the object detection model M1 from the RAM 24 and executes a first detection process Y1 in the RAM 24 using the object detection model M1. In this case, as shown in FIG. 3, the memory capacity utilization rate of the memory area of ​​the RAM 24 increases. When the object detection model M1 outputs a detection result including position information (center x coordinate, center y coordinate) of a box AR1 enclosing the character "sky," the processor 22 releases the memory area of ​​the RAM 24 upon completion of the first detection process Y1. Specifically, the processor 22 deletes the object detection model M1 from the RAM 24. As a result, the memory capacity utilization rate of the memory area of ​​the RAM 24 decreases, as shown in FIG. 3.

[0050] In this state where the memory capacity utilization rate is reduced, the processor 22 executes the next second detection process Y2. The processor 22 first retrieves the object detection model M2 from the storage 26 and loads the object detection model M2 into the RAM 24. The processor 22 then reads the object detection model M2 from the RAM 24 and executes the second detection process Y2 using the object detection model M2 in the RAM 24. In this case, as shown in FIG. 3, the memory capacity utilization rate of the memory area of ​​the RAM 24 increases. When the object detection model M2 outputs a detection result including position information (center x coordinate, center y coordinate) of the box AR0 enclosing the object "pointer," the processor 22 releases the memory area of ​​the RAM 24 upon completion of the second detection process Y2. Specifically, the processor 22 deletes the object detection model M2 from the RAM 24. As a result, as shown in FIG. 3, the memory capacity utilization rate of the memory area of ​​the RAM 24 decreases.

[0051] In this way, with the memory capacity utilization rate down, the processor 22 executes the last third detection process Y3. First, the processor 22 retrieves the object detection model M3 from the storage 26 and reads the object detection model M3 into the RAM 24. Then, the processor 22 reads the object detection model M3 from the RAM 24 and executes the third detection process Y3 using the object detection model M3 in the RAM 24. In this case, as shown in FIG. 3, the memory capacity utilization rate of the memory area possessed by the RAM 24 increases. When the object detection model M3 outputs a detection result including the position information (center x coordinate, center y coordinate) of the box AR2 surrounding the character "full", the processor 22 releases the memory area possessed by the RAM 24 in response to the end of the third detection process Y3. Specifically, the processor 22 deletes the object detection model M3 from the RAM 24. As a result, as shown in FIG. 3, the memory capacity utilization rate of the memory area possessed by the RAM 24 decreases.

[0052] In this way, every time the processor 22 executes a split process, the processor 22 deletes the object detection model My used in the split process from the RAM 24 to release the used area, and repeats the sequential process the number of times equal to the number of split processes.

[0053] When all the split processes are completed, the processor 22 estimates the remaining fuel value based on the combination of the detection results by the object detection model M1, the detection results by the object detection model M0, and the detection results by the object detection model M2. Specifically, the processor 22 executes an estimation process of reading the scale indicated by the pointer of the remaining fuel meter MT based on the combination of the position information of the box AR1 surrounding the character "empty", the position information of the box AR0 surrounding the object "pointer", and the position information of the box AR2 surrounding the character "full".

[0054] The position information of the box AR1, the position information of the box AR0, and the position information of the box AR2 each include a center x coordinate and a center y coordinate. For example, the processor 22 can estimate and calculate the scale indicated by the pointer of the remaining fuel meter MT based on the respective center y coordinates.

[0055] [2. Processing Procedure] 4 is a diagram illustrating the procedure of information processing according to the first embodiment. In FIG. 4, the procedure of information processing will be explained based on the overall image of the first embodiment explained in FIG.

[0056] The imaging device 18 periodically captures an image of the monitoring target. Then, the processor 22 determines whether or not a captured image IM has been acquired by capturing an image of the monitoring target (step S401). While a captured image IM has not been acquired (step S401; No), the processor 22 waits until a captured image IM is acquired.

[0057] When the captured image IM is acquired (step S401; Yes), the processor 22 reads the captured image IM into the RAM 24 (step S402).

[0058] Processor 22 also determines whether or not individual detection processes using each of a plurality of object detection models My prepared according to the type of object to be detected have all been completed for the number of object detection models My (step S403).

[0059] If all of the individual detection processes using each object detection model My have not been completed (step S403; No), processor 22 retrieves one of the unused object detection models My that corresponds to the current processing stage n from storage 26 and reads it into RAM 24 (step S404).

[0060] In this state, the processor 22 executes the n-th detection process corresponding to the current processing stage n, using the object detection model My read out from the RAM 24 and the captured image IM (step S405).

[0061] Processor 22 determines whether the n-th detection process has ended based on whether object detection model My has output a detection result (step S406). If object detection model My has not output a detection result and the n-th detection process has not ended (step S406; No), processor 22 waits until the n-th detection process ends.

[0062] When the n-th detection process ends with the object detection model My outputting the detection result (step S406; Yes), the processor 22 stores the detection result by the object detection model My (step S407).

[0063] Next, the processor 22 deletes the object detection model My used in the n-th detection process from the RAM 24 (step S408).

[0064] Processor 22 increments the number of the current processing stage n by one and moves to the next processing stage, thereby repeating step S403 and subsequent steps (step S409) so that detection processing can be performed using another unused object detection model My. The repetition of steps S403 to S409 is performed individually and sequentially for each object detection model My.

[0065] When all individual detection processes using each object detection model My have been completed (step S403; Yes), processor 22 estimates the state of the monitored object based on the combination of detection results (box position information) for each object detection model My saved in step S407 (step S410).

[0066] Then, under the control of the processor 22, the communication I / F 20 uploads the estimation result obtained in step S410 to the cloud (server device 30) (step S411).

[0067] (Second embodiment) [1. Overall picture] Fig. 5 is a diagram illustrating an overall image of the second embodiment. Fig. 5 shows a scene in which information processing according to the second embodiment is performed in an estimation process in which the remaining fuel value is read from a captured image IM of an analog remaining fuel meter MT.

[0068] First, division according to the second embodiment will be described. If the original captured image IM acquired by the imaging device 18 is left as it is, the image size (number of pixels) will be large, and there is a problem that the smart device 10 cannot be equipped with a RAM 24 having a memory area that can accommodate this image size, or the memory area capacity of the RAM 24 is small and the captured image IM cannot be read as is.

[0069] For this reason, it is possible to reduce the image size by shrinking the original captured image IM, but reducing the image size reduces the resolution, which causes another problem: the detection accuracy of the object detection model decreases.

[0070] Therefore, in the second embodiment, a detection process for each divided image IMn obtained by dividing a captured image IM is performed individually and sequentially, in which a predetermined object is detected from the divided image IMn using an object detection model. The divided image IMn has a smaller image size than the captured image IM, but is not a reduced image, so the resolution itself is maintained. For this reason, the second embodiment realizes a highly versatile method that can execute various tasks even when memory capacity is small, from the perspective of dividing an image rather than dividing processing as in the first embodiment.

[0071] Furthermore, the detection process for each divided image IMn is executed individually and sequentially, during which the divided image IMn is repeatedly read into the RAM 24 and discarded from the RAM 24.

[0072] Furthermore, the detection process for each divided image IMn may be performed individually and sequentially using multiple lightweight models for each captured image IM consisting of divided images IMn, rather than using a single highly functional object detection model Mx. The lightweight model here refers to a lightweight model specialized for detecting one type of object. For this reason, the information processing according to the second embodiment may also be combined with the information processing according to the first embodiment. In this regard, the flow of the information processing according to the second embodiment will be described using the example of FIG. 5. The example shown in FIG. 3 will also be used as appropriate.

[0073] As shown in Fig. 5, the processor 22 may obtain nine divided images IMn by dividing the captured image IM into nine parts. Fig. 5 shows divided images IM1, IM2, IM3, etc. (hereinafter referred to as "divided images IM1 to IM9") as examples of the nine divided images IMn. Note that the number of divisions of the captured image IM by the processor 22 is not limited to nine, but it is preferable to limit the number of divisions to a number that does not result in the detection object being divided into too many parts according to the image division.

[0074] 5 also shows a scene in which a first detection process Y1, which is a first-stage detection process using object detection model M1, then a second detection process Y2, which is a second-stage detection process using object detection model M0, and finally a third detection process Y3, which is a third-stage detection process using object detection model M2, is performed. Specifically, in the information processing according to the second embodiment, the first detection process Y1, the second detection process Y2, and the third detection process Y3 are performed individually and sequentially for each captured image IM composed of divided images IMn.

[0075] In each of the first detection process Y1, the second detection process Y2, and the third detection process Y3, the process of detecting a target object from each of the divided images IM1 to IM9 is executed individually and sequentially.

[0076] A more specific explanation will be given using the example of segmented image IM1. The processor 22 retrieves the object detection model M1 from the storage 26 and loads the object detection model M1 into the RAM 24. The processor 22 also loads into the RAM 24 segmented image IM1, which is one of segmented images IMn obtained by dividing the captured image IM into nine segments. The processor 22 then executes a detection process in the RAM 24 to cause the object detection model M1 to detect an object of type K1 (sky) from the segmented image IM1. In this case, the memory capacity utilization rate of the memory area of ​​the RAM 24 increases. When the object detection model M1 outputs a detection result including position information (center x coordinate, center y coordinate) of a box enclosing the characters "sky" and the reliability (probability) of the presence of the characters "sky" at that position, the processor 22 ends the current detection process and frees the memory area of ​​the RAM 24. Specifically, the processor 22 deletes the segmented image IM1 from the RAM 24. As a result, the memory capacity utilization rate of the memory area of ​​the RAM 24 decreases.

[0077] The processor 22 also sequentially performs the detection process performed on the segmented image IM1 on each of the segmented images IM2 to IM9, but sequentially deletes the segmented image IMn used in the current detection process from the RAM 24 and moves on to the next detection process. As a result, the processor 22 can obtain detection results (box position information and probability) by the object detection model M1 for each of the segmented images IM1 to IM9. For example, as shown in FIG. 5, the processor 22 can obtain a box indicating the position of the characters "sky" for each of the segmented images IM1 to IM9. Therefore, the processor 22 extracts the detection result RE1 with the highest probability from the detection results associated with each of the segmented images IM1 to IM9. According to the detection result RE1, the segmented image IMn (in the example of FIG. 5, the segmented image IM2) that is most likely to contain the characters "sky" is identified from among the segmented images IM1 to IM9. Therefore, the processor 22 can use this segmented image IMn to estimate the remaining fuel value. The above is a specific example of the first detection process Y1 executed in units of captured images IM.

[0078] Therefore, when the first detection process Y1 ends, the processor 22 deletes the object detection model M1 from the RAM 24, thereby further reducing memory capacity utilization, and then executes the second detection process Y2. The second detection process Y2 differs in that the object detection model M0 is used, but otherwise the process proceeds in the same manner as the method described for the first detection process Y1. As a result of executing the second detection process Y2, the processor 22 can obtain detection results (box position information and probability) by the object detection model M0 for each of the divided images IM1 to IM9. For example, as shown in FIG. 5, the processor 22 can obtain boxes indicating the position of the object "pointer" for each of the divided images IM1 to IM9. Therefore, the processor 22 extracts the detection result RE2 with the highest probability from the detection results associated with each of the divided images IM1 to IM9. According to the detection result RE2, the most likely divided image IMn (in the example of Figure 5, divided image IM5) among divided images IM1 to IM9 in which the object "needle" is present is identified, and therefore the processor 22 can use this divided image IMn to estimate the value of remaining fuel.

[0079] Furthermore, when the second detection process Y2 ends, the processor 22 deletes the object detection model M3 from the RAM 24, thereby further reducing memory capacity utilization, and then executes the second detection process Y2. The third detection process Y3 differs in that the object detection model M2 is used, but otherwise the process proceeds in the same manner as the method described for the first detection process Y1. As a result of executing the third detection process Y3, the processor 22 can obtain detection results (box position information and probability) by the object detection model M2 for each of the divided images IM1 to IM9. For example, as shown in FIG. 5, the processor 22 can obtain boxes indicating the position of the character "full" for each of the divided images IM1 to IM9. Therefore, the processor 22 extracts the detection result RE3 with the highest probability from the detection results associated with each of the divided images IM1 to IM9. According to the detection result RE3, the most likely divided image IMn (in the example of Figure 5, divided image IM8) contains the character "Full" among divided images IM1 to IM9, and therefore the processor 22 can use this divided image IMn to estimate the value of remaining fuel.

[0080] When all detection processes are completed, processor 22 estimates the value of remaining fuel based on a combination of the most probable detection result RE1 among the detection results for each segmented image IMn by object detection model M1, the most probable detection result RE2 among the detection results for each segmented image IMn by object detection model M0, and the most probable detection result RE3 among the detection results for each segmented image IMn by object detection model M2. In the example of Fig. 5, processor 22 executes an estimation process to read the scale indicated by the needle of the remaining fuel meter MT based on a combination of positional information of a box enclosing the character "empty" in segmented image IM2, positional information of a box enclosing the object "needle" in segmented image IM5, and positional information of a box enclosing the character "full" in segmented image IM8.

[0081] [2. Processing Procedure] 6 is a diagram illustrating the procedure of information processing according to the second embodiment. In FIG. 6, the procedure of information processing is explained based on the overall image of the first embodiment explained in FIG.

[0082] The imaging device 18 periodically captures an image of the monitoring target. Then, the processor 22 determines whether or not a captured image IM has been acquired by capturing an image of the monitoring target (step S601). While a captured image IM has not been acquired (step S601; No), the processor 22 waits until a captured image IM is acquired.

[0083] When a captured image IM is acquired (step S601; Yes), the processor 22 divides the captured image IM into a predetermined number of divided images IMn (step S602).

[0084] In addition, the processor 22 retrieves one of the unused object detection models My from the storage 26, which are prepared according to the type of object to be detected, and reads it into the RAM 24 (step S603).

[0085] The processor 22 also determines whether or not the individual detection processes using the divided images IMn have all been completed for the number of divided images IMn (step S604). In other words, the processor 22 determines whether or not the detection processes using one object detection model My have been completed for each captured image IM made up of divided images IMn.

[0086] If all of the individual detection processes using each divided image IMn have not been completed (step S604; No), the processor 22 acquires one of the unused divided images IMn and reads it into the RAM 24 (step S605).

[0087] In this state, the processor 22 executes the current detection process using the object detection model My read out from the RAM 24 and the divided image IMn (step S606).

[0088] The processor 22 determines whether the current detection process has ended based on whether the object detection model My has output a detection result (step S607). If the object detection model My has not output a detection result and the detection process has not ended (step S607; No), the processor 22 waits until the current detection process ends.

[0089] When the object detection model My has output the detection result and thus the current detection process has ended (step S607; Yes), the processor 22 stores the detection result by the object detection model My (step S608).

[0090] Next, the processor 22 deletes the divided image IMn used in this detection process from the RAM 24, and returns to step S604 (step S609). As a result, the processor 22 repeats step S603 and subsequent steps so that the detection process can be performed on another unused divided image IMn. The repetition of steps S604 to S609 causes the detection process for each divided image IMn to be performed individually and sequentially using one object detection model My.

[0091] When all of the individual detection processes using the divided images IMn have been completed (step S604; Yes), the processor 22 deletes the object detection model My used in the current detection process from the RAM 24 (step S610).

[0092] The processor 22 also determines whether or not all individual detection processes using the object detection models My prepared according to the types of objects to be detected have been completed for the number of object detection models My (step S611). In other words, the processor 22 determines whether or not all detection processes have been completed for each captured image IM composed of divided images IMn.

[0093] If all of the individual detection processes using each object detection model My have not been completed (step S611; No), processor 22 returns the process to step S603.

[0094] On the other hand, when all individual detection processes using each object detection model My have been completed (step S611; Yes), processor 22 extracts the detection result with the highest reliability from the detection results (box position information) for each object detection model My saved in step S608 (step S612).

[0095] Then, the processor 22 estimates the state of the monitoring target based on the combination of the extracted detection results (step S613).

[0096] Then, under the control of the processor 22, the communication I / F 20 uploads the estimation result obtained in step S613 to the cloud (server device 30) (step S614).

[0097] (Other embodiments) In the second embodiment, the detection process for each divided image IMn is performed individually and sequentially using a plurality of lightweight models for each captured image IM consisting of the divided images IMn. However, the detection process for each divided image IMn may be performed using a single high-performance object detection model Mx.

[0098] (others) Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0099] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0100] Furthermore, the above-described embodiments can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0101] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that include the aspects described in the "present invention" section and that have been modified and improved in various ways based on the knowledge of those skilled in the art. [Explanation of symbols]

[0102] 1 System 10. Smart Devices 12. Computer 18 Imaging device 22 processors 24 RAM 26 Storage

Claims

1. a processor that executes a specific process for estimating a state of a monitoring target based on a captured image of the monitoring target; a memory for storing instructions and data used to execute the specific process; the processor performs sequential processing in which the specific processing is divided into a plurality of processing units and each divided processing is individually executed, by releasing the memory area used in the previous divided processing, and then executing the current divided processing using the free memory area after the release. Information processing system.

2. the segmentation process is a process of detecting each of predetermined objects necessary for estimating the state of the monitoring target from the captured image, and is an individual process executed by a plurality of object detection models prepared for detecting each of the predetermined objects, The processor: performing the sequential processing of deleting from the memory the object detection models used in the previous division process to free up the memory's used area, executing the current division process using the free memory area after the freeing, and deleting from the memory the object detection models used in the current division process to free up the memory's used area, the processing being repeated the same number of times as the number of division processes; The information processing system according to claim 1 .

3. The processor estimates a state of the monitored object based on a combination of the detection results for each of the object detection models. The information processing system according to claim 2 .

4. a processor that executes a specific process for estimating a state of a monitoring target based on a captured image of the monitoring target; a memory for storing instructions and data used to execute the specific process; the processor performs sequential processing of individually executing the specific processing for each divided image obtained by dividing the captured image into a plurality of images, by releasing the used area of ​​the memory used in the previous specific processing, and then executing the current specific processing using the free area of ​​the memory after the release. Information processing system.

5. the identification process for each divided image is a process of detecting a predetermined object necessary for estimating the state of the monitoring target from the captured image using an object detection model as the identification process, for each divided image obtained by dividing the captured image into a plurality of images; The processor: performing the sequential processing of deleting from the memory the divided images used in the previous specified process to free up the memory's used area, executing the current specified process using the free area of ​​the memory after the freeing, and deleting from the memory the divided images used in the current specified process to free up the memory's used area, the processing of which is repeated the number of times equal to the number of the divided images; The information processing system according to claim 4 .

6. the processor estimates a state of the monitoring target based on a combination of the detection results for each of the divided images. The information processing system according to claim 5 .

7. the processor executes the identification process for each of the divided images using a plurality of the object detection models, and estimates the state of the monitored object based on a combination of the detection results for each of the object detection models with a higher reliability among the detection results by the plurality of the object detection models. The information processing system according to claim 6 .

8. A program executed by a processor that executes a specific process for estimating a state of a monitoring target based on a captured image of the monitoring target, A program for causing the processor to execute a sequential process in which the specific process is divided into multiple processing units and each divided process is executed individually, by freeing up the memory area used in the previous divided process in the memory that stores the instructions and data used to execute the specific process, and then executing the current divided process using the free memory area after the freeing up.

9. A program executed by a processor that executes a specific process for estimating a state of a monitoring target based on a captured image of the monitoring target, A program for causing the processor to execute a sequential process in which the specific process is executed individually for each divided image obtained by dividing the captured image into multiple images, by freeing up the memory area used in the previous specific process among the memory in which the instructions and data used to execute the specific process are stored, and then executing the current specific process using the freed up memory area.

Images