Information processing device, information processing method, and program

Abstract

To share the remaining processing that has not been executed with the cloud side on the basis of a processing content, thereby reducing processing delays when edge-side processing cannot be executed within a specified period.SOLUTION: An information processing device includes a first image processing unit that performs first image processing on an image for each processing period and generates first information, a second image processing unit that performs second image processing using the first information during the processing period and generates second information, a first estimation unit that estimates whether the second image processing can be completed within the current processing period on the basis of the time required for the first image processing and the first information, and a distribution unit that distributes the first information to be used in the remaining second image processing to the second image processing unit and a third image processing unit that performs the second image processing in a processing period after the current processing period when it is estimated that the second image processing cannot be completed during the current processing period.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In cloud computing, a server computer (cloud server) on the cloud executes information processing using data aggregated from edge devices arranged at the peripheral part (edge) of a computer network. However, when communication delay, network load, etc. increase, it may affect information processing. Therefore, edge computing has been proposed.

[0003] In edge computing, part of the information processing is executed using an edge device and a server computer (edge server) arranged in the peripheral area of the edge device, and the processed data is transmitted to a server computer on the cloud. Thus, in edge computing, by reinforcing cloud computing, the influence of communication delay, network load, etc. is reduced.

[0004] However, since edge devices and edge servers are not usually as high-performance as cloud servers, when the amount of information to be processed increases, processing delay may occur in edge devices and edge servers.

[0005] As a related technique, Patent Document 1 discloses an information processing apparatus that reduces an excess or deficiency of data transferred to a processing node that takes over data processing. The information processing apparatus of Patent Document 1 manages a plurality of processing nodes including a buffer and a processing unit that processes data held in the buffer. Further, the information processing apparatus predicts the boundary between processed data and unprocessed data in the buffer at the predicted arrival time when the resource load of the processing node during data processing reaches a predetermined amount, and based on the prediction, transfers the unprocessed data from the side with a slower processing order toward the boundary to another processing node that takes over data processing.

Prior Art Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2022-74864 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] However, the information processing device described in Patent Document 1 determines whether or not to distribute the processing performed by the edge device and edge server used in edge computing (edge-side processing) between the edge side and the cloud server (cloud side) based on resource load, and therefore cannot reduce processing delay.

[0008] One example of the purpose of this disclosure is to reduce processing delays by distributing the remaining unprocessed tasks to the cloud based on the processing content, if it is estimated that edge-side processing cannot be completed within a predetermined period. [Means for solving the problem]

[0009] To achieve the above objective, the information processing device in one aspect of this disclosure is: A first image processing unit performs a first image processing operation on images acquired at predetermined processing intervals to generate first information, During the aforementioned processing period, a second image processing unit performs a second image processing using the first information to generate the second information, A first estimation unit estimates whether the second image processing can be completed within the current processing period, based on the time required for the first image processing and the first information. If it is estimated that the second image processing cannot be completed within the current processing period, a distribution unit distributes the first information used in the remaining second image processing to the second image processing means and a third image processing means, which is provided separately from the second image processing means and performs the second image processing, in order to perform the remaining second image processing in a predetermined number of processing periods after the current processing period. It is characterized by having the following features.

[0010] Furthermore, in order to achieve the above objectives, the information processing method in one aspect of this disclosure is: Information processing device, The first image processing is performed on the images acquired at predetermined processing intervals to generate the first information. During the aforementioned processing period, a second image processing step is performed using the first information to generate the second information. Based on the time required for the first image processing and the first information, it is estimated whether the second image processing can be completed within the current processing period. If it is estimated that the second image processing cannot be completed within the current processing period, the first information used in the remaining second image processing is distributed to the information processing device and to other information processing devices that perform the second image processing, which are provided separately from the information processing device, in order to perform the remaining second image processing over a predetermined number of processing periods after the current processing period. It is characterized by the following:

[0011] Furthermore, in order to achieve the above objectives, the program in one aspect of this disclosure is On the computer, The first image processing is performed on the images acquired at predetermined processing intervals to generate the first information. During the aforementioned processing period, a second image processing is performed using the first information to generate the second information. Based on the time required for the first image processing and the first information, it is estimated whether the second image processing can be completed within the current processing period. When it is estimated that the second image processing cannot be completed during the current processing period, in order to execute the remaining second image processing in a preset number of processing periods after the current processing period, the first information used in the remaining second image processing is distributed to the computer and another computer that executes the second image processing provided separately from the computer. characterized by causing the execution of the instruction.

Effect of the Invention

[0012] As described above, according to the present disclosure, when it is estimated that the processing on the edge side cannot be executed within a predetermined period, by distributing the remaining processing that could not be processed to the cloud side based on the processing content, the processing delay can be reduced.

Brief Description of the Drawings

[0013] [Figure 1] FIG. 1 is a diagram for explaining an example of the system of Embodiment 1. [Figure 2] FIG. 2 is a diagram for explaining an example of the operation of the system of Embodiment 1. [Figure 3] FIG. 3 is a diagram for explaining an example of the operation of the system of Embodiment 1. [Figure 4] FIG. 4 is a diagram for explaining an example of the operation of the information processing device on the edge side of Embodiment 1. [Figure 5] FIG. 5 is a diagram for explaining an example of the system of Embodiment 2. [Figure 6] FIG. 6 is a diagram for explaining an example of the operation of the system of Embodiment 2. [Figure 7] FIG. 7 is a diagram for explaining an example of the operation of the information processing device on the edge side of Embodiment 2. [Figure 8] FIG. 8 is a diagram for explaining an example of the system of Example 1. [Figure 9] FIG. 9 is a diagram for explaining an example of the system of Example 2. [Figure 10]FIG. 10 is a diagram for explaining an example of a computer that realizes the information processing apparatus in Embodiments 1 and 2 and Examples 1 and 2.

Mode for Carrying Out the Invention

[0014] Hereinafter, embodiments will be described with reference to the drawings. In the drawings described below, elements having the same function or corresponding functions are denoted by the same reference numerals, and repeated description thereof may be omitted.

[0015] (Embodiment 1) In Embodiment 1, an edge-side information processing apparatus provided in a system that realizes edge computing executes a predetermined process in each of a plurality of processing periods.

[0016] If the edge-side information processing apparatus estimates that a predetermined process cannot be completed in the current processing period, the remaining processes of the predetermined process that could not be executed in the current processing period are executed by the edge-side information processing apparatus and the cloud-side information processing apparatus in a period after the current processing period.

[0017] Specifically, in the edge-side information processing apparatus, if there is a spare period for executing the remaining processes in one or more processing periods after the current processing period, the remaining processes are executed using the spare period. Also, the cloud-side information processing apparatus is made to execute the remaining processes earlier.

[0018] Thereafter, the processing result corresponding to the remaining process is acquired from the one that has completed the remaining process earlier between the edge-side information processing apparatus and the cloud-side information processing apparatus.

[0019] In other words, if the information processing unit on the edge side completes the remaining processing faster than the information processing unit on the cloud side, the results of the remaining processing performed by the information processing unit on the edge side are obtained. Conversely, if the information processing unit on the cloud side completes the remaining processing faster than the information processing unit on the edge side, the results of the remaining processing performed by the information processing unit on the cloud side are obtained.

[0020] Thus, in Embodiment 1, the processing results corresponding to the remaining processing can be obtained from the user who completes the remaining processing first, thus reducing processing delay compared to conventional methods.

[0021] For example, in an edge-side information processing device, there may be cases where sufficient buffer time cannot be secured in the processing period beyond the current processing period, resulting in delays in completing the remaining processing. In such cases, the remaining processing is executed on the cloud-side information processing device. If the cloud-side information processing device completes the remaining processing quickly, the results of the remaining processing executed on the cloud-side information processing device are used. As a result, the remaining processing that would have been executed in the current processing period can be completed earlier, thus reducing the overall processing delay.

[0022] [System Configuration] Embodiment 1 will be described in detail using Figure 1. Figure 1 is a diagram illustrating an example of the system of Embodiment 1.

[0023] System 100 is a system that realizes edge computing. System 100 in Figure 1 has an information processing device 10 located on the edge side, an information processing device 20 located on the cloud side, and a network 30. Information processing devices 10 and 20 communicate with each other via the network 30.

[0024] The information processing device 10 may be, for example, a CPU (Central Processing Unit), a programmable device such as an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or a circuit equipped with one or more of these, a server computer, a personal computer, a mobile terminal, etc.

[0025] The information processing device 20 may be, for example, a CPU, a programmable device such as an FPGA, a GPU, or a circuit equipped with one or more of these, or one or more server computers.

[0026] Network 30 is a general communication network constructed using communication lines such as the Internet, LAN (Local Area Network), dedicated lines, telephone lines, corporate networks, mobile communication networks, Bluetooth®, and WiFi (Wireless Fidelity).

[0027] ●The information processing device 10 on the edge side will be described below. As shown in Figure 1, the information processing device 10 includes a first image processing unit 11, a second image processing unit 12, a first estimation unit 13, a distribution unit 14, and a fourth image processing unit 15.

[0028] The first image processing unit 11 acquires images in a time series at predetermined processing intervals T1, performs first image processing on the acquired images, and generates first information. Images are acquired in a time series from, for example, an imaging device, a storage device, etc.

[0029] The first piece of information is used in the second image processing step, which will be described later. Furthermore, the first piece of information contains multiple input data points used in the second image processing step.

[0030] The second image processing unit 12 performs a second image processing using the first information during processing period T1, and generates the second information. The second information is the information used in the third image processing, which will be described later.

[0031] The first estimation unit 13 estimates whether the second image processing can be completed within the current processing period T1_0, based on the time required for the first image processing (period Ti1) and the first information generated during the first image processing.

[0032] Specifically, the first estimation unit 13 estimates the time (period Ti2) that can be used for the second image processing during the current processing period T1_0, and estimates whether the second image processing can be completed using the first information within the estimated period Ti2.

[0033] One possible method for estimating whether the second image processing can be completed during the period Ti2 is to first measure the processing time (average or worst-case) t2 for one input data in the second image processing in the information processing device 10, and then compare the magnitude of Ti2 and Ti2 × n using the number of data points n included in the first information.

[0034] In this way, by using easily observable factors that can change the processing time during execution (the number of input data in the previous example) and a known value (average or worst-case value) of the processing time per unit of those factors (the processing time for one input data in the previous example), the processing time required for the second image processing can be estimated. By comparing this estimated processing time with the period Ti2, it is possible to estimate whether the second image processing can be completed.

[0035] If the distribution unit 14 estimates that the second image processing cannot be completed in the current processing period T1_0, it distributes the first information used for the remaining second image processing to the second image processing unit 12 and the third image processing unit 21, which is provided separately from the second image processing unit 12 and performs the second image processing, in order to have the remaining second image processing executed in a predetermined number of processing periods (period T3) after the current processing period T1_0.

[0036] Specifically, in Embodiment 1, the distribution unit 14 distributes all the first information used in the remaining second image processing to the second image processing unit 12 and the third image processing unit 21 in order to execute the remaining second image processing during a predetermined number of processing periods (period T3) after the current processing period T1_0.

[0037] If, during the current processing period T1_0, the first information that was to be processed in the second image processing had 100 input data points, and 60 input data points of the first information were actually used in the second image processing during the current processing period T1_0, then the remaining 40 input data points of the first information will be used in the second image processing.

[0038] Once the remaining first information to be used in the second image processing is distributed, the second image processing unit 12 executes the second image processing using all of the remaining first information within a predetermined number of processing periods (period T3) after the current processing period T1_0, during which time there is sufficient leeway to execute the remaining second image processing. Similarly, the third image processing unit 21 also executes the second image processing using all of the remaining first information to be used in the second image processing.

[0039] The fourth image processing unit 15 obtains the second information generated by the remaining second image processing from the second image processing unit 12 and the third image processing unit 21, starting with the one that completes the remaining second image processing first. The fourth image processing unit 15 also performs the third image processing using the second information generated in the current processing period T1_0 and the second information generated in a predetermined number of processing periods (period T3) after the current processing period T1_0.

[0040] ●The information processing device 20 on the cloud side will be explained. As shown in Figure 1, the information processing device 20 has the third image processing device 21 described above. In the example in Figure 1, the fourth image processing device 15 is provided in the edge-side information processing device 10, but it may also be provided in the cloud-side information processing device 20.

[0041] The third image processing unit 21 first receives all of the remaining first information to be used in the second image processing, transmitted from the distribution unit 14, via the network 30. Next, the third image processing unit 21 uses all of the acquired first information to be used in the second image processing to perform the second image processing and generate the second information. Next, the third image processing unit 21 transmits the generated second information to the fourth image processing unit 15 via the network 30.

[0042] [System Operation] Figures 2 and 3 illustrate an example of the operation of the system of Embodiment 1. Period T2 in Figure 2A has five processing periods T1_0, T1_1, T1_2, T1_3, and T1_4. However, the number of processing periods is not limited to five.

[0043] Furthermore, Figure 2A shows an example in which the first and second image processing are completed within processing periods T1_0, T1_1, T1_2, T1_3, and T1_4, and the third image processing is completed in processing period T1_4. In other words, in a case like Figure 2A, the edge-side information processing device 10 alone can complete the first, second, and third image processing within period T2.

[0044] However, as shown in Figure 2B, during processing period T1_0, the second image processing may not be completed within processing period T1_0, resulting in an overflow of processing time. Therefore, during the period Ts shown in Figure 3A, the first estimation unit 13 estimates whether the second image processing can be completed within the current processing period T1_0 based on the time Ti1 required for the first image processing and the first information.

[0045] Next, if the first estimation unit 13 estimates that the second image processing cannot be completed in the current processing period T1_0, the distribution unit 14 distributes the first information to be used in the remaining second image processing to the second image processing unit 12 and the third image processing unit 21 in order to perform the remaining second image processing in the period T3, which is after the current processing period T1_0.

[0046] Specifically, the distribution unit 14 transmits all of the first information to be used in the remaining second image processing to the third image processing unit 21 of the information processing device 20 on the cloud side via the network 30. Next, as shown in Figure 3B, during period To, the third image processing unit 21 performs the remaining second image processing using all of the first information.

[0047] Furthermore, as shown in Figure 3C, the second image processing unit 12 performs the second image processing (overflow processing) in each of the buffer periods To1, To2, and To3 of the processing periods T1_1, T1_2, and T1_3, using the remaining first information used in the second image processing that can be executed in the buffer periods To1, To2, and To3.

[0048] Next, the fourth image processing unit 15 obtains the second information generated by the remaining second image processing from whichever of the two image processing units, the second image processing unit 12 or the third image processing unit 21, completes the remaining second image processing first. In the case of C in Figure 3, the third image processing unit 21 on the cloud side completes the remaining second image processing first, so the fourth image processing unit 15 obtains the second information from the third image processing unit 21.

[0049] Next, the fourth image processing unit 15 performs a third image processing operation using all the second information generated during each of the processing periods T1_0, T1_1, T1_2, T1_3, and T1_4.

[0050] [Device operation] The operation of the edge-side information processing device 10 in Embodiment 1 will be explained using Figure 4. Figure 4 is a diagram illustrating an example of the operation of the edge-side information processing device in Embodiment 1. In the following explanation, the diagram will be referred to as appropriate. In Embodiment 1, the information processing method is implemented by operating the information processing device 10. Therefore, the explanation of the information processing method in Embodiment 1 will be replaced by the following explanation of the operation of the information processing device.

[0051] As shown in Figure 4, the first image processing unit 11 acquires images in a time series for each processing period T1 (step A1). Next, the first image processing unit 11 performs the first image processing to generate the first information and stores the generated first information in memory (step A2). In step A2, the time required for the first image processing (period Ti1) is also measured.

[0052] Next, the first estimation unit 13 estimates whether the second image processing can be completed within the current processing period T1_0, based on the time required for the first image processing (period Ti1) and the first information generated in the first image processing (step A3).

[0053] Specifically, in step A3, the first estimation unit 13 estimates the time (period Ti2) that can be used for the second image processing in the current processing period T1_0, and estimates whether the second image processing can be completed using the first information within the estimated period Ti2.

[0054] Next, if the distribution unit 14 estimates that the second image processing cannot be completed in the current processing period T1_0 (Step A4: No), it distributes the first information used for the remaining second image processing to the second image processing unit 12 and the third image processing unit 21, which is provided separately from the second image processing unit 12 and performs the second image processing, in order to have the remaining second image processing performed in a predetermined number of processing periods (period T3) after the current processing period T1_0 (Step A5).

[0055] Specifically, in step A5, the distribution unit 14 distributes all of the remaining first information to be used in the second image processing to the second image processing unit 12 and the third image processing unit 21 over a predetermined number of processing periods (period T3) after the current processing period T1_0. The third image processing unit 21 then uses all of the remaining first information to be used in the second image processing to perform the second image processing and generate the second information.

[0056] If the distribution unit 14 estimates that the second image processing can be completed within the current processing period T1_0 (step A4: Yes), it proceeds to the processing in step A6.

[0057] Next, the second image processing unit 12, in the current processing period T1_0, performs the second image processing using the first information that will not be used in the remaining second image processing to generate the second information, and stores the generated second information in memory (step A6).

[0058] Next, if the fourth image processing unit 15 has obtained the second information corresponding to the remaining second image processing from the third image processing unit 21 (Step A7: Yes), that is, if the third image processing unit 21 generates the second information corresponding to the remaining second image processing faster than the second image processing unit 12, the fourth image processing unit 15 stores the second information corresponding to the remaining second image processing in memory (Step A12).

[0059] Furthermore, the second piece of information in step A7 includes not only the result of the distribution in the preceding step A5, but also the second piece of information resulting from the distribution in step A5 in the previous loop (e.g., T1_0 for T1_1).

[0060] Furthermore, if the third image processing unit 21 has not obtained the second information corresponding to the remaining second image processing (step A7: No), the process proceeds to step A8.

[0061] Next, if there is a remaining second image processing step (Step A8: Yes) and there is a buffer period (Step A9: Yes), the second image processing unit 12 performs the second image processing step (Step A10) during the buffer period using the first information to be used in the remaining second image processing step.

[0062] Furthermore, if there are no remaining second image processing steps (Step A8: No) and there is no buffer time (Step A9: No), the process proceeds to Step A1.

[0063] Next, if the second image processing unit 12 has completed all of the remaining second image processing (Step A11: Yes), it stores the second information corresponding to the remaining second image processing in memory (Step A12). If it has not completed all of the remaining second image processing (Step A11: No), it proceeds to the process in Step A1.

[0064] Next, in each processing period T3, if all second image processing is completed or distributed (Step A13: Yes) and there are any incomplete second image processing (Step A14: Yes), the process waits for the completion of the third image processing and obtains the second information (Step A15). Then, the fourth image processing unit 15 uses all the second information generated in each processing period T3 to perform the third image processing (Step A16).

[0065] Furthermore, if, in each processing period of period T3, all second image processing is completed or distributed (Step A13: Yes) and there is no incomplete second image processing (Step A14: No), the fourth image processing unit 15 uses all the second information generated in each processing period of period T3 to perform the third image processing (Step A16).

[0066] Furthermore, if, during each processing period T3, the fourth image processing unit 15 has not completed or distributed all of the second image processing (step A13: No), it proceeds to the processing in step A1.

[0067] In Embodiment 1, the processes described in steps A1 to A16 above are repeatedly executed.

[0068] [Effects of Embodiment 1] As described above, according to Embodiment 1, the second information generated by the remaining second image processing is acquired from the second image processing unit 12 and the third image processing unit 21, whichever completes the remaining second image processing earlier, thereby reducing the overall processing delay.

[0069] [program] The program in Embodiment 1 can be any program that causes a computer to execute steps A1 to A16 shown in Figure 4. By installing and running this program on a computer, the edge-side information processing device and information processing method in Embodiment 1 can be realized. In this case, the computer's processor functions as a first image processing unit 11, a second image processing unit 12, a first estimation unit 13, a distribution unit 14, and a fourth image processing unit 15, and performs processing.

[0070] Furthermore, the program in Embodiment 1 may be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as one of the following: the first image processing unit 11, the second image processing unit 12, the first estimation unit 13, the distribution unit 14, or the fourth image processing unit 15.

[0071] (Embodiment 2) In Embodiment 2, the edge-side information processing device, provided in the system that realizes edge computing, executes predetermined processing in each of the multiple processing periods.

[0072] If the edge-side information processing device estimates that it cannot complete a predetermined process within the current processing period, it will execute the remaining processes that could not be performed within the current processing period using both the edge-side information processing device and the cloud-side information processing device in a period following the current processing period.

[0073] Specifically, the edge-side information processing device estimates whether there is sufficient time to perform the remaining processing in each of the one or more processing periods following the current processing period.

[0074] Next, if the edge-side information processing device estimates that there is sufficient buffer time, it will use that buffer time to execute the remaining processing. Conversely, if it estimates that there is no buffer time, it will prompt the cloud-side information processing device to execute the remaining processing early.

[0075] Thus, in Embodiment 2, processing is divided between the information processing device on the edge side and the information processing device on the cloud side, and processing results corresponding to the remaining processing are obtained from both. This allows the remaining processing that would have been executed within the processing period to be completed earlier, thus reducing processing delays compared to conventional methods.

[0076] [System Configuration] Embodiment 2 will be described in detail using Figure 5. Figure 5 is a diagram illustrating an example of the system of Embodiment 2.

[0077] System 500 is a system that realizes edge computing. System 500 in Figure 5 has an information processing device 10a located on the edge side, an information processing device 20a located on the cloud side, and a network 30. In addition, information processing devices 10a and 20a communicate with each other via the network 30.

[0078] Note that the hardware descriptions for the information processing device 10a, information processing device 20a, and network 30 are the same as those for the information processing device 10, information processing device 20, and network 30 in Figure 1, so the descriptions are omitted.

[0079] ●The information processing device 10a on the edge side will be described below. As shown in Figure 5, the information processing device 10a includes a first image processing unit 11, a second image processing unit 12, a first estimation unit 13, a second estimation unit 51, a distribution unit 52, and a fifth image processing unit 53.

[0080] The first image processing unit 11, the second image processing unit 12, and the first estimation unit 13 have already been described in Embodiment 1, so they will be omitted here.

[0081] If the second estimation unit 51 estimates that the second image processing cannot be completed in the current processing period T0_1, it estimates whether there is sufficient time to perform the remaining second image processing in the current processing period based on the changes between the first image acquired in a processing period after the current processing period T1_0 and the second image acquired in a processing period before that processing period.

[0082] The difference between the first and second images is the difference between the pixels of the first image and the pixels of the second image. The pixel difference Sub can be calculated, for example, as shown in Equation 1, by letting W be the width and H be the height of the images, p1 be the pixel position (x,y) of the first image, and p2 be the pixel position (x,y) of the second image, and it can be obtained by the following formula. Note that p1 and p2 can be represented as 24-bit values ​​(32-bit values ​​encompassing each of the RGB channels represented by 8 bits), or normalized floating-point 16-bit and 32-bit values, respectively.

[0083]

number

[0084] Whether or not there is a buffer period is estimated to exist if the pixel difference Sub is less than or equal to a pre-set difference threshold Tha (Sub ≤ Tha). In other words, the larger the pixel difference, the greater the change between images, so the second image processing will increase accordingly, and the buffer period can be expected to be relatively shorter. Conversely, the smaller the difference, the smaller the change between images, so the time required for the second image processing will be shorter, and the buffer period can be expected to be relatively longer.

[0085] The differential threshold Tha can be determined in advance through experiments, simulations, or other means.

[0086] If the distribution unit 52 estimates that there is no slack time, it distributes the remaining first information to be used in the second image processing to the third image processing unit 21. The third image processing unit 21 then uses the distributed first information to perform the second image processing.

[0087] Conversely, if it is estimated that there is a buffer period, the second image processing unit 12 uses the first information to be used in the remaining second image processing during the buffer period to perform the second image processing.

[0088] The fifth image processing unit 53 acquires the second information generated by the second image processing unit 12 and the third image processing unit 21, and uses the acquired second information to perform the third image processing.

[0089] ●The information processing device 20a on the cloud side will be explained. As shown in Figure 5, the information processing device 20a has the third image processing device 21 described above. In the example in Figure 5, the fifth image processing device 53 is provided in the edge-side information processing device 10a, but it may also be provided in the cloud-side information processing device 20a. The third image processing device 21 has already been described in Embodiment 1, so it will be omitted here.

[0090] [System Operation] Figure 6 is a diagram illustrating an example of the operation of the system of Embodiment 2. Note that period T2 in C of Figure 6 has five processing periods T1_0, T1_1, T1_2, T1_3, and T1_4. However, the number of processing periods is not limited to five.

[0091] Furthermore, Figure 6C shows an example in which the first and second image processing are completed within processing periods T1_0, T1_1, T1_2, T1_3, and T1_4, and the third image processing is completed in processing period T1_4.

[0092] As shown in Figure 6A, during processing period T1_0, the second image processing may not be completed within processing period T1_0, resulting in an overflow of processing time. In such cases, during period Ts shown in Figure 6C, the first estimation unit 13 estimates whether the second image processing can be completed within the current processing period T1_0 based on the time Ti1 required for the first image processing and the first information.

[0093] Next, if it is estimated that the second image processing cannot be completed in the current processing period T1_0, the second estimation unit 51 estimates whether there is a buffer period in the processing period T1_1 that follows the current processing period T1_0.

[0094] Specifically, the second estimation unit 51 estimates whether there is sufficient time in the processing period T1_1 to perform the remaining second image processing, based on the changes between the first image acquired in processing period T1_1, which is after the current processing period T1_0, and the second image acquired in processing period T1_0, which is before the processing period T1_1.

[0095] In Figure 6C, since it was estimated that there was a buffer period in processing period T1_1, the second image processing (overflow processing) is performed in buffer period To1 using the first information that will be used in the remaining second image processing.

[0096] Next, in Figure 6C, since it is estimated that there is no buffer time in the current processing period T1_2, the distribution unit 14 distributes the first information that has not been input to the second image processing from the first information output during processing periods T1_0, T1_1, and T1_2 to the third image processing unit 21.

[0097] Subsequently, as shown in Figure 6B, during period Toa, the third image processing unit 21 performs a second image processing using the first information that was not used during the buffer period To1.

[0098] Next, the fifth image processing unit 53 acquires the second information generated by both the second image processing unit 12 and the third image processing unit 21. In the case of C in Figure 6, the fifth image processing unit 53 uses the second information generated by the second image processing unit 12 and the third image processing unit 21 to perform the third image processing. [Device operation] The operation of the edge-side information processing device 10a in Embodiment 2 will be explained with reference to Figure 7. Figure 7 is a diagram illustrating an example of the operation of the edge-side information processing device in Embodiment 2. In the following explanation, the diagram will be referred to as appropriate. In Embodiment 2, the information processing method is implemented by operating the information processing device 10a. Therefore, the explanation of the information processing method in Embodiment 2 will be replaced by the following explanation of the operation of the information processing device.

[0099] As shown in Figure 7, the first image processing unit 11 acquires images in a time series for each processing period T1 (step B1). Next, the first image processing unit 11 performs the first image processing to generate the first information and stores the generated first information in memory (step B2). In step B2, the time required for the first image processing (period Ti1) is also measured.

[0100] Next, the first estimation unit 13 estimates whether the second image processing can be completed within the processing period (step B3) based on the time required for the first image processing (period Ti1) and the first information generated in the first image processing.

[0101] Specifically, in step B3, the first estimation unit 13 estimates the time (period Ti2) that can be used for the second image processing during processing period T1, and estimates whether the second image processing can be completed using the first information within the estimated period Ti2.

[0102] Next, if it is estimated that the second image processing can be completed during processing period T1 (Step B4: Yes), the second estimation unit 51 estimates whether there is sufficient time during that processing period to perform the remaining second image processing (Step B5).

[0103] Specifically, in a case like C in Figure 6, in step B5, it is estimated whether or not there is a buffer period To1 in processing period T1_1 based on the change between the first image acquired in processing period T1_1 and the second image acquired in processing period T1_0 prior to processing period T1_1.

[0104] If the distribution unit 14 estimates that the second image processing cannot be completed within the current processing period T1_0 (step B4: No), it proceeds to the processing in step B7.

[0105] Next, if it is estimated that there is sufficient time during the processing period to perform the remaining second image processing (Step B6: Yes), the process proceeds to Step B8.

[0106] Furthermore, if it is estimated that there is insufficient time to perform the remaining second image processing during the processing period (Step B6: No), the distribution unit 52 distributes the first information to be used in the remaining second image processing to the third image processing unit 21 (Step B7). The third image processing unit 21 then performs the second image processing using the distributed first information.

[0107] Next, the second image processing unit 12, during processing period T1, performs the second image processing using the remaining information from the first image processing unit that will not be used in the second image processing unit, generates the second information, and stores the generated second information in memory (step B8).

[0108] Next, if there is a buffer period (Step B6: Yes) and there are remaining second image processing steps (Step B9: Yes), the second image processing unit 12 performs the second image processing step during the buffer period using the first information to be used in the remaining second image processing steps (Step B10).

[0109] Next, if the second image processing unit 12 completes the remaining second image processing during the buffer period (Step B11: Yes), it stores the second information corresponding to the remaining second image processing during the buffer period in memory (Step B12). If the remaining first and second image processing during the buffer period has not been completed (Step B11: No), the process proceeds to Step B1.

[0110] Next, if the fifth image processing unit 53 has obtained second information (the result of performing second image processing on the distributed first information) from the third image processing unit 21 (step B13: Yes), it stores the obtained second information in memory (step B14).

[0111] Furthermore, if the fifth image processing unit 53 has not obtained the second information from the third image processing unit 21 (step B13: No), the process proceeds to step B15.

[0112] Next, in each processing period T1 of period T3, if all second image processing is completed or distributed (Step B15: Yes) and there are any incomplete second image processing (Step B16: Yes), the system waits for the completion of the third image processing and obtains the second information (Step B17). After that, the fifth image processing unit 53 uses all the second information generated in each processing period T3 to perform the third image processing (Step B18).

[0113] Furthermore, if, in each processing period of period T3, all second image processing is completed or distributed (Step B15: Yes) and there is no incomplete second image processing (Step B16: No), the fifth image processing unit 53 performs third image processing using all the second information generated in each processing period of period T3 (Step B18).

[0114] Furthermore, if, in each processing period T1 of period T3, all of the second image processing is not completed or distributed (step B15: No), the fifth image processing unit 53 proceeds to the processing of step B1.

[0115] In Embodiment 2, the processes described in steps B1 to B18 above are repeatedly executed. [Effects of Embodiment 2] As described above, according to Embodiment 2, the processing is divided between the second image processing unit 12 and the third image processing unit 21, and the second information generated by the remaining second image processing is obtained from both. This allows the remaining second image processing, which would have been performed within the processing period, to be completed earlier, thus reducing processing delay compared to the conventional method.

[0116] [program] The program in Embodiment 2 can be any program that causes a computer to execute steps B1 to B18 shown in Figure 7. By installing and running this program on a computer, the edge-side information processing device and information processing method in Embodiment 1 can be realized. In this case, the computer's processor functions as a first image processing unit 11, a second image processing unit 12, a first estimation unit 13, a second estimation unit 51, a distribution unit 52, and a fifth image processing unit 53, and performs processing.

[0117] Furthermore, the program in Embodiment 2 may be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as one of the following: the first image processing unit 11, the second image processing unit 12, the first estimation unit 13, the second estimation unit 51, the distribution unit 52, or the fifth image processing unit 53.

[0118] (Example 1) Example 1 will be described in detail using Figure 8. Figure 8 is a diagram illustrating an example of the system in Example 1.

[0119] System 800 is a system that realizes edge computing. System 800 in Figure 8 has an information processing device 10b located on the edge side, an information processing device 20b located on the cloud side, and a network 30. Information processing devices 10b and 20b communicate with each other via the network 30.

[0120] Note that the hardware descriptions for the information processing device 10b, information processing device 20b, and network 30 are the same as those for the information processing device 10, information processing device 20, and network 30 in Figure 1, so the descriptions are omitted.

[0121] ●The information processing device 10b on the edge side will be described below. As shown in Figure 8, the information processing device 10b includes a first image processing unit 11, a second image processing unit 12, a first estimation unit 13, a distribution unit 14, and a fourth image processing unit 15.

[0122] The first image processing unit 11 includes a detection unit 81 and a generation unit 82. The second image processing unit 12 includes a feature extraction unit 83. The fourth image processing unit 15 includes an action recognition unit 84.

[0123] The detection unit 81 performs image processing to detect object images corresponding to one or more objects captured in the image. For example, the detection unit 81 detects object identification information that identifies the detected object image and object position information that represents the position of the object image on the image from the image. It estimates the object category (class) and a rectangle (bounding box) that includes the position containing the object.

[0124] Object identification information is information that represents, for example, the category (class) of the object (tracking ID). Object position information is information that represents the bounding box containing the position of the object.

[0125] The generation unit 82 associates image identification information, object identification information, and object position information for each processing period, and generates first information. First information is generated for each image.

[0126] The feature extraction unit 83 performs a feature extraction process to extract features from each object image and generates second information. This second information is, for example, a value in a multidimensional vector space, i.e., a multidimensional vector quantity.

[0127] The action recognition unit 84 acquires the second information from either the second image processing unit 12 or the third image processing unit 21, whichever comes first, and uses the second information to perform action recognition processing to generate the third information. Action recognition processing is, for example, processing to recognize the actions of an object. Specifically, if the object is a person, it recognizes actions such as walking or dancing.

[0128] ●We will now explain the information processing device 20b on the cloud side. As shown in Figure 8, the information processing device 20b has the third image processing unit 21 described above. The third image processing unit 21 has the feature extraction unit 83 described above.

[0129] In the example shown in Figure 8, the fourth image processing unit 15 is located on the edge-side information processing unit 10b, but it may also be located on the cloud-side information processing unit 20b.

[0130] (Example 2) Example 2 will be described in detail using Figure 9. Figure 9 is a diagram illustrating an example of the system in Example 2.

[0131] System 900 is a system that realizes edge computing. System 900 in Figure 9 has an information processing device 10c located on the edge side, an information processing device 20c located on the cloud side, and a network 30. The information processing device 10c and the information processing device 20c communicate with each other via the network 30.

[0132] Note that the hardware descriptions for the information processing unit 10c, information processing unit 20c, and network 30 are the same as those for the information processing unit 10, information processing unit 20, and network 30 in Figure 1, so the descriptions are omitted.

[0133] ●The information processing device 10c on the edge side will be explained. As shown in Figure 9, the information processing device 10c includes a first image processing unit 11, a second image processing unit 12, a first estimation unit 13, a distribution unit 52, and a fifth image processing unit 53.

[0134] The first image processing unit 11 includes a detection unit 81 and a generation unit 82. The second image processing unit 12 includes a feature extraction unit 83. The fifth image processing unit 53 includes an action recognition unit 84. Note that the detection unit 81, generation unit 82, feature extraction unit 83, and action recognition unit 84 were explained in Example 1, so their explanations are omitted here.

[0135] I will now explain the second estimation section. In the second estimation unit 51 of Example 2, for example, if it is estimated that the second image processing cannot be completed in the current processing period T0_1, it may estimate whether there is a buffer period in the next processing period based on the difference between the number of object images in the first image and the number of object images in the second image.

[0136] Specifically, if the difference Sub2 between the number of object images in the first image and the number of object images in the second image is less than or equal to a predetermined threshold Thb (Sub2 ≤ Thb), then a margin period is estimated. The difference threshold Thb can be determined through experiments, simulations, or other methods.

[0137] Furthermore, in the second estimation unit 51 of Example 2, for example, if it is estimated that the second image processing cannot be completed in the current processing period T0_1, it may estimate in the next processing period whether there is a buffer period based on the inter-frame movement amount of the object image in the first image that has moved in the second image.

[0138] Specifically, for each of the object images (one or more targets), the inter-frame movement amount (Mov) of the object image in the first image to the second image is calculated, and the calculated inter-frame movement amounts (Mov) are summed up. Next, if the summed inter-frame movement amount (MovTol) is less than or equal to a predetermined threshold (Thc) (MovTol ≤ Thb), it is estimated that there is a margin period. The difference threshold (Thc) can be determined through experiments, simulations, etc.

[0139] The inter-frame movement amount (Mov) represents, for example, the sum or average distance that the center position (center point) of the bounding box of an object image with the same object identification information has moved between the first image and the second image.

[0140] As mentioned above, if the number of object images (objects) corresponding to the target object changes little, and the movement (amount of positional movement within the camera) of each object is also small, there is a high probability that the change between the image in the next processing period and the image in the current processing period will be small. In this case, for example, the second image processing can be omitted and the past results can be used as a substitute, and it can be estimated that the time for the second image processing (feature extraction) in the next processing period will be reduced. Therefore, it can be determined that there is sufficient time to perform the second image processing (feature extraction) that was overflowed in the current processing period.

[0141] Conversely, if the number of objects changes significantly and each object exhibits rapid movement, there is a high probability that the image changes between the current processing period and the next processing period will be substantial. Therefore, it can be estimated that the time required for the second image processing (feature extraction) in the next processing period will be large, or cannot be reduced. Consequently, it can be concluded that there is insufficient time to perform the second image processing (feature extraction) that was overflowing during the current processing period.

[0142] ●This section explains the information processing device 20c on the cloud side. As shown in Figure 9, the information processing device 20c has the third image processing unit 21 described above. The third image processing unit 21 has the feature extraction unit 83 described above.

[0143] In the example shown in Figure 9, the fifth image processing unit 53 is located in the edge-side information processing unit 10c, but it may also be located in the cloud-side information processing unit 20c.

[0144] [Physical configuration] Here, we will describe a computer that implements an information processing device by executing the programs in Embodiments 1 and 2 and Examples 1 and 2, using Figure 10. Figure 10 is a diagram illustrating an example of a computer that implements an information processing device in Embodiments 1 and 2 and Examples 1 and 2.

[0145] As shown in Figure 10, the computer 110 comprises a CPU 111, main memory 112, storage device 113, input interface 114, display controller 115, data reader / writer 116, and communication interface 117. Each of these components is connected to each other via a bus 121 to enable data communication. In addition to the CPU 111, or in place of the CPU 111, the computer 110 may also include a GPU or FPGA.

[0146] The CPU 111 loads the programs (codes) from Embodiments 1 and 2 and Examples 1 and 2, stored in the storage device 113, into the main memory 112 and performs various calculations by executing them in a predetermined order. The main memory 112 is typically a volatile storage device such as DRAM (Dynamic Random Access Memory). The programs from Embodiments 1 and 2 and Examples 1 and 2 are provided stored in a computer-readable recording medium 120. The programs from Embodiments 1 and 2 and Examples 1 and 2 may also be distributed over the internet connected via the communication interface 117. The recording medium 120 is a non-volatile recording medium.

[0147] Specific examples of the storage device 113 include hard disk drives and semiconductor storage devices such as flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

[0148] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0149] Specific examples of the recording medium 120 include general-purpose semiconductor memory devices such as CF (Compact Flash®) and SD (Secure Digital), magnetic recording media such as Flexible Disks, and optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0150] Furthermore, the information processing devices in Embodiments 1 and 2, and Examples 1 and 2, can also be implemented using hardware corresponding to each part, rather than a computer on which the program is installed. In addition, the information processing device may be partially implemented by a program and the remaining part by hardware.

[0151] Although the invention has been described above with reference to Embodiments 1 and 2, the invention is not limited to the embodiments described above. Various modifications to the structure and details of the invention can be made that will be understood by those skilled in the art within the scope of the invention. [Industrial applicability]

[0152] As described above, if it is estimated that edge-side processing cannot be completed within a predetermined period, the remaining unprocessed work can be distributed to the cloud side based on the processing content, thereby reducing processing delays. This is also useful in the field of edge computing. [Explanation of symbols]

[0153] 10, 10a, 10b, 10c Information Processing Device 11 First Image Processing Unit 12 Second Image Processing Unit 13 First Estimation Section 14 Distribution section 15. Fourth Image Processing Unit 20, 20a, 20b, 20c Information Processing Device 21 Third Image Processing Unit 30 Networks 51 Second Estimation Section 52 Distribution section 53 Fifth Image Processing Unit 81 Detection unit 82 Generation part 83 Feature Extraction Unit 84 Behavior Recognition Department 110 Computer 111 CPU 112 Main Memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Readers / Writers 117 Communication Interface 118 Input devices 119 Display device 120 recording media 121 Bus

Claims

1. A first image processing means that performs a first image processing on images acquired at predetermined processing intervals and generates first information, During the aforementioned processing period, a second image processing means is provided that performs a second image processing using the first information to generate the second information, A first estimation means that estimates whether the second image processing can be completed within the current processing period, based on the time required for the first image processing and the first information, If it is estimated that the second image processing cannot be completed within the current processing period, a distribution means distributes the first information used in the remaining second image processing to the second image processing means and a third image processing means, which is provided separately from the second image processing means and performs the second image processing, in order to perform the remaining second image processing in a predetermined number of processing periods after the current processing period. An information processing device having

2. The distribution means distributes all of the first information used in the remaining second image processing to the second image processing means and the third image processing means over a predetermined number of processing periods following the current processing period. The second image processing means, if there is a buffer period in a predetermined number of processing periods after the present time during which the remaining second image processing can be performed, uses the first information to be used in the remaining second image processing to perform the second image processing during the buffer period. The information processing apparatus according to claim 1.

3. Furthermore, the system includes a fourth image processing means that, from the second image processing means and the third image processing means, acquires the second information generated by the remaining second image processing from whichever of the two completes the remaining second image processing first, and uses the acquired second information to perform the third image processing. The information processing apparatus according to claim 2.

4. Furthermore, if it is estimated that the second image processing cannot be completed within the current processing period, the system has a second estimation means that estimates whether there is sufficient time within the processing period to perform the remaining second image processing, based on the changes between the first image acquired in a processing period after the current processing period and the second image acquired in a processing period before that processing period. The information processing apparatus according to claim 1.

5. If the second image processing means estimates that there is a buffer period, it will perform the second image processing using the first information to be used in the remaining second image processing during the buffer period. If the distribution means is estimated to have no available margin period, it distributes the first information to be used in the remaining second image processing to the third image processing means. The information processing apparatus according to claim 4.

6. The third image processing means performs the second image processing using the distributed first information. The information processing apparatus according to claim 5.

7. Furthermore, the system includes a fifth image processing means that acquires the second information generated by the second image processing means and the third image processing means, and performs a third image processing using the acquired second information. The information processing apparatus according to claim 6.

8. The second estimation means estimates whether or not the margin period exists based on the difference between the pixels of the first image and the pixels of the second image. The information processing apparatus according to claim 4.

9. Information processing device, The first image processing is performed on the images acquired at predetermined processing intervals to generate the first information. During the aforementioned processing period, a second image processing step is performed using the first information to generate the second information. Based on the time required for the first image processing and the first information, it is estimated whether the second image processing can be completed within the current processing period. If it is estimated that the second image processing cannot be completed within the current processing period, the first information used in the remaining second image processing is distributed to the information processing device and to other information processing devices that perform the second image processing, which are provided separately from the information processing device, in order to perform the remaining second image processing over a predetermined number of processing periods after the current processing period. Information processing methods.

10. On the computer, The first image processing is performed on the images acquired at predetermined processing intervals to generate the first information. During the aforementioned processing period, a second image processing is performed using the first information to generate the second information. Based on the time required for the first image processing and the first information, it is estimated whether the second image processing can be completed within the current processing period. If it is estimated that the second image processing cannot be completed within the current processing period, the first information used in the remaining second image processing is distributed to the computer and another computer, separately provided to perform the second image processing, in order to execute the remaining second image processing over a predetermined number of processing periods after the current processing period. A program that contains instructions.

Images