Image processing device, image processing system, image processing method, and program

Abstract

To change a frame rate of a video image transmitted to an analysis apparatus in accordance with a performance of the analysis apparatus that analyzes the video image.SOLUTION: An image processing device includes: a video image acquisition part configured to acquire a captured video image including a plurality of frame images continuous in time series; a transmission part configured to transmit the acquired captured video image to an analysis apparatus at a predetermined transmission frame rate; an analysis result acquisition part configured to acquire an analysis result of the captured video image from the analysis apparatus; a calculation part configured to calculate, using the analysis result, a staying number that is the number of frame images for which an analysis has not been completed within a predetermined time; and a frame rate changing part configured to change a frame rate of the captured video image transmitted by the transmitting means so that the staying number falls within a predetermined range. The transmission part transmits the captured video image to the analysis apparatus at the transmission frame rate changed by the frame rate changing part.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing system, an image processing method, and a program. [Background technology]

[0002] A technology that uses AI (Artificial Intelligence) to analyze video is known. An analysis device that employs this technology is used to analyze video captured by a camera. The frame rate of the video captured by the camera is not necessarily a frame rate that can be analyzed by the analysis device. In such cases, a process is required to change the frame rate of the captured video to a frame rate that can be analyzed by the analysis device.

[0003] The aggregation device described in Patent Document 1 inquires of an administrator about a minimum required rate, which is the minimum frame rate required for an analysis device to perform analysis. The aggregation device also transmits video that satisfies the minimum required rate to the analysis device. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 082864 Summary of the Invention [Problem to be solved by the invention]

[0005] The frame rate required for an analysis device to perform its expected functions is not a fixed value but varies depending on the video being analyzed. For example, the frame rate required for the analysis device to perform its functions to the fullest also depends on the number of analysis targets included in the captured video. This can make it complicated and difficult for an average user to change the frame rate to an optimal value depending on the video. Therefore, it is desirable to be able to change the frame rate of video captured by a capture device without user intervention.

[0006] The object of the present disclosure is to provide an image processing device, an image processing system, an image processing method, and a program that are capable of changing the frame rate of video sent to an analysis device in accordance with the performance of the analysis device that analyzes the video. [Means for solving the problem]

[0007] An image processing device according to one embodiment of the present disclosure includes a video acquisition unit that acquires a shot video composed of a plurality of successive frame images in chronological order, a transmission unit that transmits the acquired shot video to an analysis device at a predetermined transmission frame rate, an analysis result acquisition unit that acquires analysis results of the shot video from the analysis device, a calculation unit that uses the analysis results to calculate a number of remaining frame images that are not analyzed within a certain period of time, and a frame rate modification unit that modifies the transmission frame rate so that the number of remaining frame images falls within a predetermined range, and the transmission unit transmits the shot video to the analysis device at the transmission frame rate modified by the frame rate modification unit.

[0008] In one aspect of the image processing method of the present disclosure, a computer acquires a captured image composed of a plurality of successive frame images in chronological order, transmits the acquired captured image to an analysis device at a predetermined transmission frame rate, obtains an analysis result of the captured image from the analysis device, uses the analysis result to calculate a number of remaining frame images, which is the number of frame images for which analysis was not completed within a certain period of time, changes the transmission frame rate so that the number of remaining frame images falls within a predetermined range, and transmits the captured image to the analysis device at the changed transmission frame rate.

[0009] A program according to one aspect of the present disclosure causes a computer to perform the following processes: acquire a captured video composed of a plurality of consecutive frame images in chronological order; transmit the acquired captured video to an analysis device at a predetermined transmission frame rate; acquire an analysis result of the captured video from the analysis device; use the analysis result to calculate a number of remaining frames, which is the number of frame images for which analysis was not completed within a certain period of time; change the transmission frame rate so that the number of remaining frames falls within a predetermined range; and transmit the captured video to the analysis device at the changed transmission frame rate. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to provide an image processing device, an image processing system, an image processing method, and a program that are capable of changing the frame rate of video sent to an analysis device in accordance with the performance of the analysis device that analyzes the video. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a conceptual diagram illustrating an image processing system according to the present disclosure. [Figure 2] 1 is a block diagram illustrating an example of a configuration of an image processing device according to the present disclosure. [Figure 3] FIG. 10 is a conceptual diagram for explaining an example of a frame number in the present disclosure. [Figure 4] FIG. 10 is a diagram illustrating an example of display information in the present disclosure. [Figure 5] FIG. 10 is a diagram illustrating an example of display information in the present disclosure. [Figure 6] 10 is a flowchart illustrating an example of an operation of the image processing device according to the present disclosure. [Figure 7] 10 is a flowchart illustrating an example of an operation of the image processing device according to the present disclosure. [Figure 8] 1 is a block diagram illustrating an example of a configuration of an image processing system according to the present disclosure. [Figure 9]1 is a block diagram illustrating an example of a configuration of an image processing device according to the present disclosure. [Figure 10] 10 is a flowchart illustrating an example of an operation of the image processing device according to the present disclosure. [Figure 11] FIG. 2 is a block diagram showing an example of a hardware configuration for executing control and processing in each embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The embodiments described below are limited in a manner that is technically preferable for implementing the present disclosure, but the scope of the invention is not limited to the following. In all drawings used to describe the following embodiments, the same reference numerals are used for similar parts unless otherwise stated. In the following embodiments, repeated explanations of similar configurations and operations may be omitted. Furthermore, the UI (User Interface) shown in the drawings is an example and may be replaced with another UI.

[0013] (First embodiment) First, an image processing system according to a first embodiment will be described with reference to the drawings. When transmitting video captured by a camera to an analysis device, the image processing system according to this embodiment changes the frame rate of the video to be transmitted to the analysis device in accordance with the performance of the analysis device.

[0014] FIG. 1 is a conceptual diagram illustrating an image processing system according to the present disclosure. As shown in FIG. 1, the image processing system 1 includes an image processing device 10 and an analysis device 80. The image processing system 1 may also include a photographing device 70. The image processing device 10 is communicatively connected to both the photographing device 70 and the analysis device 80. The image processing device 10 is implemented by an information processing device such as a stationary terminal device, a laptop terminal device, or a portable terminal device. The photographing device 70 is a device that captures video of the surrounding area in which it is installed. For example, the photographing device 70 may be implemented by a surveillance camera, a drive recorder, or a camera mounted on a portable terminal device. The analysis device 80 may be implemented by an on-premise server or a cloud server. The analysis device 80 is equipped with an AI engine that performs image analysis. For example, the AI ​​engine may detect prohibited behavior from video. Prohibited behavior may include risky behavior or criminal activity. For example, the analysis target of the AI ​​engine may be a person, an animal, a vehicle, or the like. The types of AI engines and analysis targets described above are merely examples and are not limiting.

[0015] The frame rate at which imaging device 70 can capture images is predetermined for each imaging device 70. A frame rate recommended for analysis processing by analysis device 80 is also predetermined. In the following description, the frame rate recommended for analysis by analysis device 80 will also be referred to as the recommended frame rate. Generally, the higher the frame rate value, the higher the analysis accuracy of analysis device 80, but the higher the frame rate value, the more delay there will be in the analysis processing.

[0016] The imaging device 70 transmits captured video captured at a predetermined frame rate (capture frame rate) to the image processing device 10. The image processing device 10 transmits frame images constituting the received captured video to the analysis device 80. The analysis device 80 analyzes the images received from the image processing device 10 and transmits the analysis results of the frame images to the image processing device 10. The image processing device 10 changes the frame rate of the frame images transmitted to the analysis device 80 so that the number of frames remaining in the analysis device 80 falls within a predetermined range. The number of frames remaining is the number of frame images for which analysis has not been completed within a certain period of time. The predetermined range is a range that is equal to or greater than a predetermined minimum number of frames remaining and equal to or less than a predetermined maximum number of frames remaining. For example, if the number of frames remaining in the analysis device 80 is less than the predetermined minimum number of frames remaining, the image processing device 10 increases the frame rate of the video transmitted to the analysis device 80. Furthermore, if the number of residing images in analysis device 80 is smaller than a predetermined maximum number of residing images, image processing device 10 increases the frame rate of the video to be transmitted to analysis device 80. By image processing device 10 controlling the frame rate in this manner, the performance of analysis device 80 can be maximized.

[0017] (composition) 2 is a diagram showing an example of the configuration of an image processing device according to the present disclosure. The image processing device 10 includes a video acquisition unit 11, a transmission unit 12, an analysis result acquisition unit 13, a calculation unit 14, a frame rate change unit 15, a generation unit 16, a display unit 17, and a storage unit 18.

[0018] The video acquisition unit 11 acquires captured video composed of a plurality of frame images that are successive in time series. For example, the video acquisition unit 11 acquires captured video captured in real time by the camera device 70. The video acquisition unit 11 may acquire video captured in the past by the camera device 70 or video edited from video captured in the past by the camera device 70. The video captured in the past by the camera device 70 or video edited from video captured in the past by the camera device 70 is stored in, for example, the storage unit 18. In this case, the video acquisition unit 11 acquires from the storage unit 18 the video captured in the past by the camera device 70 or video edited from video captured in the past by the camera device 70.

[0019] The transmitting unit 12 transmits the acquired captured video to the analyzing device 80 at a predetermined transmission frame rate. The transmission frame rate is the frame rate at which the video is transmitted to the analyzing device 80. The transmission frame rate is different from the capturing frame rate, which is the frame rate at which the capturing device captures video. The predetermined transmission frame rate is a frame rate determined using the recommended frame rate of the analyzing device 80, or a frame rate determined by the frame rate changing unit 15.

[0020] The transmitting unit 12 determines the transmission frame rate for transmitting to the analysis device 80 as follows.

[0021] The value of the transmission frame rate when initially transmitting video, i.e., the initial value of the transmission frame rate, is determined using the recommended frame rate of the analysis device 80. First, the transmission unit 12 inquires of the analysis device 80 about the recommended frame rate. The recommended frame rate is a single value, such as "30 fps (Frames Per Second)." Alternatively, the recommended frame rate is specified as a range, such as "20 fps to 30 fps." If the recommended frame rate is a single value, the transmission unit 12 determines the initial value of the transmission frame rate to be the value of the recommended frame rate. If the recommended frame rate is specified as a range, for example, the transmission unit 12 sets the initial value of the transmission frame rate to the minimum value of the recommended frame rate. For example, if the recommended frame rate is "20 fps to 30 fps," the transmission unit 12 determines the minimum value, "20 fps," as the initial value of the transmission frame rate. Setting the initial value to the minimum value of the recommended frame rate is just an example; the median, average, or maximum value of the recommended frame rate may also be used as the initial value. Alternatively, the initial value of the transmission frame rate may be a value that does not depend on the recommended frame rate. For example, the initial value of the transmission frame rate may be 1.

[0022] When the transmitting unit 12 receives an instruction to change the transmission frame rate from the frame rate changing unit 15, the transmitting unit 12 determines the transmission frame rate determined by the frame rate changing unit 15 as the frame rate to be transmitted to the analyzing device 80.

[0023] After determining the transmission frame rate to be transmitted to analysis device 80, transmission unit 12 changes the transmission frame rate of the video acquired by video acquisition unit 11 to the determined transmission frame rate. Then, transmission unit 12 transmits the video with the changed frame rate to analysis device 80.

[0024] The transmitter 12 may assign a number to each of the frame images constituting the video before transmitting the video. In the following description, the number assigned to the frame image will also be referred to as a frame number. FIG. 3 is a conceptual diagram for explaining an example of frame numbers in the present disclosure. The transmitter 12 assigns a number to each of the consecutive frame images in time series. In the example of FIG. 3, the video transmitted to the analysis device 80 is composed of n frame images (n is a natural number). In this case, the transmitter 12 assigns a frame number to each frame image so that the numbers are in ascending order when the frame images are arranged in time series. For example, the transmitter 12 assigns frame number "1" to the first frame image F in time series and frame number "n" to the last frame image F in time series. Note that in the example of FIG. 3, the frame numbers are in ascending order in time series, but they may also be in descending order.

[0025] The analysis result acquisition unit 13 acquires the analysis results of the captured video from the analysis device 80. When the transmission unit 12 assigns a frame number and transmits the data to the analysis device 80, the analysis result acquisition unit 13 acquires the analysis results that associate the analysis results with the frame numbers.

[0026] The calculation unit 14 calculates the number of remaining frame images, which is the number of frame images for which analysis was not completed within a certain period of time, using the analysis results acquired from the analysis result acquisition unit 13. For example, the calculation unit 14 calculates the difference between the frame number of the frame image last transmitted by the transmission unit 12 and the frame number associated with the analysis result last acquired by the analysis result acquisition unit 13 as the number of remaining frame images. The calculation unit 14 may also calculate the difference between the number of frame images transmitted by the transmission unit 12 and the number of analysis results acquired by the analysis result acquisition unit 13 as the number of remaining frame images.

[0027] The frame rate change unit 15 changes the transmission frame rate so that the number of remaining packets falls within a predetermined range. A "minimum number of remaining packets" and a "maximum number of remaining packets" are determined in advance according to an allowable delay time. The frame rate change unit 15 changes the frame rate so that the number of remaining packets falls within a range between the minimum number of remaining packets and the maximum number of remaining packets.

[0028] If the retention number is less than the minimum retention number, the frame rate change unit 15 increases the value of the transmission frame rate. Below, a specific example of increasing the value of the transmission frame rate will be described. For example, it is assumed that the minimum retention number is set to 5. Furthermore, it is assumed that the transmission frame rate of the video transmitted by the transmission unit 12 is 10 fps, and the retention number calculated by the calculation unit 14 is 4. In this case, the frame rate change unit 15 changes the transmission frame rate from 10 fps to 11 fps, for example. In this specific example, the frame rate change unit 15 has been described as increasing the transmission frame rate by 1, but this is just an example, and the value by which the increase is made is not limited.

[0029] If the pooled number is greater than the maximum pooled number, the frame rate change unit 15 reduces the value of the transmission frame rate. A specific example of reducing the value of the transmission frame rate will be described below. For example, it is assumed that the maximum pooled number is set to 10. Furthermore, it is assumed that the transmission frame rate of the video transmitted by the transmitter 12 is 30 fps, and the pooled number calculated by the calculator 14 is 11. In this case, the frame rate change unit 15 changes the transmission frame rate from 30 fps to 29 fps, for example. In this specific example, the frame rate change unit 15 reduces the transmission frame rate by 1, but this is merely an example, and the value by which the frame rate is reduced is not limited.

[0030] The frame rate change unit 15 may change the transmission frame rate when the state in which the staying number is outside a predetermined range continues for a certain period of time. In other words, the frame rate change unit 15 may change the transmission frame rate when the staying number is less than the minimum staying number, rather than immediately changing the transmission frame rate. Similarly, the frame rate change unit 15 may change the transmission frame rate when the staying number is greater than the maximum staying number, rather than immediately changing the transmission frame rate when the staying number is greater than the maximum staying number, rather than immediately changing the transmission frame rate. This can reduce the processing cost of the image processing device 10.

[0031] The generation unit 16 generates display information including information indicating the transmission frame rate of the video to be transmitted to the analysis device 80. The display information generated by the generation unit 16 will be described with reference to Figs. 4 and 5. Figs. 4 and 5 are diagrams showing an example of the display information in the present disclosure. Note that the display information shown in Figs. 4 and 5 is an example and is not limited to this. The display information may include information other than that shown in Figs. 4 and 5.

[0032] Display information 100 shown in Fig. 4 includes areas 101 to 103. Area 101 displays information indicating a predetermined minimum and maximum number of staying messages. Area 102 displays information indicating the current number of staying messages. Area 103 displays information indicating the current transmission frame rate. In other words, the display information includes at least one of information indicating the predetermined minimum and maximum number of staying messages, information indicating the current number of staying messages, and information indicating the current transmission frame rate.

[0033] When the frame rate change unit 15 changes the transmission frame rate, the generation unit 16 may generate display information further including information indicating that the transmission frame rate has been changed. The display information 110 shown in FIG. 5 includes areas 104 to 106 in addition to areas 101 to 103. Area 104 displays a message notifying that the transmission frame rate has been changed. Area 105 displays information indicating the number of queued frames before the transmission frame rate was changed. Area 106 displays information indicating the frame rate before the transmission frame rate was changed. The generation unit 16 may also devise a display method to make it easier for the user to know that the transmission frame rate has been changed. For example, the generation unit 16 may highlight a message notifying that the transmission frame rate has been changed, as in area 104.

[0034] The display unit 17 displays the display information generated by the generation unit 16. Note that, in the present embodiment, the display information is described as being displayed on the display unit 17, but is not limited to this. For example, the display information generated by the generation unit 16 may be transmitted to an information processing terminal communicably connected to the image processing device 10 and displayed on a display unit included in the information processing terminal.

[0035] The storage unit 18 stores information required for processing by the image processing device 10. The storage unit 18 stores a minimum number of remaining images and a maximum number of remaining images. The storage unit 18 may store images previously captured by the image capture device 70, or images edited from images previously captured by the image capture device 70. The storage unit 18 may store information other than the above-mentioned information, such as information temporarily required for processing by the image processing device 10. For example, the storage unit 18 may store a recommended frame rate for the analysis device 80, the frame number of a frame image transmitted by the transmission unit 12, etc.

[0036] (operation) Next, the operation of the image processing device 10 in this embodiment will be described with reference to Figures 6 and 7. Figures 6 and 7 are flowcharts for explaining an example of the operation of the image processing device in the present disclosure.

[0037] [Processing at first transmission] 6, the flow of processing when transmitting video to analysis device 80 for the first time will be described. First, video acquisition unit 11 acquires captured video made up of a plurality of frame images that are consecutive in time series from imaging device 70 (step S101).

[0038] Next, the transmitting unit 12 inquires about the recommended frame rate from the analyzing device 80 (step S102). The transmitting unit 12 acquires the recommended frame rate transmitted from the analyzing device 80.

[0039] Next, the transmitting unit 12 uses the recommended frame rate to determine the transmission frame rate at which to transmit to the analysis device 80 (step S103).

[0040] Next, the transmitter 12 changes the frame rate of the video acquired in step S101 to the transmission frame rate determined in step S103 (step S104).

[0041] Next, the transmitting unit 12 transmits the video with the changed frame rate to the analyzing device 80 (step S105). The above is the processing of the image processing device 10 when transmitting video to the analyzing device 80 for the first time. After completing the series of processing shown in Fig. 6, the image processing device 10 performs the processing shown in Fig. 7.

[0042] [Processing other than the first transmission] Next, a process flow after the first transmission of video to analysis device 80 will be described with reference to Fig. 7. First, analysis result acquisition unit 13 acquires the analysis results of the video from analysis device 80 (step S106).

[0043] Next, the calculation unit 14 uses the analysis results acquired in step S106 to calculate the number of remaining frame images, which is the number of frame images for which analysis was not completed within a certain period of time (step S107). For example, the calculation unit 14 calculates the difference between the most recent frame number transmitted by the transmission unit 12 and the frame number associated with the analysis result acquired by the analysis result acquisition unit 13 as the remaining number. The calculation unit 14 may also calculate the difference between the number of frame images transmitted by the transmission unit 12 and the number of analysis results acquired by the analysis result acquisition unit 13 as the remaining number.

[0044] Next, the frame rate change unit 15 changes the transmission frame rate of the video to be transmitted to the analysis device so that the staying number calculated in step S107 falls within a predetermined range. First, the frame rate change unit 15 determines whether the staying number is within a range of a minimum staying value or more and a maximum staying number or less (step S108). If the staying number is within a range of a minimum staying value or more and a maximum staying number or less (Yes in step S108), the frame rate change unit 15 does not change the value of the frame rate and proceeds to the processing of step S110.

[0045] If the staying number is not within the range of the minimum staying number or more and the maximum staying number or less (No in step S108), the frame rate change unit 15 changes the value of the transmission frame rate (step S109). If the staying number is less than the minimum staying number, the frame rate change unit 15 increases the value of the transmission frame rate. Also, if the staying number is greater than the maximum staying number, the frame rate change unit 15 decreases the value of the transmission frame rate. Note that the frame rate change unit 15 may change the transmission frame rate if the staying number remains outside a predetermined range for a certain period of time. If the number of consecutive times it is determined in step S108 that the staying number is not within the range of the minimum staying value or more and the maximum staying number or less exceeds a predetermined number, the frame rate change unit 15 may change the value of the transmission frame rate in step S109.

[0046] Next, the transmitting unit 12 transmits the video acquired by the video acquiring unit 11 to the analyzing device 80 at a predetermined transmission frame rate (step S110). If the frame rate changing unit 15 did not change the transmission frame rate (Yes in step S108), the transmitting unit 12 transmits video at the same frame rate as the most recent transmission frame rate used by the transmitting unit 12 to the analyzing device 80. If the frame rate changing unit 15 changed the transmission frame rate (step S109), the transmitting unit 12 transmits video at the transmission frame rate determined by the frame rate changing unit 15 to the analyzing device 80.

[0047] Next, the generating unit 16 generates display information including information indicating the transmission frame rate (step S111).

[0048] Next, the display unit 17 displays the display information generated in step S111 (step S112).

[0049] Next, the image processing device 10 determines whether a predetermined termination condition is satisfied (step S113). For example, the termination condition is receiving a termination request from the user. The image processing device 10 repeats the processes of steps S106 to S112 at predetermined time intervals until the termination condition is satisfied (No in step S113). When the termination condition is satisfied (Yes in step S113), the image processing device 10 terminates the process.

[0050] As described above, the image processing device of this embodiment includes a video acquisition unit, a transmission unit, an analysis result acquisition unit, a calculation unit, and a frame rate change unit. The video acquisition unit acquires captured video composed of a plurality of frame images that are consecutive in time series. The transmission unit transmits the acquired captured video to the analysis device at a predetermined transmission frame rate. The transmission unit also transmits the captured video to the analysis device at the transmission frame rate changed by the frame rate change unit. The analysis result acquisition unit acquires the analysis results of the captured video from the analysis device. The calculation unit uses the analysis results to calculate the number of remaining frame images, which is the number of frame images whose analysis was not completed within a certain period of time. The frame rate change unit changes the transmission frame rate so that the number of remaining frame images falls within a predetermined range.

[0051] Generally, the higher the frame rate, the higher the analysis accuracy of the analysis device, but the higher the frame rate, the more delay occurs in the analysis process. Therefore, a frame rate that allows the analysis device to maximize its performance is one that maintains an acceptable delay in the analysis process and has the largest possible frame rate. The image processing device of this embodiment controls the frame rate of the video sent to the analysis device so that the number of frame images whose analysis has not been completed within a certain period of time, that is, the number of remaining frame images, falls within a predetermined range. In other words, the image processing device of this embodiment can change the frame rate of the video sent to the analysis device to match the performance of the analysis device that analyzes the video. This allows the analysis device to maximize its performance.

[0052] Furthermore, the image processing device of this embodiment can optimize the frame rate according to the number of analysis targets included in the captured video. As the number of analysis targets increases, the processing required by the analysis device increases, while as the number of analysis targets decreases, the processing required by the analysis device decreases. The processing required by the analysis device directly leads to delays in the analysis processing by the analysis device. Therefore, as the number of analysis targets increases, the number of remaining images also increases, so it is desirable to reduce the frame rate of the video transmitted to the analysis device. Furthermore, as the number of analysis targets decreases, the number of remaining images also decreases, so it is desirable to increase the frame rate of the video transmitted to the analysis device. Performing these processes manually requires labor. Furthermore, these tasks are complex and may pose a barrier to general users. The image processing device of this embodiment automatically controls the frame rate of the video transmitted to the analysis device, thereby resolving these issues. In other words, the image processing device of this embodiment can change the frame rate of the video transmitted to the analysis device to match the performance of the analysis device analyzing the video. This maximizes the performance of the analysis device.

[0053] The image processing device according to one aspect of the present embodiment further includes a generation unit that generates display information including information indicating the transmission frame rate of the captured video to be transmitted to the analysis device. The image processing device according to one aspect of the present embodiment further includes a display unit that displays the display information generated by the generation unit. This allows a user checking the display unit of the image processing device to understand the frame rate to be transmitted to the analysis device.

[0054] In an image processing device according to one aspect of the present embodiment, the predetermined range is a range equal to or greater than a predetermined minimum number of staying images and equal to or less than a predetermined maximum number of staying images. The display information further includes at least one of information indicating the predetermined minimum number of staying images, information indicating the predetermined maximum number of staying images, and information indicating the number of staying images calculated by the calculation unit. This allows a user checking the display unit of the image processing device to understand at least one of the predetermined minimum number of staying images, the predetermined maximum number of staying images, and the number of staying images calculated by the calculation unit.

[0055] In the image processing device according to one aspect of the present embodiment, when the frame rate changing unit changes the transmission frame rate, the generating unit generates display information further including information indicating that the transmission frame rate has been changed, thereby allowing a user checking the display unit of the image processing device to understand that the frame rate of the video to be transmitted to the analyzing device has been changed.

[0056] In an image processing device according to an aspect of this embodiment, the predetermined range is a range equal to or greater than a predetermined minimum number of queued images and equal to or less than a predetermined maximum number of queued images. In the image processing device according to an aspect of this embodiment, the frame rate change unit increases the frame rate when the queued image number is less than the minimum number of queued images. Generally, the higher the frame rate, the higher the analysis accuracy of the analysis device. In the image processing device according to an aspect of this embodiment, by increasing the frame rate when the queued image number is less than the minimum number of queued images, the analysis accuracy of the analysis device can be improved. In addition, in the image processing device according to an aspect of this embodiment, the frame rate change unit decreases the frame rate when the queued image number is greater than the maximum number of queued images. Generally, the larger the frame rate, the more delay occurs in the analysis process. In the image processing device according to an aspect of this embodiment, by decreasing the frame rate when the queued image number is greater than the maximum number of queued images, the delay in the analysis device can be reduced. In other words, the image processing device according to an aspect of this embodiment automatically determines a frame rate that allows the delay in the analysis process to be within an acceptable range and has the largest possible frame rate. In other words, according to the image processing device of one aspect of the present embodiment, the frame rate of the video transmitted to the analysis device can be changed in accordance with the performance of the analysis device that analyzes the video, thereby maximizing the performance of the analysis device.

[0057] In an image processing device according to one aspect of the present embodiment, the transmission unit assigns a frame number to each frame image constituting the video to be transmitted in ascending chronological order and transmits the assigned frame number. Also, in an image processing device according to one aspect of the present embodiment, the analysis result acquisition unit acquires analysis results in which the analysis results are associated with frame numbers. Also, in one aspect of the present embodiment, the calculation unit calculates the difference between the frame number of the frame image last transmitted by the transmission unit and the frame number associated with the analysis result last acquired by the analysis result acquisition unit as the number of remaining images. Some AI engines installed in the analysis device transmit analysis results for all received frame images, while others transmit analysis results in which some of the received frame images have been thinned out. According to the image processing device according to one aspect of the present embodiment, by assigning frame numbers to the frame images constituting the video and calculating the number of remaining images using the frame numbers, the number of remaining images can be calculated for any of the AI ​​engines described above.

[0058] In the image processing device according to one aspect of the present embodiment, the initial value of the transmission frame rate is determined using a recommended frame rate recommended for the analysis process of the analysis device, thereby minimizing the number of times the frame rate needs to be changed.

[0059] (Variation) In the present embodiment, the image processing device 10 and the analysis device 80 have been described as being realized by separate devices, but they may also be realized by the same device. FIG. 8 is a block diagram showing an example of the configuration of an image processing system according to the present disclosure. As shown in FIG. 8, the image processing device 10 may be realized as part of the functions of the analysis device 80-2. With such a configuration, there is no need to provide an intermediary device between the imaging device 70 and the analysis device 80-2.

[0060] (Second embodiment) In this embodiment, an image processing device 20 will be described, which has a simplified configuration of the image processing device in the first embodiment. In the following description, the same parts as in the first embodiment will be omitted as appropriate.

[0061] (composition) Next, the configuration of the image processing device in this embodiment will be described with reference to the drawings. Fig. 9 is a block diagram showing an example of the configuration of the image processing device in the present disclosure. The image processing device 20 includes a video acquisition unit 21, a transmission unit 22, an analysis result acquisition unit 23, a calculation unit 24, and a frame rate change unit 25.

[0062] The video acquisition unit 21 acquires a shot video composed of a plurality of frame images that are consecutive in time series. The transmission unit 22 transmits the acquired shot video to the analysis device at a predetermined transmission frame rate. The transmission unit 22 also transmits the shot video to the analysis device at a transmission frame rate changed by the frame rate change unit 25. The analysis result acquisition unit 23 acquires the analysis results of the shot video from the analysis device. The calculation unit 24 uses the analysis results to calculate the number of remaining frame images, which is the number of frame images whose analysis was not completed within a certain time. The frame rate change unit 25 changes the transmission frame rate so that the number of remaining frame images falls within a predetermined range.

[0063] (operation) Next, an example of the operation of the image processing device in this embodiment will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of the operation of the image processing device in the present disclosure.

[0064] First, the video acquisition unit 21 acquires a shot video made up of a plurality of frame images that are successive in time series (step S21).

[0065] Next, the transmitting unit 22 transmits the acquired captured video to the analysis device at a predetermined transmission frame rate (step S22).

[0066] Next, the analysis result acquisition unit 23 acquires the analysis result of the captured video from the analysis device (step S23).

[0067] Next, the calculation unit 24 uses the analysis results to calculate the number of remaining frame images, which is the number of frame images for which analysis has not been completed within a certain period of time (step S24).

[0068] Next, the frame rate change unit 25 changes the transmission frame rate so that the number of remaining packets falls within a predetermined range (step S25).

[0069] Next, the transmitting unit 22 transmits the captured video to the analysis device at the transmission frame rate changed by the frame rate changing unit 25 (step S26).

[0070] The image processing device of this embodiment controls the frame rate of the video to be sent to the analysis device so that the number of remaining frame images, which is the number of frame images whose analysis has not been completed within a certain time, falls within a predetermined range. In other words, the image processing device of this embodiment can change the frame rate of the video to be sent to the analysis device in accordance with the performance of the analysis device that analyzes the video.

[0071] (Hardware configuration) The functions of each of the components in each of the embodiments of the present disclosure described above can be realized not only as hardware but also as a computer device or firmware under program control.

[0072] 11 is a diagram showing an example of a hardware configuration in which an image processing device according to the present disclosure is realized by a computer device 90 including a processor. The image processing device of each embodiment is realized by the computer device 90. As shown in FIG. 11, the computer device 90 includes a processor 91, a memory 92, a storage device 93 such as a hard disk for storing programs, an input / output interface 94 for connecting input devices and output devices, and a communication interface 95 for connecting to a network.

[0073] The processor 91 loads a program (instructions) stored in a storage device 93 or the like into the memory 92. For example, the program is a software program for executing the control and processing in the present disclosure. The processor 91 executes the program loaded into the memory 92. The processor 91 executes the program to execute the control and processing in the present disclosure.

[0074] The storage device 93 may be, for example, an optical disk, a flexible disk, a magneto-optical disk, an external hard disk, or a semiconductor memory. Some storage media in the storage device are non-volatile storage devices, and the programs are recorded therein. The programs may also be downloaded from an external computer (not shown) connected to a communication network.

[0075] The input device connected to the input / output interface 94 is realized by, for example, a mouse or a keyboard, and is used for input operations. Similarly, the output device connected to the input / output interface 94 is realized by, for example, a display, and is used for displaying and checking output results.

[0076] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0077] Some or all of the above embodiments can also be described as follows:

[0078] (Appendix 1) an image acquisition means for acquiring a captured image made up of a plurality of frame images that are successive in time series; a transmitting means for transmitting the acquired photographed video to an analyzing device at a predetermined transmission frame rate; an analysis result acquisition means for acquiring an analysis result of the captured video from the analysis device; a calculation means for calculating a remaining number, which is the number of frame images for which analysis has not been completed within a certain period of time, using the analysis results; a frame rate changing means for changing the transmission frame rate so that the number of remaining frames falls within a predetermined range; The transmission means transmits the captured video to the analysis device at the transmission frame rate changed by the frame rate change means. (Appendix 2) a generation means for generating display information including information indicating the transmission frame rate of the captured video to be transmitted to the analysis device; 2. The image processing device according to claim 1, further comprising: a display unit that displays the display information. (Appendix 3) the predetermined range is a range equal to or greater than a predetermined minimum number of staying packets and equal to or less than a predetermined maximum number of staying packets, The image processing device according to claim 2, wherein the display information further includes at least one of information indicating the minimum staying number, information indicating the maximum staying number, or information indicating the calculated staying number. (Appendix 4) When the transmission frame rate is changed by the frame rate change means, 4. The image processing device according to claim 3, wherein the generating means generates the display information further including information indicating that the transmission frame rate has been changed. (Appendix 5) the predetermined range is a range equal to or greater than a predetermined minimum number of staying packets and equal to or less than a predetermined maximum number of staying packets, The frame rate changing means If the number of remaining frames is less than the minimum number of remaining frames, the value of the transmission frame rate is increased; 2. The image processing device according to claim 1, wherein, when the number of remaining images is greater than the maximum number of remaining images, the value of the transmission frame rate is reduced. (Appendix 6) the transmitting means assigns a frame number to each of the frame images constituting the captured video to be transmitted in ascending order in chronological order, and transmits the frame numbers; the analysis result acquisition means acquires an analysis result in which the analysis result of the frame image is associated with a frame number; The image processing device described in Appendix 1, wherein the calculation means calculates the difference between the frame number of the frame image last transmitted by the transmission means and the frame number associated with the analysis result last acquired by the analysis result acquisition means as the number of remaining images. (Appendix 7) 2. The image processing device according to claim 1, wherein the initial value of the transmission frame rate is determined using a recommended frame rate recommended for the analysis process of the analysis device. (Appendix 8) An image processing device according to any one of Supplementary Notes 1 to 7; an analysis device that receives the captured video transmitted from the image processing device and transmits an analysis result of the captured video to the image processing device. (Appendix 9) The computer Acquire a captured video consisting of multiple frame images that are consecutive in time series, Transmitting the captured video to an analysis device at a predetermined transmission frame rate; acquiring an analysis result of the captured video from the analysis device; Using the analysis results, a retention number is calculated, which is the number of frame images whose analysis has not been completed within a certain time period. changing the transmission frame rate so that the number of remaining packets falls within a predetermined range; an image processing method for transmitting the captured video to the analysis device at the changed transmission frame rate; (Appendix 10) On the computer, A process of acquiring a captured video composed of a plurality of frame images that are successive in time series; a process of transmitting the acquired captured video to an analysis device at a predetermined transmission frame rate; A process of acquiring an analysis result of the captured video from the analysis device; A process of calculating a remaining number, which is the number of frame images whose analysis has not been completed within a certain time, using the analysis results; changing the transmission frame rate so that the number of remaining packets falls within a predetermined range; and transmitting the captured video to the analysis device at the changed transmission frame rate.

[0079] In addition, in the above appendices, some or all of the configurations described in appendices 2 to 7 that are dependent on appendices 1 may also be dependent on appendices 9 and 10 in the same dependent relationship as appendices 2 to 7. Furthermore, not limited to appendices 1, 9, and 10, some or all of the configurations described as appendices may be made dependent on various hardware, software, various recording means for recording software, or systems, within the scope of each of the above-mentioned embodiments. [Explanation of symbols]

[0080] 10, 20 Image processing device 11, 21 Video acquisition unit 12, 22 Transmitter 13, 23 Analysis result acquisition part 14, 24 Calculation section 15, 25 Frame rate change section 16 Generation part 17 Display 18 Memory section 70 Imaging equipment 80 Analyzer 90 Computer Equipment 91 processors 92 memory 93 Storage device 94 Input / Output Interface 95 Communication Interface

Claims

1. an image acquisition means for acquiring a captured image made up of a plurality of frame images that are successive in time series; a transmitting means for transmitting the acquired photographed video to an analyzing device at a predetermined transmission frame rate; an analysis result acquisition means for acquiring an analysis result of the captured video from the analysis device; a calculation means for calculating a remaining number, which is the number of frame images for which analysis has not been completed within a certain period of time, using the analysis results; a frame rate changing means for changing the transmission frame rate so that the number of remaining frames falls within a predetermined range; The transmission means transmits the captured video to the analysis device at the transmission frame rate changed by the frame rate change means.

2. a generation means for generating display information including information indicating the transmission frame rate of the captured video to be transmitted to the analysis device; The image processing device according to claim 1 , further comprising: a display unit that displays the display information.

3. the predetermined range is a range equal to or greater than a predetermined minimum number of staying packets and equal to or less than a predetermined maximum number of staying packets, The image processing device according to claim 2 , wherein the display information further includes at least one of information indicating the minimum staying number, information indicating the maximum staying number, and information indicating the calculated staying number.

4. When the transmission frame rate is changed by the frame rate change means, The image processing device according to claim 3 , wherein the generating means generates the display information further including information indicating that the transmission frame rate has been changed.

5. the predetermined range is a range equal to or greater than a predetermined minimum number of staying packets and equal to or less than a predetermined maximum number of staying packets, The frame rate changing means If the number of remaining frames is less than the minimum number of remaining frames, the value of the transmission frame rate is increased; The image processing device according to claim 1 , wherein when the number of remaining images is greater than the maximum number of remaining images, the value of the transmission frame rate is reduced.

6. the transmitting means assigns a frame number to each of the frame images constituting the captured video to be transmitted in ascending order in chronological order, and transmits the frame numbers; the analysis result acquisition means acquires an analysis result in which the analysis result of the frame image is associated with a frame number; 2. The image processing device according to claim 1, wherein the calculation means calculates the remaining number as the difference between the frame number of the frame image last transmitted by the transmission means and the frame number associated with the analysis result last acquired by the analysis result acquisition means.

7. The image processing device according to claim 1 , wherein the initial value of the transmission frame rate is determined using a recommended frame rate recommended for the analysis process of the analysis device.

8. An image processing device according to any one of claims 1 to 7; an analysis device that receives the captured video transmitted from the image processing device and transmits an analysis result of the captured video to the image processing device.

9. The computer Acquire a captured video consisting of multiple frame images that are consecutive in time series, Transmitting the captured video to an analysis device at a predetermined transmission frame rate; acquiring an analysis result of the captured video from the analysis device; Using the analysis results, a retention number is calculated, which is the number of frame images whose analysis has not been completed within a certain time period. changing the transmission frame rate so that the number of remaining packets falls within a predetermined range; an image processing method for transmitting the captured video to the analysis device at the changed transmission frame rate;

10. On the computer, A process of acquiring a captured video composed of a plurality of frame images that are successive in time series; a process of transmitting the acquired captured video to an analysis device at a predetermined transmission frame rate; A process of acquiring an analysis result of the captured video from the analysis device; A process of calculating a remaining number, which is the number of frame images whose analysis has not been completed within a certain time, using the analysis results; changing the transmission frame rate so that the number of remaining packets falls within a predetermined range; and transmitting the captured video to the analysis device at the changed transmission frame rate.

Images