Video transmission device, video transmission method and program

Abstract

To provide a video transmission device, a video transmission method and a program that properly utilize video data for inference processing when transmitting the video data.SOLUTION: In a video processing system, a video transmission device 200 comprises: an encoding part 221 which encodes respective division images, obtained by dividing a frame image constituting a video, respectively at bit rates for division images set by the division images; a transmission part 222 which transmits the encoded frame images through a transmission line; and an acquisition part 223 which acquires an estimated band as an estimated value of the band of the transmission line and inference accuracy by the division images in predetermined inference processing by the division images of the frame image, and the encoding part 221 sets the bit rates for the division images used for the encoding by the division images based upon the inference accuracy by the division images so that the frame image can be encoded to be transmitted within the estimated band of the transmission line.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present disclosure relates to a video transmission device, a video transmission method, and a program. [Background technology]

[0002] Video data is transmitted to a device equipped with AI (Artificial Intelligence) for the purpose of inferring from the video data using AI. In this case, in order to reduce the transmission cost of the video data, the video data is compressed during encoding to reduce the amount of code. For example, Patent Document 1 proposes a technology for encoding each region of an image at low or high compression using a predetermined inference result for the image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2022 / 079791 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when transmitting video data, depending on the state of the transmission path, it may not be possible to transmit video data with an appropriate code amount for inference, which may result in a case where appropriate inference cannot be performed. In other words, when transmitting video data, there is a problem in that the video data may not be appropriately used for inference processing.

[0005] Therefore, the object of the present disclosure is to solve the above-mentioned problem that when transmitting video data, there are cases where the video data cannot be appropriately used for inference processing. [Means for solving the problem]

[0006] A video transmission device according to an embodiment of the present disclosure includes: an encoding unit that encodes each divided image obtained by dividing a frame image that constitutes a video image at a bit rate for the divided image that is set for each divided image; a transmitting unit that transmits the encoded frame image via a transmission path; an acquisition unit that acquires an estimated bandwidth that is an estimated value of the bandwidth of the transmission path and an inference accuracy for each divided image when a predetermined inference process is performed for each divided image of the frame image; Equipped with the encoding unit sets a bit rate for the divided images to be used for encoding each of the divided images based on the inference accuracy for each of the divided images so that the frame images can be encoded so as to be transmittable within the estimated bandwidth of the transmission path. The structure is as follows. Furthermore, a video transmission method according to an embodiment of the present disclosure includes: Each divided image obtained by dividing the frame images that make up the video is encoded at a bit rate for the divided image that is set for each divided image, A video transmission method for transmitting the encoded frame images via a transmission path, comprising: an estimated bandwidth that is an estimated value of the bandwidth of the transmission path, and an inference accuracy for each divided image when a predetermined inference process is performed for each divided image of the frame image; a bit rate for each of the divided images to be used for encoding the divided images is set based on the inference accuracy for each of the divided images so that the frame images can be encoded so as to be transmittable within the estimated bandwidth of the transmission path; The structure is as follows. Furthermore, a video transmission method according to an embodiment of the present disclosure includes: Each divided image obtained by dividing the frame images that make up the video is encoded at a bit rate for the divided image that is set for each divided image, A video transmission method for transmitting the encoded frame images via a transmission path, comprising: an estimated bandwidth that is an estimated value of the bandwidth of the transmission path, and an inference accuracy for each divided image when a predetermined inference process is performed for each divided image of the frame image; a bit rate for each of the divided images to be used for encoding the divided images is set based on the inference accuracy for each of the divided images so that the frame images can be encoded so as to be transmittable within the estimated bandwidth of the transmission path; Have the computer perform the process, The structure is as follows. [Effects of the Invention]

[0007] With the above-described configuration, the present disclosure makes it possible to appropriately utilize video data for inference processing when transmitting the video data. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of a video processing system according to the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating image processing by a video processing system according to the present disclosure. [Figure 3] FIG. 1 is a diagram illustrating image processing by a video processing system according to the present disclosure. [Figure 4] FIG. 1 is a diagram illustrating image processing by a video processing system according to the present disclosure. [Figure 5] FIG. 1 is a diagram illustrating image processing by a video processing system according to the present disclosure. [Figure 6] 1 is a flowchart illustrating a processing operation of a video processing system according to the present disclosure. [Figure 7] 1 is a flowchart illustrating a processing operation of a video processing system according to the present disclosure. [Figure 8] 1 is a flowchart illustrating a processing operation of a video processing system according to the present disclosure. [Figure 9] 1 is a flowchart illustrating a processing operation of a video processing system according to the present disclosure. [Figure 10]1 is a flowchart illustrating a processing operation of a video processing system according to the present disclosure. [Figure 11] 1 is a flowchart illustrating a processing operation of a video processing system according to the present disclosure. [Figure 12] 1 is a block diagram showing a hardware configuration of a video transmission device according to the present disclosure. [Figure 13] 1 is a block diagram showing a configuration of a video transmission device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009] First Embodiment A first embodiment of the present disclosure will be described with reference to the drawings, which may be relevant to any embodiment.

[0010] As an example, the video processing system in this embodiment is used to detect people based on video from a surveillance camera and discover suspicious people. The video processing system is configured to include an encoding / transmission device 110 that encodes video data and transmits it over a transmission path, and a receiving device 120 that is connected to the encoding / transmission device 110 via a transmission path N and receives the encoded video data transmitted over the transmission path and performs inference processing such as discovering suspicious people. However, the video processing system in this disclosure may be used for any purpose, may encode and transmit any video data, and may perform any inference processing using such video data.

[0011] The encoding / transmission device 110 (video transmission device) is composed of one or more information processing devices each including a calculation device and a storage device. As shown in Fig. 1, the encoding / transmission device 110 includes an estimated bandwidth calculation unit 111, a feedback implementation determination unit 112, a coding control unit 113, an accuracy information storage unit 114, a code amount adjustment unit 115, a coding unit 116, and a transmission unit 117. The functions of the estimated bandwidth calculation unit 111, the feedback implementation determination unit 112, the coding control unit 113, the accuracy information storage unit 114, the code amount adjustment unit 115, the coding unit 116, and the transmission unit 117 can be realized by the calculation device executing a program for realizing each function stored in the storage device.

[0012] The receiving device 120 is configured with one or more information processing devices each including a calculation device and a storage device. As shown in Fig. 1, the receiving device 120 includes a receiving unit 121, a decoding unit 122, an inference unit 123, and an inference accuracy calculation unit 124. The functions of the receiving unit 121, the decoding unit 122, the inference unit 123, and the inference accuracy calculation unit 124 can be realized by the calculation device executing a program for realizing each function stored in the storage device. Each component will be described in detail below.

[0013] The estimated bandwidth calculation unit 111 uses the input transmission channel bandwidth information to calculate a predicted value of the bandwidth of the transmission channel N at the next moment, and outputs the calculated value to the feedback implementation determination unit 112. In the present disclosure, the predicted value of the bandwidth of the transmission channel N is referred to as the estimated bandwidth.

[0014] The transmission channel bandwidth information that is input to the estimated bandwidth calculation unit 111 is, for example, estimated by the receiving device 120 as described below, and is fed back from the receiving device 120 to the encoding and transmitting device 110. For example, the estimated bandwidth calculation unit 111 uses the transmission channel bandwidth information obtained by the receiving unit 121 of the receiving device 120 when transmitting video to calculate an estimated bandwidth, which is a predicted value of the transmission channel bandwidth at the next moment. As an example, a conceivable method is to use fluctuation data of the transmission channel bandwidth as input and statistically calculate the estimated bandwidth using a pre-generated prediction model.

[0015] The feedback execution determination unit 112 inputs the estimated bandwidth, determines whether there is a bandwidth available for transmitting multiple encoded video signals, and switches the processing unit that will perform the next process. Here, the feedback execution determination unit 112 determines whether the estimated bandwidth is greater than a preset threshold, and if it is greater than the threshold, determines that there is sufficient bandwidth, and if it is equal to or less than the threshold, determines that there is not sufficient bandwidth. Note that the threshold can be set externally. The feedback execution determination unit 112 also outputs the estimated bandwidth input from the estimated bandwidth calculation unit 111 as is to the processing unit that will perform the next process.

[0016] If the feedback implementation determination unit 112 determines that there is sufficient bandwidth, the encoding control unit 113 performs processing to calculate the inference accuracy for each code amount. As will be described later, the purpose is to perform highly accurate inference even when there is not sufficient bandwidth by utilizing information calculated when there is sufficient bandwidth.

[0017] If there is bandwidth available, the encoding unit 116 will perform encoding at multiple bit rates (second encoding process). At this time, it is necessary to select which value to use from among the candidate bit rate values. In the present disclosure, these candidate bit rate values ​​are referred to as bit rate setting candidate values, and the type and number of these values ​​can be set externally.

[0018] Here, if there is sufficient bandwidth, the receiving device 120 performs processing in the inference accuracy calculation unit 124. At this time, a correct inference result is prepared, and the inference accuracy of each inference result is calculated based on that result. For this reason, it is necessary to prepare detection results with high inference accuracy. Generally, images coded at a high bit rate are less likely to deteriorate in image quality, resulting in high inference accuracy. Therefore, one of the bit rate setting candidate values ​​is set as a high bit rate that is higher than the others, and the inference accuracy calculation unit 124 determines that the result coded at this rate is the correct inference result. In the present disclosure, this intentionally high bit rate among the bit rate setting candidate values ​​is referred to as the correct bit rate.

[0019] The encoding control unit 113 inputs the estimated bandwidth, selects multiple bit rates to be encoded from among the bit rate setting candidate values ​​so as not to exceed the estimated bandwidth, and instructs the encoding unit 116 on the selected bit rate values. The selected bit rate is stored, and the most recently selected bit rate is not selected, thereby comprehensively selecting all candidates. Note that, because inference accuracy cannot be calculated unless an image encoded at the correct bit rate is transmitted, the correct bit rate is selected every time as an exception. Note that for bit rates other than the correct bit rate, one or more bit rates lower than the correct bit rate value are selected.

[0020] The encoding unit 116 then inputs a frame image of the video data to be encoded, performs multiple encoding operations on the entire frame image at each bit rate instructed by the encoding control unit 113, and outputs each encoded video signal. To notify subsequent stages that multiple encoded video signals have been transmitted, the encoding unit 116 simultaneously adds additional information indicating that the signals are the encoding results of the same frame to the video signals and outputs them. Furthermore, for images encoded at the correct bit rate, additional information that identifies the image is added to the video signal.

[0021] Next, the transmitting unit 117 transmits the coded video signal obtained from the coding unit 116 to the transmission path N. At this time, the additional information added to the coded video signal is also transmitted at the same time.

[0022] Next, the receiving unit 121 in the receiving device 120 receives the multiple coded video signals and the additional information attached thereto transmitted from the coding / transmission device 110 via the transmission path N, and outputs them to the decoding unit 122. At this time, the receiving unit 121 also transmits the bandwidth information of the transmission path N to the estimated bandwidth calculation unit 111 of the coding / transmission device 110 via the transmission path N.

[0023] Next, the decoding unit 122 decodes the multiple coded video signals input from the receiving unit 121, associates them with the additional information that has been added, and outputs them to the inference unit 123.

[0024] Next, the inference unit 123 performs inference processing using an arbitrary image recognition / object detection algorithm with the frame images of the video signal decoded by the decoding unit 122 as input. One example is an object detection method based on deep learning, such as R-CNN (Region Based Convolutional Neural Networks). At this time, if there are inference results at the correct bit rate, the inference results are linked with additional information and output to the inference accuracy calculation unit 124 to calculate the inference accuracy. Furthermore, the information output as the inference result may be, for example, rectangular position information indicating the position of an object within the frame. Note that, when the inference result is used for purposes such as monitoring, the inference result from the receiving device 120 is needed immediately, and therefore, the inference result can also be output. Note that any inference processing may be performed.

[0025] Next, the inference accuracy calculation unit 124 calculates the accuracy of the inference based on the inference result input from the inference unit 123 and transmits it to the accuracy information storage unit 114 of the encoding / transmission device 110 via the transmission path N.

[0026] The calculation of inference accuracy is performed for each sub-picture (divided image) obtained by dividing a frame image in video data. The sub-picture division method must be set in advance, and the same division method must be specified for the encoding / transmission device 110 and the reception device 120. FIG. 2 shows an example of a frame image to be encoded / transmitted, and an example of how the image is divided into sub-pictures. In this example, an image showing, for example, a school building and the plaza in front of it is shown, and one image is divided into four sub-pictures vertically and five horizontally. Note that the number of divisions into sub-pictures and the division shape are not limited to those shown in FIG. 2.

[0027] The inference accuracy calculation unit 124 first determines the inference result at the correct bit rate as the inference result to be used as the correct answer based on additional information linked to the video signal, and then determines the bit rate at which each inference result was coded.The inference accuracy at each bit rate is calculated using the determined correct inference result as a reference.This inference result is calculated not for the entire image, but for each sub-picture within the image.

[0028] FIG. 3 shows an example of information output by the inference accuracy calculation unit 124. Here, as an example, of the subpictures divided as shown in FIG. 2, the subpicture located at row 3, column 2 in the left diagram of FIG. 3 (3-1) is referred to as "subpicture 32," and the subpicture located at row 3, column 5 is referred to as "subpicture 35." The right table shown in FIG. 3 (3-2) is an example of the information that can be obtained. In this example, the inference accuracy of subpictures 32 and 35 is calculated when the bit rate, i.e., the code amount, is 0.2 Mbps, 0.5 Mbps, and 1 Mbps. Note that the right table is only a fragment of the information that can be obtained; it is necessary to calculate the inference accuracy for all divided subpictures and all bit rate setting candidate values. As an example of an index of inference accuracy, mAP (mean average precision) is used.

[0029] The accuracy information holding unit 114 receives inference accuracy information for each sub-picture with respect to the bit rate from the inference accuracy calculation unit 124. At this time, if calculation of the inference accuracy for a certain or higher candidate value among the bit rate setting candidate values ​​has been completed, the accuracy information holding unit 114 calculates the relationship between the code amount and the inference accuracy.

[0030] FIG. 4 shows an example of the relationship (relationship information) between the amount of coding for each subpicture and the inference accuracy calculated by the accuracy information storage unit 114. Of the subpictures divided in the same manner as in FIG. 3, the subpicture located at row 3, column 2 is referred to as "subpicture 32" and the subpicture located at row 3, column 5 is referred to as "subpicture 35" as shown in the left diagram of FIG. 4(4-1). The middle diagram of FIG. 4(4-2) and the right diagram of FIG. 4(4-3) show an example of the relationship between the amount of coding and the inference accuracy. That is, the graphs show the change in the inference accuracy with respect to the change in the bit rate for each subpicture. The inference accuracy information obtained by the inference accuracy calculation unit 124 is complemented by any method to calculate the relationship characteristics between the amount of coding and the inference accuracy, and the information is stored.

[0031] The above-mentioned relationship characteristics are likely to change depending on the environment and time of day. As an example, "subpicture 35" shows the front of a school building, and since it is thought that many people and other detection targets appear in the morning and evening, it is estimated that the inference accuracy characteristics of this subpicture will change depending on the time of day. Therefore, by periodically recalculating this relationship characteristic and updating it to the latest information, it is possible to allocate a larger amount of code to subpictures that are important at that time in the code amount allocation process when bandwidth is limited, as described below.

[0032] Furthermore, the above-mentioned relationship characteristics are not limited to being calculated by the above-mentioned method for calculating inference accuracy, but can be used in subsequent processing by obtaining information corresponding to the calculation results from outside. The above-mentioned relationship characteristics may be calculated by the inference accuracy calculation unit 124 of the receiving device 120, or by another information processing device. The above-mentioned relationship characteristics may be prepared in advance, or such information may be input to the encoding / transmission device 110 and stored in the accuracy information holding unit 114.

[0033] Next, a description will be given of a case where it is determined that there is insufficient bandwidth as a result of the determination made by the feedback implementation determination unit 112. In this case, the encoding / transmission device 110 encodes the frame image by suppressing the code amount so as not to exceed the upper limit of the estimated bandwidth, and performs processing in the code amount adjustment unit 115 to allocate the optimal code amount to each sub-picture in the frame image.

[0034] The code amount adjustment unit 115 calculates the allocation of code amounts to each subpicture so as to maximize the inference accuracy within the range of the estimated bandwidth, using the estimated bandwidth input from the feedback implementation determination unit 112 and the relationship characteristics between code amount and inference accuracy obtained from the accuracy information storage unit 114. In other words, when there is not enough bandwidth, the code amount adjustment unit 115 will encode each subpicture of the frame image at the subpicture bit rate set for that subpicture, and calculates and sets the value of the code amount, which is the bit rate for each subpicture. The code amount adjustment unit 115 then outputs the calculated code amount value to the encoding unit 116.

[0035] As an example of a method for calculating the optimal code amount allocation, the objective is to maximize the inference accuracy of the entire image, and the condition is that the code amount of the entire image is smaller than the estimated bandwidth. By utilizing the relationship characteristics of the inference accuracy to the code amount of each sub-picture as shown in Figures 4(4-2) and (4-3), it is possible to solve the optimization problem of adjusting the code amount of each sub-picture.

[0036] Following the code amount adjustment unit 115, the encoding unit 116 inputs the frame image to be encoded, adjusts the quantization parameters of each sub-picture so that the code amount of each sub-picture becomes the code amount specified by the code amount adjustment unit 115, and performs encoding, and outputs an encoded video signal.

[0037] FIG. 5 shows an example of an image encoded by adjusting the code amount, taking a bandwidth limit of 1 Mbps as an example. The left diagram in FIG. 5 (5-1) shows an image frame to be transmitted, and the right table in FIG. 5 (5-2) shows an example of the code amount (Mbps) allocated to each subpicture. For subpicture 11, which shows only fixed objects such as buildings and has no relation to detection targets such as moving objects, it is expected that the relationship characteristics of inference accuracy will not increase significantly even if the code amount is increased, and therefore a large code amount is not allocated in the allocation process. On the other hand, for subpicture 35, which shows a detection target such as a person, it is expected that the relationship characteristics of inference accuracy will increase as the code amount is increased, and therefore a relatively large code amount is allocated in the allocation process. In other words, the code amount adjustment unit 115 uses the relationship characteristics described above to allocate a larger code amount to a subpicture whose inference accuracy changes more significantly with changes in the code amount. Furthermore, for subpictures whose inference accuracy increases with increasing code amount, such as areas showing moving objects, a larger code amount is allocated.

[0038] Next, the transmitting unit 117 transmits the coded video signal acquired from the coding unit 116 to the transmission path N.

[0039] Next, the receiving unit 121 in the receiving device 120 receives the coded video signal sent from the transmitting unit 117 via the transmission path N and outputs it to the decoding unit 122. At this time, the receiving unit 121 also transmits the bandwidth information of the transmission path N to the estimated bandwidth calculation unit 111 of the coding / transmitting device 110 via the transmission path N.

[0040] Next, the decoding unit 122 decodes the coded video signal input from the receiving unit and outputs it to the inference unit 123. Next, the inference unit 123 performs inference using any image recognition or object detection algorithm, using the frame images of the video signal decoded by the decoding unit 122 as input. Furthermore, when the inference result is required immediately in the receiving device 120 for purposes such as surveillance, the inference result can be output.

[0041] [Operation] Fig. 6 is a flowchart showing the processing operations of the encoding / transmission device 110 and a part of the processing of the reception device 120. Note that Fig. 6 shows the processing of one frame in a video from start to finish.

[0042] The estimated bandwidth calculation unit 111 acquires bandwidth information from the receiving side, predicts and outputs an estimated bandwidth (step S1). Next, the feedback execution determination unit 112 uses the estimated bandwidth to determine whether there is sufficient bandwidth and switches the subsequent processing (step S2).

[0043] In step S2, if it is determined that there is sufficient bandwidth, i.e., the estimated bandwidth exceeds the specified threshold (Yes in step S2), a process of calculating the inference accuracy for each bit rate is performed while transmitting the video. Specifically, the encoding / transmission device 110 selects a coding bit rate (step S3), encodes the video (step S4), and transmits the encoded video to the receiving device 120 (step S5). The video is then processed on the receiving device 120 side (step S6), and thereafter the encoding / transmission device 110 performs a process of calculating the relationship between the code amount and the inference accuracy (step S7).

[0044] Here, the details of the processing of steps S3 to S5 in Fig. 6 are shown in Fig. 7. Furthermore, the details of the processing of step S6 in Fig. 6 are shown in Fig. 8, and the details of the processing of step S7 in Fig. 6 are shown in Fig. 9.

[0045] 7, if it is determined that there is sufficient bandwidth, the feedback implementation determination unit 112 inputs the estimated bandwidth to the encoding control unit 113 (step S21). Next, the encoding control unit 113 determines which value from among the bit rate setting candidate values ​​to use for encoding based on the predicted bandwidth (step S22), and outputs this to the encoding unit 116 together with information for identifying that multiple encoding operations have been performed (step S23).

[0046] Next, the encoding unit 116 receives the video signal and performs multiple encoding at the specified bit rate (step S24). Next, the transmission unit 117 transmits each of the multiple encoded video signals received from the encoding unit 116 to the receiving device 120 (step S25).

[0047] 8, when multiple coded video signals are input to the receiving device 120, the receiving unit 121 receives each signal (step S31). The decoding unit 122 then decodes each of the multiple coded video signals (step S32). The inference unit 123 then performs inference using an arbitrary image recognition or object detection algorithm with the decoded frame images as input (step S33). The inference accuracy calculation unit 124 then calculates the inference accuracy for each bit rate and each sub-picture based on the input inference results (step S35), and feeds it back to the transmitting side (step S36).

[0048] 9, the encoding / transmission device 110 acquires inference accuracy information from the reception device 120 (step S41) and stores it in the accuracy information holding unit 114. Next, the accuracy information holding unit 114 determines whether sufficient inference accuracy information is stored to calculate the relationship between the code amount and the inference accuracy (step S42), and if stored (Yes in step S42), calculates the relationship between the code amount and the inference accuracy (step S43). This completes the processing for one frame when there is room in the prediction band.

[0049] 7, if it is determined that there is no bandwidth available, i.e., that the estimated bandwidth does not exceed the specified threshold (No in step S2), encoding is performed so that the inference accuracy is maximized within the limited estimated bandwidth. Specifically, the bit rate allocated to each subpicture is calculated and specified (steps S8 and S9), the encoding is performed, and the video is transmitted to the receiving device 120 (steps S10 and S11), after which the video is processed on the receiving device 120 side (step S12).

[0050] Here, the details of the processing of steps S8 to S11 in Fig. 6 are shown in Fig. 10. Also, the details of the processing of step S12 in Fig. 6 are shown in Fig. 11.

[0051] As shown in FIG. 10, if it is determined that there is insufficient bandwidth, the feedback implementation determination unit 112 inputs the estimated bandwidth to the code amount adjustment unit 115 (step S51). Next, the accuracy information storage unit 114 determines whether the relationship between the code amount and the inference accuracy has been calculated (step S52). If the determination result shows that the relationship between the code amount and the inference accuracy has not been calculated (No in step S52), it instructs not to perform an optimal code amount allocation process (step S53). If optimal code amount allocation is not performed, as an example, the entire image is coded with a uniform code amount. Next, the code amount adjustment unit 115 allocates the entire image with a uniform bit rate so as not to exceed the bandwidth upper limit (step S54).

[0052] Next, the encoding unit 116 adjusts the quantization parameter of each subpicture based on the instructed bit rate allocation (step S55). Next, the encoding unit 116 inputs the video signal and performs encoding using the instructed code amount allocation (step S56). Next, the transmission unit 117 transmits the encoded video signal input from the encoding unit 116 (step S57).

[0053] On the other hand, if the result of the determination in step S52 above is that the relationship between the code amount and the inference accuracy has been calculated (Yes in step S52), the calculated relationship between the code amount and the inference accuracy is output (step S58) to the code amount adjustment unit 115. The code amount adjustment unit 115 uses the input estimated bandwidth, the code amount, and the relationship between the inference accuracy to calculate a code amount allocation that does not exceed the bandwidth upper limit and maximizes the overall inference accuracy (step S59).

[0054] Next, the encoding unit 116 adjusts the quantization parameter of each subpicture based on the instructed bit rate allocation (step S55). Next, the encoding unit 116 inputs the video signal and performs encoding using the instructed code amount allocation (step S56). Next, the transmission unit 117 transmits the encoded video signal input from the encoding unit 116 (step S57).

[0055] 11, when one coded video signal is transmitted from the receiving device 120, the receiving unit 121 receives the signal (step S61). Next, the decoding unit 122 decodes the input coded video signal (step S62). Next, the inference unit 123 uses the decoded frame image as input and performs inference using any image recognition or object detection algorithm (step S63). This completes the processing of one frame when there is no room in the prediction bandwidth.

[0056] As described above, in this embodiment, by checking the transition of inference accuracy relative to the code amount when there is sufficient bandwidth, and by using the calculated inference accuracy information when the bandwidth is limited, it is possible to perform encoding that is advantageous for inference without exceeding the bandwidth upper limit. This allows frame images to be transmitted reliably even in an environment where the bandwidth of the transmission path N is unstable, particularly when the bandwidth is limited, and enables encoding to be performed so that inference is optimized according to the characteristics of the content displayed in each area of ​​the frame image. As a result, the receiving side can reliably receive video of a quality suitable for inference, allowing appropriate inference to be performed on the video.

[0057] <Second embodiment> Next, a second embodiment of the present disclosure will be described with reference to the drawings. This embodiment shows an outline of the configuration of the video transmission device described in the above embodiment. Note that Figures 12 and 13 are diagrams for explaining the configuration, and these diagrams may be relevant to any of the embodiments.

[0058] First, the hardware configuration of the video transmission device 200 will be described with reference to Fig. 12. The video transmission device 200 is configured as a general information processing device, and is equipped with the following hardware configuration, as an example. ·CPU (Central Processing Unit) 201 (computing unit) ROM (Read Only Memory) 202 (storage device) RAM (Random Access Memory) 203 (storage device) Programs 204 loaded into RAM 203 A storage device 205 for storing the program group 204 A drive device 206 that reads and writes from a storage medium 210 external to the information processing device A communication interface 207 that connects to a communication network 211 outside the information processing device Input / output interface 208 for inputting and outputting data Bus 209 connecting each component

[0059] 12 shows an example of the hardware configuration of the information processing device that is the video transmission device 200, and the hardware configuration of the information processing device is not limited to the above-described case. For example, the information processing device may be configured with a part of the above-described configuration, such as excluding the drive device 206. Furthermore, the information processing device may use a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof, instead of the above-described CPU.

[0060] The video transmission device 200 can be equipped with an encoding unit 221, a transmitting unit 222, and an acquiring unit 223 shown in Fig. 13 by having the CPU 201 acquire and execute the group of programs 204. The group of programs 204 is stored in advance in the storage device 205 or the ROM 202, for example, and is loaded into the RAM 203 and executed by the CPU 201 as needed. The group of programs 204 may be supplied to the CPU 201 via the communication network 211, or may be stored in advance in the storage medium 210, and the drive device 206 may read out the programs and supply them to the CPU 201. However, the encoding unit 221, the transmitting unit 222, and the acquiring unit 223 described above may be constructed using dedicated electronic circuits for realizing such means.

[0061] The encoding unit 221 encodes each divided image obtained by dividing a frame image constituting a video at a bit rate for the divided image set for each divided image. The transmission unit 222 transmits the encoded frame images via a transmission path. The acquisition unit 223 acquires an estimated bandwidth, which is an estimate of the bandwidth of the transmission path, and the inference accuracy for each divided image when a predetermined inference process is performed on each divided image of the frame image. The encoding unit 221 then sets the bit rate for the divided image to be used when encoding each divided image based on the inference accuracy for each divided image, so that the frame images can be encoded so that they can be transmitted within the estimated bandwidth of the transmission path.

[0062] By configuring the present disclosure as described above, it is possible to reliably transmit video within the estimated bandwidth of the transmission path, and each divided image is encoded at a bit rate suitable for inference, allowing appropriate inference to be performed on such divided images.

[0063] In addition, at least one of the functions of the encoding unit 221, the transmitting unit 222, and the acquiring unit 223 described above may be executed by an information processing device installed and connected anywhere on the network, that is, they may be executed by so-called cloud computing.

[0064] The above-described program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program can also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0065] Although the present disclosure has been described above with reference to the above-described embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each of the above-described embodiments can be combined with other embodiments as appropriate.

[0066] <Additional Notes> A part or all of the above-described embodiments can be described as follows: The following provides an overview of the configurations of a video transmission device, a video transmission method, and a program according to the present disclosure. However, the present disclosure is not limited to the following configurations. (Appendix 1) an encoding unit that encodes each divided image obtained by dividing a frame image that constitutes a video image at a bit rate for the divided image that is set for each divided image; a transmitting unit that transmits the encoded frame image via a transmission path; an acquisition unit that acquires an estimated bandwidth that is an estimated value of the bandwidth of the transmission path and an inference accuracy for each divided image when a predetermined inference process is performed for each divided image of the frame image; Equipped with the encoding unit sets a bit rate for the divided images to be used for encoding each of the divided images based on the inference accuracy for each of the divided images so that the frame images can be encoded so as to be transmittable within the estimated bandwidth of the transmission path. Video transmission equipment. (Appendix 2) 10. The video transmission device according to claim 1, the acquisition unit acquires an inference accuracy for each bit rate and for each divided image when the entire frame image is encoded at each of a plurality of different bit rates; the encoding unit sets a bit rate for each of the divided images based on the inference accuracy for each bit rate and for each of the divided images. Video transmission equipment. (Appendix 3) 3. A video transmission device according to claim 2, the acquisition unit acquires relationship information representing a change in the inference accuracy with respect to a change in the bit rate for each of the divided images; the encoding unit sets a bit rate for each of the divided images based on the relationship information. Video transmission equipment. (Appendix 4) 4. The video transmission device according to claim 3, the encoding unit sets a larger value of the bit rate for the divided image for which the change in inference accuracy with respect to a change in bit rate is larger than the bit rate for the divided image for which the change in inference accuracy with respect to a change in bit rate is small, based on the relationship information. Video transmission equipment. (Appendix 5) 3. A video transmission device according to claim 2, the encoding unit performs a second encoding process of encoding the entire frame image at a plurality of different bit rates, the transmitting unit transmits the frame images subjected to the second encoding process for each bit rate to a receiving device, the acquiring unit acquires from the receiving device an inference accuracy in a predetermined inference process performed for each of the divided images by the receiving device on the frame images for each bit rate that have been subjected to the second encoding process. Video transmission equipment. (Appendix 6) 6. The video transmission device according to claim 5, the acquisition unit determines whether the estimated bandwidth has sufficient bandwidth for transmitting the frame image based on a preset criterion; the encoding unit performs the second encoding process when there is a margin, and when there is no margin, sets a bit rate for the divided image and performs the encoding process at the bit rate for the divided image; the transmitting unit transmits the frame images encoded by the second encoding process to the receiving device when there is a margin, and transmits the frame images encoded by the encoding process at the bit rate for the divided images to the receiving device when there is no margin. Video transmission equipment. (Appendix 7) 10. The video transmission device according to claim 1, the acquisition unit acquires bandwidth information during data transmission and reception via the transmission path, and estimates a value of a bandwidth after the transmission path based on the bandwidth information to acquire the estimated bandwidth. Video transmission equipment. (Appendix 8) Each divided image obtained by dividing the frame images that make up the video is encoded at a bit rate for the divided image that is set for each divided image, A video transmission method for transmitting the encoded frame images via a transmission path, comprising: an estimated bandwidth that is an estimated value of the bandwidth of the transmission path, and an inference accuracy for each divided image when a predetermined inference process is performed for each divided image of the frame image; a bit rate for each of the divided images to be used for encoding the divided images is set based on the inference accuracy for each of the divided images so that the frame images can be encoded so as to be transmittable within the estimated bandwidth of the transmission path; Video transmission method. (Appendix 9) 9. The video transmission method according to claim 8, further comprising: Obtaining an inference accuracy for each bit rate and for each divided image when the entire frame image is encoded at a plurality of different bit rates; setting a bit rate for each of the divided images based on the inference accuracy for each bit rate and for each of the divided images; Video transmission method. (Appendix 9.1) 10. The video transmission method according to claim 9, further comprising: Acquire relationship information that indicates a change in the inference accuracy with respect to a change in the bit rate for each of the divided images; setting a bit rate for each of the divided images based on the relationship information; Video transmission method. (Appendix 9.2) 9.1, a video transmission method according to claim 9.1, comprising: Based on the relationship information, a larger bit rate is set for the divided image in which the change in inference accuracy with respect to a change in bit rate is larger than that for the divided image in which the change in inference accuracy with respect to a change in bit rate is small. Video transmission method. (Appendix 9.3) 10. The video transmission method according to claim 9, further comprising: performing a second encoding process for encoding the entire frame image at a plurality of different bit rates; transmitting the frame images for each bit rate that have been subjected to the second encoding process to a receiving device; acquiring from the receiving device an inference accuracy in a predetermined inference process performed for each of the divided images by the receiving device on the frame images for each bit rate that have been subjected to the second encoding process; Video transmission method. (Appendix 9.4) 9.3, a video transmission method according to claim 9.3, comprising: determining whether the estimated bandwidth is sufficient for transmitting the frame image based on a preset criterion; If there is a margin, the second encoding process is performed, and the frame image obtained by the second encoding process is transmitted to the receiving device; If there is no margin, a bit rate for the divided image is set, encoding is performed at the bit rate for the divided image, and the frame image encoded at the bit rate for the divided image is transmitted to the receiving device. Video transmission method. (Appendix 9.5) 9. The video transmission method according to claim 8, further comprising: acquiring bandwidth information during data transmission and reception via the transmission path, and estimating a value of a bandwidth after the transmission path based on the bandwidth information to acquire the estimated bandwidth; Video transmission method. (Appendix 10) Each divided image obtained by dividing the frame images that make up the video is encoded at a bit rate for the divided image that is set for each divided image, A video transmission method for transmitting the encoded frame images via a transmission path, comprising: an estimated bandwidth that is an estimated value of the bandwidth of the transmission path, and an inference accuracy for each divided image when a predetermined inference process is performed for each divided image of the frame image; a bit rate for each of the divided images to be used for encoding the divided images is set based on the inference accuracy for each of the divided images so that the frame images can be encoded so as to be transmittable within the estimated bandwidth of the transmission path; A program that causes a computer to perform a process. [Explanation of symbols]

[0067] 110 Encoding and transmitting equipment 111 Estimated Bandwidth Calculation Unit 112 Feedback implementation judgment unit 113 Encoding control unit 114 Accuracy information holding unit 115 Code amount adjustment section 116 Encoding section 117 Transmitter 120 receiving device 121 Receiving unit 122 Decoding unit 123 Reasoning part 124 Inference accuracy calculation unit 200 Video transmission equipment 201 CPU 202 ROM 203 RAM 204 Programs 205 Storage device 206 Drive unit 207 Communication Interface 208 Input / Output Interface 209 Bus 210 Storage medium 211 Communication Network 221 Encoding section 222 Transmitter 223 Acquisition Department

Claims

1. an encoding unit that encodes each divided image obtained by dividing a frame image that constitutes a video image at a bit rate for the divided image that is set for each divided image; a transmitting unit that transmits the encoded frame image via a transmission path; an acquisition unit that acquires an estimated bandwidth that is an estimated value of the bandwidth of the transmission path and an inference accuracy for each divided image when a predetermined inference process is performed for each divided image of the frame image; Equipped with the encoding unit sets a bit rate for the divided images to be used for encoding each of the divided images based on the inference accuracy for each of the divided images so that the frame images can be encoded so as to be transmittable within the estimated bandwidth of the transmission path. Video transmission equipment.

2. 2. The video transmission device according to claim 1, the acquisition unit acquires an inference accuracy for each bit rate and for each divided image when the entire frame image is encoded at each of a plurality of different bit rates; the encoding unit sets a bit rate for each of the divided images based on the inference accuracy for each bit rate and for each of the divided images. Video transmission equipment.

3. 3. The video transmission device according to claim 2, the acquisition unit acquires relationship information representing a change in the inference accuracy with respect to a change in the bit rate for each of the divided images; the encoding unit sets a bit rate for each of the divided images based on the relationship information. Video transmission equipment.

4. 4. The video transmission device according to claim 3, the encoding unit sets a larger value of the bit rate for the divided image for which the change in inference accuracy with respect to a change in bit rate is larger than the bit rate for the divided image for which the change in inference accuracy with respect to a change in bit rate is small, based on the relationship information. Video transmission equipment.

5. 3. The video transmission device according to claim 2, the encoding unit performs a second encoding process of encoding the entire frame image at a plurality of different bit rates, the transmitting unit transmits the frame images subjected to the second encoding process for each bit rate to a receiving device, the acquiring unit acquires from the receiving device an inference accuracy in a predetermined inference process performed for each of the divided images by the receiving device on the frame images for each bit rate that have been subjected to the second encoding process. Video transmission equipment.

6. 6. The video transmission device according to claim 5, the acquisition unit determines whether the estimated bandwidth has sufficient bandwidth for transmitting the frame image based on a preset criterion; the encoding unit performs the second encoding process when there is a margin, and when there is no margin, sets a bit rate for the divided image and performs the encoding process at the bit rate for the divided image; the transmitting unit transmits the frame images encoded by the second encoding process to the receiving device when there is a margin, and transmits the frame images encoded by the encoding process at the bit rate for the divided images to the receiving device when there is no margin. Video transmission equipment.

7. 2. The video transmission device according to claim 1, the acquisition unit acquires bandwidth information during data transmission and reception via the transmission path, and estimates a value of a bandwidth after the transmission path based on the bandwidth information to acquire the estimated bandwidth. Video transmission equipment.

8. Each divided image obtained by dividing the frame images that make up the video is encoded at a bit rate for the divided image that is set for each divided image, A video transmission method for transmitting the encoded frame images via a transmission path, comprising: an estimated bandwidth that is an estimated value of the bandwidth of the transmission path, and an inference accuracy for each divided image when a predetermined inference process is performed for each divided image of the frame image; a bit rate for each of the divided images to be used for encoding the divided images is set based on the inference accuracy for each of the divided images so that the frame images can be encoded so as to be transmittable within the estimated bandwidth of the transmission path; Video transmission method.

9. 9. A video transmission method according to claim 8, Obtaining an inference accuracy for each bit rate and for each divided image when the entire frame image is encoded at a plurality of different bit rates; setting a bit rate for each of the divided images based on the inference accuracy for each bit rate and for each of the divided images; Video transmission method.

10. Each divided image obtained by dividing the frame images that make up the video is encoded at a bit rate for the divided image that is set for each divided image, A video transmission method for transmitting the encoded frame images via a transmission path, comprising: an estimated bandwidth that is an estimated value of the bandwidth of the transmission path, and an inference accuracy for each divided image when a predetermined inference process is performed for each divided image of the frame image; a bit rate for each of the divided images to be used for encoding the divided images is set based on the inference accuracy for each of the divided images so that the frame images can be encoded so as to be transmittable within the estimated bandwidth of the transmission path; A program that causes a computer to perform a process.

Images