Information processing device, information processing method, and information processing program

Abstract

This information processing device comprises: an image data acquisition unit that acquires image data; a target data amount acquisition unit that acquires a target data amount of a transmission amount of the image data; and an estimation unit that estimates image quality, by referring to the target data amount, for each of a plurality of regions included in an image indicated by the image data.

Description

Information processing device, information processing method, and information processing program

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program.

[0002] One known video data compression technique is to reduce the amount of coded data by varying the quality of each area included in an image, for each image that makes up the image. For example, Patent Literature 1 discloses a technique in which importance is calculated for each area included in an image captured by an imaging device, and the image quality of areas with high importance is maintained high and the image quality of areas with low importance is reduced.

[0003] Such a technique attempts to reduce the image quality of the remaining areas without reducing the image quality of a certain percentage of the areas in descending order of calculated importance.

[0004] International Publication No. 2022 / 181367

[0005] However, in the technology of Patent Document 1 mentioned above, the ratio of areas that maintain high image quality to areas that reduce image quality is constant, so when the image resolution increases or when there is a large difference between images contained in the video, that is, when there is a large change over time in the content shown by the video, the amount of data in the encoded data after compression increases, and there is a risk that the data may exceed the available transmission capacity when transmitted.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and an exemplary purpose thereof is to provide a technology for reducing the amount of image data to a predetermined transmission data amount or less while retaining necessary information.

[0007] An information processing device according to an exemplary aspect of the present disclosure includes an image data acquisition means for acquiring image data, a target data amount acquisition means for acquiring a target data amount for transmitting the image data, and an estimation means for estimating image quality for each of a plurality of areas included in an image indicated by the image data by referring to the target data amount.

[0008] An information processing device according to an exemplary aspect of the present disclosure includes an image data acquisition means for acquiring image data, a target data amount acquisition means for acquiring a target data amount for transmitting the image data, an estimation means for estimating image quality for each of a plurality of areas included in an image indicated by the image data by referring to the target data amount, a replacement means for replacing the image data for each of the areas with alternative data according to the estimated image quality, an evaluation means for deriving an evaluation value by referring to the image data of all areas after replacement, and a learning means for updating parameters of the estimation means by referring to the evaluation value.

[0009] An information processing device according to an exemplary aspect of the present disclosure includes an image data acquisition means for acquiring image data, a target data amount acquisition means for acquiring a target data amount for transmitting the image data, an estimation means for estimating image quality for each of a plurality of areas included in an image indicated by the image data by referring to the target data amount, a replacement means for replacing the image data for each of the areas with alternative data according to the estimated image quality, an encoding means for generating encoded data by encoding the alternative data, and a transmission means for transmitting the encoded data.

[0010] An information processing method according to an exemplary aspect of the present disclosure includes at least one processor acquiring image data, acquiring a target data volume for transmitting the image data, and estimating image quality for each of a plurality of regions included in an image represented by the image data by referring to the target data volume.

[0011] An information processing method according to an exemplary aspect of the present disclosure includes at least one processor that acquires image data, acquires a target data volume for transmitting the image data, estimates image quality for each of a plurality of regions included in an image represented by the image data by referring to the target data volume, replaces the image data for each of the regions with alternative data according to the estimated image quality, derives an evaluation value by referring to the image data of all regions after replacement, and updates estimated parameters by referring to the evaluation value.

[0012] An information processing method according to an exemplary aspect of the present disclosure includes at least one processor that acquires image data, acquires a target data volume for transmitting the image data, estimates image quality for each of a plurality of regions included in an image represented by the image data by referring to the target data volume, replaces the image data for each of the regions with alternative data according to the estimated image quality, generates encoded data by encoding the alternative data, and transmits the encoded data.

[0013] An information processing program relating to an exemplary aspect of the present disclosure causes a computer to execute an image data acquisition process to acquire image data, a target data amount acquisition process to acquire a target data amount for transmitting the image data, and an estimation process to estimate image quality for each of a plurality of areas included in an image represented by the image data by referring to the target data amount.

[0014] According to an exemplary aspect of the present disclosure, an exemplary effect is achieved in that a technique can be provided that reduces the amount of image data to a predetermined transmission data amount or less while retaining necessary information.

[0015] 1 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 2 is a flow diagram showing the flow of an information processing method executed by an information processing device according to the present disclosure. FIG. 3 is a block diagram showing the configuration of another information processing device according to the present disclosure. FIG. 4 is a flow diagram showing the flow of an information processing method executed by another information processing device according to the present disclosure. FIG. 5 is a block diagram showing the configuration of an information processing system including an information processing device according to the present disclosure. FIG. 6 is a flowchart showing an example of the flow of a learning method for learning a method for estimating image quality for each region included in an image, executed by an information processing device according to the present disclosure. FIG. 7 is a flowchart of the simulation evaluation step shown in FIG. 6. FIG. 7 is a flowchart of the evaluation value calculation step shown in FIG. 6. FIG. 8 is a block diagram showing the configuration of an information processing system according to the present disclosure. FIG. 9 is a block diagram showing the configuration of an image analysis system according to the present disclosure. FIG. 10 is a flow diagram showing the flow of an image analysis method according to the present disclosure. FIG. 11 is a diagram comparing the bit rate of a replaced image of the prior art with the bit rate of a replaced image according to the present disclosure. FIG. 12 is a diagram comparing the ratio of high image quality regions in a replaced image of the prior art with the ratio of high image quality regions in a replaced image according to the present disclosure. FIG. 13 is a diagram comparing the position accuracy of a position estimation performed using an image in which the entire image is replaced with uniform image quality with the position accuracy of a replaced image according to the present disclosure. FIG. 1 is a block diagram illustrating a configuration of a computer that functions as an information processing device according to the present disclosure.

[0016] Below, exemplary embodiments of the present disclosure are described. However, the present disclosure is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technologies (part or all of the products or methods) employed in the exemplary embodiments shown below may also be included in the scope of the present disclosure. Furthermore, embodiments obtained by appropriately omitting some of the technologies employed in the exemplary embodiments shown below may also be included in the scope of the present disclosure. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present disclosure. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present disclosure.

[0017] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present disclosure, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technique employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technique employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technique shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0018] (Configuration of Information Processing Device 1) The configuration of the information processing device 1 will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 is a device that trains an estimation device that estimates the image quality of an area included in image data by referring to a target data amount. As shown in FIG. 1, the information processing device 1 includes a target data amount acquisition unit 10, an image data acquisition unit 11, an estimation unit 12, a replacement unit 13, an evaluation unit 14, and a learning unit 15. The target data amount acquisition unit 10 is an embodiment of the target data amount acquisition means set forth in the claims. The image data acquisition unit 11 is an embodiment of the image data acquisition means set forth in the claims. The estimation unit 12 is an embodiment of the estimation means set forth in the claims. The replacement unit 13 is an embodiment of the replacement means set forth in the claims. The evaluation unit 14 is an embodiment of the evaluation means set forth in the claims. The learning unit 15 is an embodiment of the learning means set forth in the claims.

[0019] As an example, the information processing device 1 can be used to process image data acquired from a viewpoint of a moving body such as an autonomous vehicle, a robot, or a drone. More specifically, the information processing device 1 can be used to process video data acquired by an imaging device mounted on the moving body. Each component included in the information processing device 1 will be described below.

[0020] The target data amount acquisition unit 10 acquires a target data amount for the transmission amount of image data. The target data amount is, for example, the transmission speed (image bit rate, BPS) when transmitting images using a wireless communication network. The target data amount acquired by the target data amount acquisition unit 10 is, for example, a value calculated from the performance of the communication network measured in advance or the performance assigned to the terminal by the operations manager. The method by which the target data amount acquisition unit 10 acquires the target data amount is not limited. As an example, the target data amount acquisition unit 10 can acquire a target data amount stored in memory. The target data amount acquisition unit 10 inputs the acquired target data amount to the estimation unit 12.

[0021] The image data acquisition unit 11 acquires image data (including input data). Input data refers to data to be input to the estimation unit 12. The image data acquired by the image data acquisition unit 11 is, for example, video data captured by a camera or the like. The image data may be still image data or video (moving image) data. In other words, the data may be a single piece of data or multiple image frame data acquired continuously. The method by which the image data acquisition unit 11 acquires image data from a camera or the like is not limited. As an example, the image data acquisition unit 11 can acquire this data using wired communication, wireless communication, or a combination of these. The image data acquisition unit 11 inputs the acquired input data to the estimation unit 12.

[0022] The estimation unit 12 estimates the image quality of each of multiple regions included in the image indicated by the image data acquired by the image data acquisition unit 11, with reference to the target data amount. The image regions may be set in any manner. For example, the regions may be regions obtained by dividing the image into equal rectangles. Alternatively, the regions may have different areas depending on the location. The estimated image quality is an image quality that enables stable image transmission. More specifically, in this embodiment, the estimated image quality is an image quality that maximizes an evaluation value (described later) while suppressing the data volume when encoding substitute data (described later) to a target data volume or less. The estimation unit 12 estimates the image quality of multiple regions with reference to image feature amounts, image differences, and the target data amount so that the transmission volume of image data is equal to or less than the target data volume. The image feature amounts vary depending on, for example, the number and complexity of edges in the image.

[0023] When estimating the appropriate image quality, the estimation unit 12 refers to the difference between the image and the image immediately preceding it. By taking the image difference into account, appropriate image quality can be estimated, taking into account the magnitude of changes in image content. Specifically, the estimation unit 12 estimates the image quality of each region so that the proportion of high-quality regions decreases as the image difference increases. In conventional techniques for calculating the importance of regions in an image, when the image difference is large, the number of regions with high importance tends to increase. However, in such conventional techniques, regions with high importance are also evaluated as having high image quality, which increases the amount of data transmitted (hereinafter also referred to as "transmission amount") for transmitting the image. This can cause image distortion, which is particularly undesirable when tracking the position of a moving object from an image using, for example, Visual Simultaneous Localization and Mapping (VSLAM). However, the estimation unit 12 of the present disclosure estimates the image quality of each region so that the proportion of high-quality regions decreases as the image difference increases. This can prevent image distortion caused by the amount of transmitted data exceeding the available communication bandwidth.

[0024] For example, the estimation unit 12 may use a neural network algorithm. For example, the estimation unit 12 may include a self-attention module. With this configuration, the image difference, complexity, and feature amount of each region of the image are projected onto a vector, and an estimated image quality for each region is output by referring to a target data amount.

[0025] The replacement unit 13 generates replaced data by replacing image data for each of a plurality of regions included in the input data with substitute data in accordance with the estimated image quality. In this embodiment, the substitute data is data that has a smaller data volume when encoded than the input data.

[0026] An example of substitute data is image data in which a quantization parameter (QP) is set larger (quantization error is increased) than the image before replacement. Furthermore, in areas where the estimated image quality is high, the quantization parameter is set smaller (quantization error is reduced) than in areas where the estimated image quality is low. However, when transmitting the substitute data, the substitute data must have a data volume that does not exceed the target data volume.

[0027] The evaluation unit 14 derives an evaluation value by referring to the image data of the entire region after replacement. In this embodiment, the evaluation value includes an index indicating to what extent the purpose of using the input data is achieved when the replaced data is referred to, for example, when the replaced data is used, and an index indicating to what extent a target data volume is achieved when the replaced data is encoded.

[0028] As an example, the evaluation value may be calculated using the detection result obtained when object detection is performed using the replaced data, the detection accuracy calculated from the correct object detection result, the data volume obtained by encoding the replaced data, and the target data volume. The detection result obtained when object detection is performed using the input data before replacement may be used as the correct data. Alternatively, the evaluation unit 14 may input the image data replaced with the substitute data into a VSLAM, compare the obtained output with the correct data to calculate a reward value, and derive the evaluation value by referring to the reward value and the data volume of the substitute data. This evaluation method will be described in detail in exemplary embodiment 2.

[0029] The learning unit 15 causes the estimating unit 12 to learn by referring to the evaluation value. That is, the learning unit 15 updates the parameters of the estimating unit 12 by referring to the evaluation value. As described above, the estimating unit 12 is a type of machine model, and parameters included in the machine model are also referred to as "estimated parameters." In this embodiment, learning refers to updating various parameters (estimated parameters) used when the estimating unit estimates image quality so as to obtain a higher evaluation value.

[0030] As an example, various parameters used by the estimation unit to estimate image quality may be adjusted and updated by changing the combination of areas to be replaced with alternative data so that the evaluation value obtained by inputting the data into object detection is increased.

[0031] In the embodiment shown in FIG. 1 , the target data amount acquisition unit 10, image data acquisition unit 11, estimation unit 12, replacement unit 13, evaluation unit 14, and learning unit 15 are described as being incorporated into a single information processing device 1, but these do not necessarily need to be incorporated into a single information processing device. For example, all or part of the target data amount acquisition unit 10, image data acquisition unit 11, estimation unit 12, replacement unit 13, evaluation unit 14, and learning unit 15 may be located separately. These may be connected via wired or wireless communication. Furthermore, all or part of the target data amount acquisition unit 10, image data acquisition unit 11, estimation unit 12, replacement unit 13, evaluation unit 14, and learning unit 15 may be located on the cloud. This also applies to the device configurations described below.

[0032] (Effects of Information Processing Device 1) As described above, the information processing device 1 according to this exemplary embodiment includes an image data acquisition unit 11 that acquires image data, a target data amount acquisition unit 10 that acquires a target data amount for the transmission amount of image data, an estimation unit 12 that estimates image quality for each of multiple regions included in the image represented by the image data by referring to the target data amount, a replacement unit 13 that replaces image data for each region with substitute data according to the estimated image quality, an evaluation unit 14 that derives an evaluation value by referring to the image data of all regions after replacement, and a learning unit 15 that updates parameters of the estimation unit 12 by referring to the evaluation value. Therefore, the information processing device 1 according to this exemplary embodiment has the advantage of providing a technology that reduces the data amount of an image to a predetermined transmission data amount or less while retaining necessary information. The necessary information includes information important for deriving the position of the imaging device that captured the image, information important for detecting objects in the image, etc.

[0033] (Flow of Information Processing Method S1) Next, the flow of the information processing method (learning method) S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flowchart showing the flow of the information processing method S1 executed by the information processing device 1. As shown in Fig. 2, the information processing method S1 includes the following steps.

[0034] First, in step S10, a target data amount is acquired. Specifically, the target data amount acquisition unit 10 (at least one processor) acquires the target data amount for the amount of image data to be transmitted. As an example, the target data amount may be a value calculated from the performance of the communication network measured in advance or the performance assigned to the terminal by the operations manager. The method by which the target data amount acquisition unit 10 acquires the target data amount is not limited. As an example, the target data amount acquisition unit 10 can acquire the target data amount stored in memory.

[0035] Next, in step S11, data to be input is acquired. Specifically, the image data acquisition unit 11 (at least one processor) acquires input data including image data to be input to the estimation unit 12. As an example, the image data acquisition unit 11 can acquire still images or video data from a camera as input. The method by which the image data acquisition unit 11 acquires image data from a camera or the like is not limited. As an example, the image data acquisition unit 11 can acquire this data using wired communication, wireless communication, or a combination thereof. Note that the order of steps S10 and S11 may be reversed, or they may be executed in parallel.

[0036] Next, in step S12, image quality is estimated. Specifically, the estimation unit 12 (at least one processor) estimates appropriate image quality for each of multiple regions included in the image indicated by the input data by referring to a target data amount. The estimated image quality is as described above. As an example, the estimation unit 12 may use a neural network algorithm. As an example, the estimation unit 12 may include a self-attention module. Furthermore, when estimating appropriate image quality, the estimation unit 12 refers to the difference between the image and the image immediately preceding the image. By taking image difference into account, appropriate image quality can be estimated taking into account the magnitude of change in image content.

[0037] Next, in step S13, the input data is replaced with alternative data. Specifically, the replacement unit 13 (at least one processor) generates replaced data by replacing multiple regions of the image included in the input data with alternative data according to the estimated values. As an example, the replacement unit 13 can replace the input data with image data having a quantization parameter set to a larger value than that of the image before replacement.

[0038] Next, in step S14, an evaluation value is derived. Specifically, the evaluation unit 14 (at least one processor) derives the evaluation value by referring to the substituted data. As an example, the evaluation unit 14 can input the substituted data into object detection and derive the evaluation value using the resulting reward value and the difference between the data amount when the substituted data is encoded and the target data amount. Alternatively, the evaluation unit 14 can input the image data substituted with alternative data into VSLAM, compare the obtained output with the correct data to calculate a reward value, and derive the evaluation value by referring to the reward value and the data amount of the alternative data.

[0039] Next, in step S15, the estimation unit 12 is trained. Specifically, the learning unit 15 (at least one processor) trains the estimation unit 12 by referring to the evaluation value. As an example, the learning unit 15 trains the estimation unit so as to increase the evaluation value. In other words, the learning unit 15 updates the estimation parameters of the estimation unit 12 by referring to the evaluation value.

[0040] (Effects of Information Processing Method S1) As described above, the information processing method S1 according to this exemplary embodiment is configured to acquire input data including image data and a target data amount, estimate an appropriate image quality for each region included in the input data while taking into account the magnitude of change in the image content, derive an evaluation value by referring to data replaced with alternative data according to the estimated value, and train the estimation unit 12 by referring to the evaluation value. Therefore, the information processing method S1 according to this exemplary embodiment has the effect of reducing the image data amount to a predetermined transmission data amount or less while retaining necessary information.

[0041] (Configuration of Information Processing Device 2) Next, the configuration of the information processing device 2 according to the present exemplary embodiment 1 will be described with reference to FIG. 3. FIG. 3 is a block diagram showing the configuration of the information processing device 2. The information processing device 2 is an image quality estimation device including a trained estimation unit 22. As shown in FIG. 3, the information processing device 2 includes a target data amount acquisition unit 20, an image data acquisition unit 21, and an estimation unit 22. The target data amount acquisition unit 20 is an embodiment of the target data amount acquisition means set forth in the claims. The image data acquisition unit 21 is an embodiment of the image data acquisition means set forth in the claims. The estimation unit 22 is an embodiment of the estimation means set forth in the claims.

[0042] The target data amount acquisition unit 20 acquires the target data amount. That is, the target data amount acquisition unit 20 may have the same configuration as the target data amount acquisition unit 10 of the information processing device 1 described above, and the target data amount acquired by the target data amount acquisition unit 20 may be the same as the target data amount acquired by the target data amount acquisition unit 10.

[0043] The image data acquisition unit 21 acquires input data including image data. That is, the image data acquisition unit 21 may have the same configuration as the image data acquisition unit 11 of the information processing device 1 described above, and the input data acquired by the image data acquisition unit 21 may be the same as the input data acquired by the image data acquisition unit 11.

[0044] The estimation unit 22 estimates appropriate image quality for each of a plurality of regions included in the image indicated by the input data acquired by the image data acquisition unit 21. The estimation unit 22 is trained by referring to the replaced data obtained by replacing the image regions included in the input data with substitute data according to the estimated values. In other words, the estimation unit 22 can use the estimation unit 12 trained by the learning unit 15 of the information processing device 1 described above.

[0045] When estimating the appropriate image quality, the estimation unit 22 refers to the difference between the image and the image immediately preceding the image. By taking the image difference into consideration, it is possible to estimate the appropriate image quality while taking into account the magnitude of changes in the image content.

[0046] (Effects of Information Processing Device 2) As described above, the information processing device 2 according to this exemplary embodiment is configured to include a target data amount acquisition unit 20, an image data acquisition unit 21, and an estimation unit 22. The target data amount acquisition unit 20 acquires a target data amount, the image data acquisition unit 21 acquires input data including image data, and the estimation unit 22 estimates appropriate image quality for each region included in the image indicated by the input data acquired by the image data acquisition unit 21. Therefore, the information processing device 2 according to this exemplary embodiment can reduce the amount of image data to a predetermined transmission data amount or less while retaining necessary information. Furthermore, the estimation unit 22 can estimate image quality by taking into account the magnitude of changes (differences) in the content of the image. Therefore, a more appropriate image quality can be estimated.

[0047] (Flow of Information Processing Method S2) Next, the flow of the information processing method (image quality estimation method) S2 according to the present exemplary embodiment 1 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of the information processing method S2 executed by the information processing device 2. As shown in Fig. 4, the information processing method S2 includes the following steps.

[0048] First, in step S20, a target data amount is acquired. Specifically, the target data amount acquisition unit 20 acquires the target data amount. For example, the target data amount may be a value calculated from the performance of the communication network measured in advance or the performance assigned to the terminal by the operations manager. The method by which the target data amount acquisition unit 20 acquires the target data amount is not limited. For example, the target data amount acquisition unit 20 can acquire the target data amount stored in memory.

[0049] Next, in step S21, input data (image data) is acquired. Specifically, the image data acquisition unit 21 acquires input data including image data. As an example, the image data acquisition unit 21 can acquire still images or video data from a camera as input. The method by which the image data acquisition unit 21 acquires image data from a camera or the like is not limited. As an example, the image data acquisition unit 21 can acquire this data using wired communication, wireless communication, or a combination of these. Note that the order of steps S20 and S21 may be reversed, or they may be executed in parallel.

[0050] Next, in step S22, image quality is estimated. Specifically, the estimation unit 22 estimates appropriate image quality for each of a plurality of regions included in the image indicated by the input data. Note that the estimation unit 22 may have learned by referring to replaced data obtained by the information processing device 1 by replacing regions included in the input data with alternative data according to the estimated values. Furthermore, when estimating appropriate image quality, the estimation unit 22 refers to the difference between the image and the image immediately preceding the image. By taking the image difference into account, appropriate image quality can be estimated taking into account the magnitude of change in the image content.

[0051] (Effects of Information Processing Method S2) As described above, the information processing method S2 according to this exemplary embodiment is configured to acquire input data including image data and a target data amount, and estimate appropriate image quality for each region included in the image indicated by the input data, taking into account the magnitude of changes in the image content. Therefore, the information processing method S2 according to this exemplary embodiment can reduce the image data amount to a predetermined transmission data amount or less while retaining necessary information. Furthermore, in the estimation step S22, appropriate image quality can be estimated for each region included in the input data, taking into account the magnitude of changes in the image content. This provides the effect of enabling more appropriate image quality to be estimated.

[0052] Second Exemplary Embodiment A second exemplary embodiment of the present disclosure will be described in detail with reference to the drawings. Note that the description of components having the same functions as those described in the first exemplary embodiment will be omitted as appropriate.

[0053] (Configuration of Information Processing Device 3) The information processing device 3 shown in FIG. 5 and described in this exemplary embodiment 2 is a device that estimates appropriate image quality for each of multiple regions included in an image shown in a video (moving image) captured by a video camera (image capture unit 41 described below) mounted on a robot (mobile body 40 described below) according to a target data volume. The robot is an example of a mobile body that is remotely controlled based on the video captured by the video camera. The video captured by the video camera is also transmitted to a remote control center as environmental video. The remote control center analyzes the images included in the video to obtain environmental information such as the robot's position information and obstacles around the robot. Based on this information, appropriate measures such as remotely controlling the robot are taken. In this way, performing video analysis processing at the remote control center reduces the computational load on the mobile body, leading to smaller and less expensive robots.

[0054] The environmental video is transmitted to a remote control center via a wireless communication network, including the Internet. However, demand for wireless networks such as the Internet has been increasing in recent years, leaving the available communication bandwidth insufficient. If the data volume of the environmental video transmitted during autonomous driving is large, there is a risk that the environmental video will be distorted and sufficient control will be impossible if the available communication bandwidth decreases due to a deterioration in communication conditions, etc. Therefore, it is preferable to reduce such risk by transmitting environmental video with a volume smaller than the available communication bandwidth.

[0055] The information processing device 3 is a device that estimates appropriate image quality for each of multiple regions of an image shown by a video in order to suppress the data volume of the video (environmental video) to be transmitted to a remote control center to a target data volume or less. In this embodiment, appropriate image quality is image quality that maximizes an evaluation value (described later) while suppressing the data volume when encoding substitute data (described later) to a target data volume or less.

[0056] As an example, consider VSLAM, a technology that can estimate the location information of a moving object by analyzing images contained in video captured by a camera. VSLAM stores a collection of images with known location information of the moving object, identifies images with known location information that have the same feature points as images contained in the video captured by the camera, and estimates the location information of the images with unknown location information as the location information of the moving object. Reducing the image quality of the entire image can reduce the amount of video data, but this reduces the accuracy of location estimation. For this reason, it is necessary to estimate appropriate image quality for each of the multiple image regions contained in the video.

[0057] (Information Processing System 300) The configuration of an information processing system 300 according to the second exemplary embodiment will be described below. Fig. 5 is a block diagram showing the configuration of the information processing system 300 including an information processing device 3. As shown in Fig. 5, the information processing system 300 includes an information processing device 3 and a mobile object 40. The information processing device 3 includes a control unit 30, a communication unit 37, an output unit 38, and a memory 39. The mobile object 40 includes an imaging unit (video camera) 41 and a transmission unit 42. The imaging unit 41 is an in-vehicle camera mounted on the mobile object 40, and the transmission unit 42 transmits the video captured by the imaging unit 41 to the information processing device 3, for example, via wireless communication.

[0058] The control unit 30 includes a target data amount acquisition unit 31, an image data acquisition unit 32, an estimation unit 33, a replacement unit 34, an evaluation unit 35, and a learning unit 36. These will be described later.

[0059] The communication unit 37 receives the video data transmitted from the transmission unit 42. The communication unit 37 may receive the video data via wireless communication or wired communication.

[0060] The output unit 38 outputs at least a part of the data generated inside the information processing device 3, such as the acquired data, the replaced data, various parameters set in the estimation unit 33, or evaluation values. The output data may be displayed on a display device or the like, or may be transmitted to the outside.

[0061] The memory 39 temporarily or non-temporarily stores various programs, various data, etc. referenced by the control unit 30 .

[0062] Next, a description will be given of the components included in the control unit 30. The target data amount acquisition unit 31, image data acquisition unit 32, estimation unit 33, replacement unit 34, evaluation unit 35, and learning unit 36 ​​of the control unit 30 have functions equivalent to those of the target data amount acquisition unit 10, image data acquisition unit 11, estimation unit 12, replacement unit 13, evaluation unit 14, and learning unit 15 of the information processing device 1.

[0063] The target data amount acquisition unit 31 acquires the target data amount stored in the memory 39 .

[0064] The image data acquisition unit 32 acquires the video transmitted from the video camera. The acquired video is video captured by the imaging unit 41.

[0065] The estimation unit 33 estimates an appropriate image quality for each of a plurality of regions included in each image shown by the video, by referring to the target data amount.

[0066] As an example, the estimation unit 33 estimates image quality using a self-attention module. The self-attention module is a known module that has an algorithm that pays attention to its own intermediate calculation results and determines the data to be read from them. By using the self-attention module, the image analysis technology can learn areas that are important (areas that should have high image quality) when processing an image.

[0067] When estimating the appropriate image quality, the estimation unit 33 refers to the difference between the image and the image immediately preceding the image. By taking the image difference into consideration, it is possible to learn a method for estimating the appropriate image quality while taking into account the magnitude of changes in the image content. The method for training the estimation unit 33 will be described later.

[0068] The replacement unit 34 replaces the image acquired by the image data acquisition unit 32 with alternative data based on the image quality obtained by the estimation unit 33. One example is image data in which each region of the image acquired by the image data acquisition unit 32 is set to the quantization parameter obtained by the estimation unit 33.

[0069] The evaluation unit 35 derives an evaluation value when the replaced image is used. Specifically, the evaluation unit 35 derives the evaluation value by, for example, referring to the output of an image analysis technique that inputs the replaced image, and the difference between the target data amount and the data amount of the replaced image. A portion of the replaced image has a large quantization error. In such a replaced image, if the data amount is equal to or less than the target data amount and does not affect the video analysis, a high evaluation value is assigned. Specific examples of the evaluation unit 35 will be described later.

[0070] The learning unit 36 ​​refers to the evaluation value derived by the evaluation unit 35 and causes the estimation unit 33 to learn. A specific example of the learning unit 36 ​​will be described later.

[0071] In the information processing device 3 described above, the image data acquired by the image data acquisition unit 32 is video data acquired from the video camera 41. However, the image data acquired by the image data acquisition unit 32 is not limited to this, and may be, for example, a virtual simulation image created by a simulation computer that simulates an image from an in-vehicle camera.

[0072] (Effects of Information Processing Device 3) As described above, the information processing device 3 according to the present exemplary embodiment 2 employs a configuration including the control unit 30, the communication unit 37, the output unit 38, and the memory 39. Therefore, in addition to the effects of the information processing device 1 according to the present exemplary embodiment 1, the information processing device 3 according to the present exemplary embodiment has the effect of being able to output and check at least a portion of data generated inside the information processing device 3, such as input data, replaced data, various set parameters, or evaluation values.

[0073] (Specific Example of Learning Method) Next, a method S3 for training the estimation unit 33, which is configured with a neural network including a self-attention module, will be described with reference to the drawings. Fig. 6 is a flowchart showing an example of the flow of the learning method (information processing method) S3 for training the estimation unit 33. The learning method S3 is a method for learning using video (learning data) used in VSLAM.

[0074] First, in step S30, the control unit 30 initializes the parameters of the neural network including the self-attention module.

[0075] Next, in step S31, the target data amount acquiring unit 31 acquires the target data amount. Specifically, the target data amount acquiring unit 31 acquires the target data amount stored in the memory 39.

[0076] Next, in step S32, the image data acquisition unit 32 acquires training data. The training data may be images captured by an on-board camera or virtual simulation images that simulate images from an on-board camera. For example, the training data may be data from an open dataset that includes images with known location information.

[0077] Next, in step S33, the control unit 30 initializes the simulator data and the learning data. In this embodiment, the simulation is performed by VSLAM. Initializing the simulator data means setting a set of images with known position information to an empty set. Additionally, initializing the learning data means setting an image selected from the video data, which is the learning data, to the first image.

[0078] Next, in step S34, the control unit 30 acquires parameters of the neural network including the self-attention module. The parameters acquired during the first simulation are parameters arbitrarily set and input by the user.

[0079] Next, in step S35, the evaluation unit 35 executes a simulation and performs an evaluation. In this embodiment, the executed simulation is an estimation of the position information of a moving object using VSLAM, and the evaluation unit 35 derives an average reward value and an average data amount from the simulation result. A detailed flow of the simulation evaluation will be described later with reference to different drawings. The average reward value and the average data amount are the average values ​​of the reward values ​​and the average data amount derived repeatedly a predetermined number of times in the simulation evaluation described later. Specifically, in the simulation evaluation, as described later, a series of simulation steps are repeated a predetermined number of times. At this time, the reward value and the data amount derived in each simulation step are accumulated. The final cumulative values ​​of the reward value and the data amount are divided by the predetermined number of times to obtain the average reward value and the average data amount.

[0080] Next, in step S36, the evaluation unit 35 calculates an evaluation value. Specifically, the evaluation unit 35 calculates the evaluation value using the average reward value, average data volume, and target data volume calculated in step S35. A detailed flow of the evaluation value calculation will be described later with reference to another drawing.

[0081] Next, in step S37, the control unit 30 updates the parameters of the neural network including the self-attention module. Specifically, the learning unit 36 ​​updates the parameters based on the magnitude of the evaluation value so that the evaluation value becomes larger.

[0082] Next, in step S38, the control unit 30 determines whether the learning data is complete. That is, it determines whether all image data selected from the video data that is the learning data has been replaced and input to the simulator. If it is determined in step S38 that the learning data is complete (step S38: YES), the control unit 30 proceeds to step S39. On the other hand, if it is determined in step S38 that the learning data is not complete (step S38: NO), the control unit 30 returns to step S34. In step S34, the control unit 30 acquires the parameters updated in step S37.

[0083] Next, in step S39, the control unit 30 determines whether or not a predetermined number of simulation evaluations have been performed. If it is determined in step S39 that the predetermined number of simulation evaluations have been performed (step S39: YES), the learning flow ends. On the other hand, if it is determined that the predetermined number of simulation evaluations have not been performed (step S39: NO), the process returns to step S33. In step S34, the control unit 30 acquires the parameters updated in step S37.

[0084] Next, the detailed flow of the simulation evaluation step of step S35 will be described with reference to the drawings. FIG. 7 is a flowchart of the simulation evaluation method S4. In this embodiment, the simulation evaluation uses VSLAM. VSLAM is a device that executes a process on a computer to estimate the vehicle's own position based on images that simulate images captured by an in-vehicle camera. For VSLAM, the open-source ORB-SLAM3 can be used, for example.

[0085] First, in step S40, the control unit 30 acquires input data from the learning data. Specifically, the estimation unit 33 acquires, via the image data acquisition unit 32, an image selected from the video as the learning data and the image immediately before the selected image as input data.

[0086] Next, in step S41, the control unit 30 estimates appropriate image quality for each of the multiple regions of the selected image. Specifically, the estimation unit 33 divides the image into multiple regions and estimates the image quality for each region using a neural network including a self-attention module. Furthermore, when estimating appropriate image quality, the estimation unit 33 refers to the difference between the selected image and the image immediately preceding the selected image. By taking the image difference into account, appropriate image quality can be learned, taking into account the magnitude of changes in the image content.

[0087] Next, in step S42, the control unit 30 replaces the image data with alternative data based on the image quality of each region estimated in step S41. Specifically, each region of the image obtained in step S40 is replaced with image data set to the quantization parameter obtained by the estimation unit 33.

[0088] Next, in step S43, the control unit 30 encodes the replaced image and obtains the amount of encoded data.

[0089] Next, in step S44, the control unit 30 inputs the replaced image into VSLAM and obtains the output result.

[0090] Examples of VSLAM output include the number of matching points (the number of points whose features match those of a set of images with known location information) and estimated location.

[0091] Next, in step S45, the control unit 30 calculates a reward value by comparing the output result of the VSLAM with the correct data stored in the memory 39. Specifically, the reward value is the ratio or difference between the output result of the VSLAM and the correct data.

[0092] For example, if correct data (e.g., the correct position) is distributed as open data, the distributed data is regarded as the correct data. If the correct data is not distributed, the image obtained by the image data acquisition unit 32 (an image with a larger amount of data than the replaced image) can be input to VSLAM and regarded as the correct data.

[0093] Next, in step S46, the evaluation unit 35 adds the data amount obtained in step S43 and the reward value obtained in step S45 to the cumulative values ​​of the data amount and reward value, respectively. The cumulative value means, for example, adding the value obtained in the nth (n is an integer greater than or equal to 2) simulation step to the sum of the values ​​obtained in the 1st to the n-1th simulation steps. In other words, the cumulative value up to the nth step is calculated by adding the value obtained in the nth step to the sum of the values ​​obtained in the 1st to the n-1th steps.

[0094] In step S47, the control unit 30 determines whether a predetermined number of simulations have been completed. The predetermined number of simulations is set to, for example, the same value as the frame rate of the video data. If it is determined in step S47 that the predetermined number of simulations have been completed (step S47: YES), the simulation evaluation step is terminated and the process proceeds to step S48. If it is determined in step S47 that the predetermined number of simulations have not been completed (step S47: NO), the process returns to step S40.

[0095] In step S48, the control unit 30 calculates the average data volume and the average reward value, and then proceeds to step S36 in Fig. 6. Specifically, the average data volume and the average reward value are calculated by dividing the cumulative values ​​of the data volume and reward value obtained in S46 by a predetermined number of times. Thereafter, the process proceeds to step S36 in Fig. 6.

[0096] Next, the detailed flow of the evaluation value calculation step S36 will be described with reference to the drawings. Fig. 8 is a flowchart of the evaluation value calculation S5.

[0097] First, in step S50, the control unit 30 determines whether the average data volume acquired in step S35 of Fig. 6 is greater than the target data volume. If it is determined in step S50 that the average data volume is greater than the target data volume (step S50: YES), the process proceeds to step S51. If it is determined that the average data volume is equal to or less than the target data volume (step S50: NO), the process proceeds to step S52.

[0098] In step S51, the control unit 30 obtains an evaluation value from the average data amount and the target data amount, and then proceeds to step S37 in FIG. 6. As an example, the difference between the target data amount and the average data amount is calculated and set as the evaluation value. In step S51, since the average data amount is greater than the target data amount, the evaluation value calculated from the difference between the target data amount and the average data amount is a negative value. Then, the process proceeds to step S37 in FIG. 6.

[0099] In step S52, the control unit 30 obtains an evaluation value from the average value of the reward values. Then, the process proceeds to step S37 in FIG. 6. As an example, the average value of the reward values ​​is set as the evaluation value. Then, the process proceeds to step S37 in FIG. 6.

[0100] (Effect of learning method) By using the above learning methods S3, S4, and S5, the estimation unit 33 can learn how to estimate appropriate image quality for each image area contained in the video obtained by the image data acquisition unit 32, taking into account the magnitude of changes in the image contained in the video.

[0101] (Information Processing System 400) Next, the configuration of an information processing system 400 according to exemplary embodiment 2 will be described. Fig. 9 is a block diagram showing the configuration of the information processing system 400 including an information processing device 4. As shown in Fig. 9, the information processing system 400 includes an information processing device 4 and a mobile object 60. The information processing device 4 is a device that estimates appropriate image quality for each region included in an image indicated by input data.

[0102] The information processing device 4 includes a control unit 50, a communication unit 54, an output unit 55, and a memory 56. The mobile object 60 includes an imaging unit (video camera) 61 and a transmission unit 62. The communication unit 54, the memory 56, the imaging unit 61, and the transmission unit 62 have the same functions as the communication unit 37, the memory 39, the imaging unit 41, and the transmission unit 42 described in the information processing system 300, and therefore, description thereof will be omitted.

[0103] The control unit 50 includes a target data amount acquisition unit 51, an image data acquisition unit 52, and an estimation unit 53. The target data amount acquisition unit 51 acquires the target data amount via a memory 56. The image data acquisition unit 52 acquires the target video via a communication unit 54. The estimation unit 53 estimates appropriate image quality for each region in the image acquired by the image data acquisition unit 52, taking into account the magnitude of image changes included in the video. The estimation unit 53 can use the estimation unit 33 trained by the information processing system 300 described above.

[0104] The information processing device 4 outputs image quality information indicating the appropriate image quality estimated by the estimation unit 53 from the output unit 55. The image quality information is at least a part of data generated inside the information processing device 4, such as the acquired image data, various parameters set in the estimation unit 53, and information regarding the appropriate image quality estimated by the estimation unit 53. The output from the output unit 55 may be in the form of an image indicating the image quality of each of multiple regions included in the target image, or may be a set of region identification information for distinguishing the multiple regions from each other and the image quality of the corresponding region. The output data may be displayed on a display device or the like, or may be transmitted to the outside.

[0105] The information processing method (estimation method) executed by the information processing device 4 is the same as the information processing method (estimation method) S2 described in the first embodiment, and therefore will not be described here.

[0106] The information processing system 300 and information processing methods S3, S4, and S5 according to the above exemplary embodiment 2 can provide a technology that can estimate appropriate image quality for each region of an image included in a video by taking into account the magnitude of change in the image included in the video. Also, the information processing system 400 and information processing method S2 according to the exemplary embodiment 2 can provide a technology that can estimate appropriate image quality for each region of an image included in a video by taking into account the magnitude of change in the image included in the video. Specifically, it is possible to estimate appropriate image quality for each region of an image included in a video captured by a video camera mounted on a moving object.

[0107] Exemplary Embodiment 3 Next, exemplary embodiment 3 of the present disclosure will be described in detail with reference to the drawings. This exemplary embodiment is one form of an exemplary embodiment of an image analysis system. An image analysis system 500 according to this exemplary embodiment is a system that estimates appropriate image quality for each of multiple regions of an image included in an acquired video, then transmits coded data coded using these estimated image quality, and analyzes the coded data using an image analysis unit. The appropriate image quality in this embodiment is an image quality that maximizes the accuracy of environmental information obtained by image analysis while suppressing the amount of data when coding alternative data, described below, to a target data amount or less.

[0108] (Configuration of Image Analysis System) The configuration of an image analysis system 500 according to exemplary embodiment 3 will be described with reference to Fig. 10. Fig. 10 is a block diagram showing the configuration of the image analysis system 500. As shown in Fig. 10, the image analysis system 500 includes an image processing device 5, an analysis device 6, and a mobile object 90.

[0109] As an example, the image analysis system 500 can be used for remote control of mobile objects such as self-driving cars, robots, and drones. In such remote control, for example, images captured by a camera installed in the mobile object are processed for each image by the image processing device 5, and the processed images are transmitted to a remote location such as a remote control center. The transmitted images are received by an analysis device 6 located at the remote control location, and the images are analyzed by an image analysis unit. This allows the mobile object to be controlled from the remote location by obtaining position information of the mobile object and environmental information such as obstacles.

[0110] The image processing device 5, analysis device 6, and mobile object 90 included in the image analysis system 500 will be described below. The image processing device 5 is an image compression device that compresses images acquired by the mobile object 90 to a predetermined transmission data volume or less while retaining necessary information. The analysis device 6 is an image analysis device that analyzes the compressed images to obtain position information of the mobile object and environmental information such as obstacles.

[0111] (Description of Mobile Body) In the exemplary embodiment 3, a mobile body 90 is illustrated and added to the description. As shown in Fig. 10, the mobile body 90 includes an imaging unit 91 and a communication unit 92. Each of these will be described below.

[0112] The imaging unit 91 may be, for example, an imaging device such as a camera. The imaging unit 91 acquires video. It is preferable that the imaging unit 91 acquires an image including an angle of view in the traveling direction (forward) of the moving body 90, but this does not limit the present exemplary embodiment. As an example, the imaging unit 91 may acquire an image including an angle of view in the opposite direction (rearward) from the traveling direction of the moving body 90, or may acquire an image including an angle of view in a direction perpendicular to the traveling direction of the moving body (lateral).

[0113] The communication unit 92 transmits the image acquired by the imaging unit 91 to the image processing device 5. As an example, the communication unit 92 may be configured to include an antenna and transmit encoded data using wireless communication, but this does not limit the present embodiment, and wired communication may be used, or a combination of wireless communication and wired communication may be used.

[0114] 10, the image processing device 5 includes a target data amount acquisition unit 70, an image data acquisition unit 71, an estimation unit 72, a replacement unit 73, an encoding unit 74, and a transmission unit 75. Each of these will be described below.

[0115] The target data amount acquiring unit 70 acquires a target data amount for the transmission amount of image data. The target data amount acquired by the target data amount acquiring unit 70 is, for example, a value calculated from the performance of the communication network measured in advance or the performance assigned to the terminal by the operations manager. The method by which the target data amount acquiring unit 70 acquires the target data amount is not limited. As an example, the target data amount acquiring unit 70 can acquire a target data amount stored in memory.

[0116] The image data acquisition unit 71 acquires video data (image data). The video data acquired by the image data acquisition unit 71 is video captured by the moving object 90.

[0117] The estimation unit 72 estimates appropriate image quality for each of a plurality of regions of the image included in the video acquired by the image data acquisition unit 71. The estimation unit 72 according to this exemplary embodiment has a configuration similar to the estimation units 22 and 53 described in the first and second exemplary embodiments, and can use the estimation unit 12 trained by the information processing device 1 described above and the estimation unit 33 trained by the information processing system 300 described above.

[0118] When estimating the appropriate image quality, the estimation unit 72 refers to the difference between the image and the image immediately preceding the image. By taking the image difference into consideration, it is possible to estimate the appropriate image quality while taking into account the magnitude of changes in the image content.

[0119] The replacement unit 73 replaces the image acquired by the image data acquisition unit 71 with alternative data in accordance with the image quality obtained by the estimation unit 72. One example of the replacement unit 73 is image data in which each area of ​​the image included in the video acquired by the image data acquisition unit 71 is set to the quantization parameter obtained by the estimation unit 72.

[0120] The encoding unit 74 generates encoded data by encoding the image generated by the replacement unit 73 .

[0121] The transmitting unit 75 transmits the coded data generated by the coding unit 74. As an example, the transmitting unit 75 may be configured to include an antenna and transmit the coded data using wireless communication, but this does not limit the present exemplary embodiment, and the transmitting unit 75 may also use wired communication or a combination of wireless communication and wired communication.

[0122] (Configuration of the Analysis Device) As shown in Fig. 10, the analysis device 6 includes a receiving unit 80, a decoding unit 81, and an analysis unit 82. Each of these will be described below.

[0123] The receiving unit 80 is an embodiment of the receiving means set forth in the claims, and the analyzing unit 82 is an embodiment of the analyzing means set forth in the claims.

[0124] The receiving unit 80 receives the encoded data transmitted from the transmitting unit 75 .

[0125] The decoding unit 81 decodes the coded data received by the receiving unit 80 and converts it into image data.

[0126] Although the specific decoding algorithm used by the decoder 81 does not limit the present exemplary embodiment, it is preferable that the decoding algorithm employed corresponds to the encoding algorithm used by the encoder 74 .

[0127] As a result, each of the multiple regions included in the image decoded by the decoding unit 81 conforms to the image quality determined by the estimation unit 72 .

[0128] The analysis unit 82 analyzes the image decoded by the decoding unit 81 to obtain position information of the moving object and environmental information such as the positions of obstacles around the moving object.

[0129] Although the specific image analysis technique used by the decoding unit 81 is not limited to this exemplary embodiment, as an example, VSLAM, a technique that can estimate the position information of a moving object by analyzing images included in a video, can be used. The environmental information obtained by the image analysis may be displayed on a display device or the like, or may be transmitted to an external device.

[0130] (Effects of the Image Analysis System) According to the image analysis system 500 configured as described above, the image processing device 5 can perform encoding processing with appropriate image quality for each region of an image included in a video captured from a moving object, taking into account the magnitude of changes in the image included in the video. As a result, the image data captured from the moving object and encoded is reduced to the target data volume, making it possible to suitably suppress the transmission load and maintain the accuracy of image analysis.

[0131] (Explanation of Analysis Method in Image Analysis System) Hereinafter, the flow of the analysis method in the image analysis system 500 according to the third exemplary embodiment will be described with reference to FIGS. 11A and 11B.

[0132] 11A and 11B are flow charts showing the flow of an analysis method in the image analysis system 500 according to exemplary embodiment 3. Fig. 11A shows a flowchart S6 showing the flow of processing in the image processing device 5. Fig. 11B shows a flowchart S7 showing the flow of processing in the analysis device 6.

[0133] As shown in FIG. 11A, a flowchart S6 showing the flow in the image processing device 5 according to the third exemplary embodiment includes steps S60 to S65.

[0134] Furthermore, a flow chart S7 in FIG. 11B showing the flow of processing in the analysis device 6 according to the third exemplary embodiment includes steps S70 to S72.

[0135] (Processing Flow in Image Processing Device) Hereinafter, the processing flow in the image processing device 5 will be described step by step.

[0136] First, in step S60, the image processing device 5 acquires a target data amount. Specifically, the target data amount acquisition unit 70 acquires the target data amount. The method by which the target data amount acquisition unit 70 acquires the target data amount is not limited. As an example, the target data amount acquisition unit 70 can acquire the target data amount stored in memory.

[0137] Next, in step S61, the image processing device 5 acquires input data. Specifically, the image data acquisition unit 71 acquires the video transmitted by the communication unit 92 as the input data.

[0138] Next, in step S62, the image processing device 5 estimates appropriate image quality for each region included in the acquired image. Specifically, the estimation unit 72 estimates appropriate image quality for each region included in the image acquired by the image data acquisition unit 71. The processing by the estimation unit 72 in this step is similar to the processing described in the image analysis system 500, and therefore will not be described again.

[0139] Next, in step S63, the image processing device 5 replaces the data with alternative data. Specifically, the replacing unit 73 replaces multiple regions included in the image acquired by the image data acquiring unit 71 with alternative data in accordance with the estimated values ​​obtained by the estimating unit 72, thereby generating replaced data.

[0140] Next, in step S64, the image processing device 5 encodes the replacement data. Specifically, the encoding unit 74 encodes the image data obtained by the replacement unit 73 to generate encoded data.

[0141] Next, in step S65, the image processing device 5 transmits the encoded data. Specifically, the transmission unit 75 transmits the encoded data obtained by the encoding unit 74. The processing by the transmission unit 75 in this step is similar to the processing described in the image analysis system 500, and therefore will not be described again.

[0142] (Explanation of Processing Flow in Analysis Device) Hereinafter, the processing flow in the analysis device 6 will be explained step by step.

[0143] First, in step S70, the analysis device 6 receives the encoded data. Specifically, the receiving unit 80 receives the encoded data transmitted from the transmitting unit 75.

[0144] Next, in step S71, the analysis device 6 decodes the coded data. Specifically, the decoding unit 81 decodes the coded data received by the receiving unit 80 and converts it into image data.

[0145] Next, in step S72, the analysis device 6 analyzes the image data. Specifically, the analysis unit 82 analyzes the image decoded by the decoding unit 81 to obtain position information of the moving object and environmental information such as the positions of obstacles around the moving object. The processing by the analysis unit 82 in this step is similar to the processing described in the image analysis system 500, and therefore will not be described again.

[0146] (Effects of the analysis method in the image analysis system) As described above, the analysis method of the image analysis system 500 can perform encoding processing with appropriate image quality for each region included in a target image captured from a moving object, taking into account the magnitude of changes in the image included in the video. As a result, the image data captured from the moving object and encoded is reduced to the target data volume, which effectively reduces the transmission load and maintains the accuracy of the image analysis.

[0147] FIG. 12 compares the change in bit rate (kb / s) over time of a replaced (compressed) image obtained using a conventional method that evaluates the importance of multiple regions into which an image is divided and sets image quality according to that importance, with a replaced (compressed) image obtained using an information processing device (e.g., information processing device 5) trained using the learning method of this embodiment. In FIG. 12, the dotted line indicates the change in bit rate of the replaced image obtained using the conventional method, and the solid line indicates the change in bit rate of the replaced image obtained using the information processing device 5 of this embodiment. As shown in the figure, the bit rate of this embodiment is kept below the target rate (target data volume) of 200 kb / s, while the bit rate of the conventional method may exceed the target rate. In this case, image distortion occurs, resulting in a decrease in the accuracy of image analysis. The operation of the information processing device 5 of this embodiment will be described with reference to FIGS. 13 and 14.

[0148] 13 shows the change over time in the bit rate (kbps) of a replacement image when a replacement (compressed) image in which the entire image is replaced with the same image quality is transmitted. In FIG. 13, the dotted line indicates the change in the bit rate for a replacement image replaced with high image quality (QP=40), and the solid line indicates the change in the bit rate for a replacement image replaced with low image quality (QP=50). As shown in the figure, the bit rate is higher when a replacement image replaced with high image quality is transmitted than when a replacement image replaced with low image quality is transmitted.

[0149] FIG. 14 compares the change over time in the percentage (%) of high-quality regions (regions with QP=40) between a case where a replaced (compressed) image obtained using a conventional method that evaluates the importance of multiple regions into which an image is divided and sets image quality according to the importance, and a case where a replaced (compressed) image obtained using an information processing device (e.g., information processing device 5) trained using the learning method of this embodiment is transmitted. In FIG. 14 , the dotted line indicates the change in the percentage of high-quality regions in the replaced image using the conventional method, and the solid line indicates the change in the percentage of high-quality regions in the replaced image using the information processing device 5 of this embodiment. As shown in the figure, the percentage of high-quality regions in the conventional method is always constant, while the percentage of high-quality regions in this embodiment changes. In particular, the percentage of high-quality regions decreases during periods where the bit rate significantly increases in FIG. 13 (e.g., from 80 seconds to 100 seconds). In this way, the information processing device 5 of this embodiment predicts an increase in bit rate from image differences and changes the percentage of high-quality regions to keep the bit rate below the target rate.

[0150] FIG. 15 is a diagram evaluating the position estimation error when estimating the position of an imaging device using VSLAM with an image in which the entire image is replaced with the same image quality, and when estimating the position of an imaging device using VSLAM with a replaced image obtained using an information processing device (e.g., information processing device 5) trained using the learning method of this embodiment. The circle (white circle) at the left end (high image quality) of the diagram represents the position estimation accuracy (RMSE, in m; the larger this value, the worse the position estimation accuracy) when position estimation is performed using a replaced image in which the entire image is replaced with a high image quality (QP=40). The circle (gray circle) at the right end (low image quality) of the diagram represents the position estimation accuracy when position estimation is performed using a replaced image in which the entire image is replaced with a low image quality (QP=50). The circle (black circle) in the center of the diagram represents the position estimation accuracy when position estimation is performed using the replaced image of this embodiment. As shown in the figure, the position estimation accuracy is significantly reduced when a replaced image in which the entire image is replaced with a low-quality image is used, whereas the position estimation accuracy of the image replaced in this embodiment is equivalent to that when an image in which the entire image is replaced with a high-quality image is used, despite the fact that the bit rate is reduced.

[0151] [Software Implementation] Some or all of the functions of the information processing devices 1, 2, 3, 4 and image processing device 5 (hereinafter referred to as "information processing device 1, etc.") may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0152] In the latter case, the information processing device 1, etc., is realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 16. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the information processing device 1, etc. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the information processing device 1, etc.

[0153] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0154] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0155] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0156] Furthermore, the functions of each of the devices may be realized by a single processor provided in a single computer, by multiple processors provided in a single computer working in cooperation, or by multiple processors provided in each of multiple computers working in cooperation. Furthermore, the programs for causing each of the devices to realize the functions may be stored in a single memory provided in a single computer, or may be distributed and stored in multiple memories provided in a single computer, or may be distributed and stored in multiple memories provided in each of multiple computers.

[0157] [Appendix 1] The present disclosure includes the technologies described in the following appendices. However, the present disclosure is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0158] (Supplementary Note 1) An information processing device comprising: an image data acquisition means for acquiring image data; a target data amount acquisition means for acquiring a target data amount for the transmission amount of the image data; and an estimation means for estimating image quality for each of a plurality of areas included in an image represented by the image data by referring to the target data amount.

[0159] (Supplementary Note 2) The information processing device according to Supplementary Note 1, wherein the estimation means estimates the image quality of the plurality of regions by referring to the image difference and the target data amount so that the transmission amount is equal to or less than the target data amount.

[0160] (Supplementary Note 3) The information processing device according to Supplementary Note 1 or 2, wherein the estimation means estimates the image quality of each region such that the proportion of high-image-quality regions decreases as the difference between the images increases.

[0161] (Supplementary Note 4) An information processing device comprising: image data acquisition means for acquiring image data; target data amount acquisition means for acquiring a target data amount for the transmission amount of the image data; estimation means for estimating image quality for each of a plurality of regions included in an image represented by the image data by referring to the target data amount; replacement means for replacing the image data for each of the regions with alternative data according to the estimated image quality; evaluation means for deriving an evaluation value by referring to the image data of all regions after replacement; and learning means for updating parameters of the estimation means by referring to the evaluation value. The estimation means may estimate the image quality of the plurality of regions by referring to the difference between the images and the target data amount so that the transmission amount is equal to or less than the target data amount. The estimation means may estimate the image quality of each region so that the proportion of high-quality regions decreases as the difference between the images increases.

[0162] (Appendix 5) The information processing device described in Appendix 4, wherein the evaluation means inputs the image data replaced with the alternative data into a VSLAM, compares the obtained output with correct data to calculate a reward value, and derives an evaluation value by referring to the reward value and the data amount of the alternative data.

[0163] (Supplementary Note 6) An information processing device comprising: image data acquisition means for acquiring image data; target data amount acquisition means for acquiring a target data amount for the transmission amount of the image data; estimation means for estimating image quality for each of a plurality of regions included in an image indicated by the image data by referring to the target data amount; replacement means for replacing the image data for each of the regions with alternative data according to the estimated image quality; encoding means for generating encoded data obtained by encoding the alternative data; and transmission means for transmitting the encoded data. The estimation means may also estimate the image quality of the plurality of regions by referring to the difference between the images and the target data amount so that the transmission amount is equal to or less than the target data amount. The estimation means may also estimate the image quality of each region so that the proportion of high-image-quality regions decreases as the difference between the images increases.

[0164] (Supplementary Note 7) An information processing method in which at least one processor executes an image data acquisition step of acquiring image data, a target data amount acquisition step of acquiring a target data amount for the transmission amount of the image data, and an estimation step of estimating image quality for each of a plurality of regions included in an image represented by the image data by referring to the target data amount. The estimation step may also estimate the image quality of the plurality of regions by referring to a difference between the images and the target data amount so that the transmission amount is equal to or less than the target data amount. The estimation step may also estimate the image quality of each region so that the proportion of high-image-quality regions decreases as the difference between the images increases.

[0165] (Supplementary Note 8) An information processing method in which at least one processor executes the following steps: an image data acquisition step of acquiring image data; a target data amount acquisition step of acquiring a target data amount for the transmission amount of the image data; an estimation step of estimating image quality for each of a plurality of regions included in an image represented by the image data by referring to the target data amount; a replacement step of replacing the image data for each of the regions with alternative data according to the estimated image quality; an evaluation step of deriving an evaluation value by referring to the image data of all the regions after replacement; and a learning step of updating estimation parameters by referring to the evaluation value. The estimation step may also estimate the image quality of the plurality of regions by referring to the difference between the images and the target data amount so that the transmission amount is equal to or less than the target data amount. The estimation step may also estimate the image quality of each region so that the proportion of high-quality regions decreases as the difference between the images increases.

[0166] (Supplementary Note 9) An information processing method in which at least one processor executes the following steps: an image data acquisition step of acquiring image data; a target data amount acquisition step of acquiring a target data amount for the transmission amount of the image data; an estimation step of estimating image quality for each of a plurality of regions included in an image indicated by the image data by referring to the target data amount; a replacement means for replacing the image data for each of the regions with alternative data according to the estimated image quality; an encoding step of generating encoded data by encoding the alternative data; and a transmission step of transmitting the encoded data. The estimation step may also estimate the image quality of the plurality of regions by referring to the image difference and the target data amount so that the transmission amount is equal to or less than the target data amount. The estimation step may also estimate the image quality of each region so that the proportion of high-image-quality regions decreases as the image difference increases.

[0167] (Appendix 10) An information processing program for causing a computer to function as the information processing device described in any one of Appendices 1 to 3, the information processing program causing a computer to function as the image data acquisition means, the target data amount acquisition means, and the estimation means.

[0168] (Supplementary Note 11) The information processing device according to Supplementary Note 4 or 5, further comprising a communication unit that receives video data and an output unit that outputs data. (Supplementary Note 12) An information processing system comprising: a mobile body having an imaging unit and a transmission unit; and the information processing device according to Supplementary Note 11. The imaging unit is an in-vehicle camera mounted on the mobile body, and the transmission unit transmits the video captured by the imaging unit to the information processing device. The communication unit receives the video data transmitted from the transmission unit. The various data output from the output unit may be displayed on a display device or the like, or may be transmitted externally.

[0169] (Supplementary Note 13) An image analysis system including the information processing device according to Supplementary Note 6 and an image analysis device. This image analysis system may further include a mobile body equipped with an imaging device.

[0170] (Appendix 14) An information processing program for causing a computer to function as the information processing device described in Appendix 4 or 5, the information processing program causing a computer to function as the image data acquisition means, the target data amount acquisition means, the estimation means, the replacement means, the evaluation means, and the learning means.

[0171] (Appendix 15) An information processing program for causing a computer to function as the information processing device described in Appendix 6, the information processing program causing a computer to function as the image data acquisition means, the target data amount acquisition means, the estimation means, the replacement means, the encoding means, and the transmission means.

[0172] [Appendix 2] The present disclosure includes the technologies described in the following appendices. However, the present disclosure is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0173] (Supplementary Note 1) An information processing device comprising at least one processor, the at least one processor executing an image data acquisition process for acquiring image data, a target data amount acquisition process for acquiring a target data amount for transmitting the image data, and an estimation process for estimating image quality for each of a plurality of areas included in an image indicated by the image data by referring to the target data amount.

[0174] The information processing device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0175] (Supplementary Note 2) In the estimation process, the at least one processor refers to the image difference and the target data volume, and estimates the image quality of the multiple regions so that the transmission volume is less than or equal to the target data volume.

[0176] (Supplementary Note 3) The information processing device according to Supplementary Note 1, wherein in the estimation process, the at least one processor estimates the image quality of each region such that the proportion of high-image-quality regions decreases as the difference between the images increases.

[0177] (Supplementary Note 4) An information processing device comprising at least one processor, the at least one processor executing an image data acquisition process for acquiring image data, a target data amount acquisition process for acquiring a target data amount for the transmission amount of the image data, an estimation process for estimating image quality for each of a plurality of areas included in an image indicated by the image data by referring to the target data amount, a replacement process for replacing the image data for each of the areas with alternative data according to the estimated image quality, an evaluation process for deriving an evaluation value by referring to image data of all areas after replacement, and a learning process for updating estimation parameters by referring to the evaluation value.

[0178] (Appendix 5) In the evaluation process, the image data replaced with the alternative data is input into a VSLAM, the resulting output is compared with the correct data to calculate a reward value, and an evaluation value is derived by referring to the reward value and the data amount of the alternative data.

[0179] (Supplementary Note 6) An information processing device comprising at least one processor, the at least one processor executing an image data acquisition process for acquiring image data, a target data amount acquisition process for acquiring a target data amount for transmitting the image data, an estimation process for estimating image quality for each of a plurality of areas included in an image indicated by the image data by referring to the target data amount, a replacement process for replacing the image data for each of the areas with alternative data in accordance with the estimated image quality, an encoding process for generating encoded data obtained by encoding the alternative data, and a transmission process for transmitting the encoded data.

[0180] (Supplementary Note 7) An information processing program for causing a computer to execute the image data acquisition process, the target data amount acquisition process, and the estimation process described in Supplementary Note 1.

[0181] This application claims priority based on Japanese Patent Application No. 2024-144458, filed August 26, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0182] 1, 2, 3, 4... Information processing device 5... Image processing device 6... Analysis device 10, 20, 31, 51, 70... Target data amount acquisition unit 11, 21, 32, 52, 71... Image data acquisition unit 12, 22, 33, 53, 72... Estimation unit 13, 34, 73... Replacement unit 14, 35... Evaluation unit 15, 36... Learning unit 30, 50... Control unit 37, 54, 92... Communication unit 38, 55... Output unit 39, 56... Memory 41, 61, 91... Imaging unit 42, 62, 75... Transmission unit 74... Encoding unit 80... Receiving unit 81... Decoding unit 82... Analysis unit 300, 400... Information processing system 500... Image processing system

Claims

1. An information processing device comprising: an image data acquisition means for acquiring image data; a target data amount acquisition means for acquiring a target data amount for the transmission amount of the image data; and an estimation means for estimating image quality for each of a plurality of areas included in an image represented by the image data by referring to the target data amount.

2. The information processing device according to claim 1, wherein said estimation means references the image difference and said target data amount to estimate the image quality of said plurality of regions so that said transmission amount is equal to or less than said target data amount.

3. The information processing device according to claim 1 or 2, wherein said estimation means estimates the image quality of each region so that the proportion of high-image-quality regions decreases as the difference between said images increases.

4. An information processing device comprising: an image data acquisition means for acquiring image data; a target data amount acquisition means for acquiring a target data amount for the transmission amount of the image data; an estimation means for estimating image quality for each of a plurality of areas included in an image indicated by the image data by referring to the target data amount; a replacement means for replacing the image data for each of the areas with alternative data according to the estimated image quality; an evaluation means for deriving an evaluation value by referring to the image data of all areas after replacement; and a learning means for updating parameters of the estimation means by referring to the evaluation value.

5. The information processing device according to claim 4, wherein said estimation means references the image difference and said target data amount to estimate the image quality of said plurality of regions so that said transmission amount is equal to or less than said target data amount.

6. The information processing device according to claim 4 or 5, wherein said estimation means estimates the image quality of each region so that the proportion of high-image-quality regions decreases as the difference between said images increases.

7. The information processing device of claim 4, wherein the evaluation means inputs the image data replaced with the alternative data into VSLAM (Visual Simultaneous Localization and Mapping), compares the obtained output with correct data to calculate a reward value, and derives the evaluation value by referring to the reward value and the data amount of the alternative data.

8. An information processing device comprising: an image data acquisition means for acquiring image data; a target data amount acquisition means for acquiring a target data amount for the transmission amount of the image data; an estimation means for estimating image quality for each of a plurality of areas included in an image indicated by the image data by referring to the target data amount; a replacement means for replacing the image data for each of the areas with alternative data in accordance with the estimated image quality; an encoding means for generating encoded data by encoding the alternative data; and a transmission means for transmitting the encoded data.

9. The information processing device according to claim 8, wherein said estimation means references the image difference and the target data amount to estimate the image quality of said plurality of regions so that said transmission amount is equal to or less than said target data amount.

10. An information processing device according to claim 8 or 9, wherein said estimation means estimates the image quality of each region so that the proportion of high-image-quality regions decreases as the difference between said images increases.

11. An information processing method in which at least one processor executes the following steps: acquiring image data; acquiring a target data volume for transmitting the image data; and estimating image quality for each of a plurality of areas included in an image represented by the image data by referring to the target data volume.

12. An information processing method in which at least one processor performs the following steps: acquire image data; acquire a target data volume for transmitting the image data; estimate image quality for each of multiple areas included in an image represented by the image data by referring to the target data volume; replace the image data for each of the areas with alternative data according to the estimated image quality; derive an evaluation value by referring to the image data of all areas after replacement; and update estimation parameters by referring to the evaluation value.

13. An information processing method in which at least one processor performs the following steps: acquire image data; acquire a target data volume for transmitting the image data; estimate image quality for each of multiple areas included in an image represented by the image data by referring to the target data volume; replace the image data for each of the areas with alternative data according to the estimated image quality; generate encoded data by encoding the alternative data; and transmit the encoded data.

14. An information processing program for causing a computer to execute an image data acquisition process for acquiring image data, a target data amount acquisition process for acquiring a target data amount for the transmission amount of the image data, and an estimation process for estimating image quality for each of a plurality of areas included in an image represented by the image data by referring to the target data amount.

Images