Object recognition rate calculation device, object recognition rate calculation method, and program

Abstract

The purpose of the present disclosure is to obtain a rate of object recognition by operators at a monitoring center.　To this end, an object recognition rate calculation device according to the present disclosure calculates, on the basis of a first parameter pertaining to the video quality of a video acquired from an imaging device that is provided to a moving body and a second parameter pertaining to the transmission status of the video with respect to an operator terminal, a final object recognition rate which indicates the proportion of persons who were actually able to recognize an object when a plurality of persons viewed the video.

Description

Object recognition rate calculation device, object recognition rate calculation method, and program

[0001] The present disclosure relates to a technique for calculating the recognition rate of an object by an operator of a monitoring center that monitors autonomous driving.

[0002] With the development of communication technology, the improvement of sensor accuracy attached to automobiles, and the improvement of autonomous driving control technology, the market scale of autonomous driving of automobiles is expanding. In autonomous driving technology, according to the level of autonomous driving, it is classified into levels from Lv.1 to Lv..5. In Lv.1 and Lv.2, the responsible entity for driving is the driver, and the autonomous driving system executes functions such as lane keeping assist and braking, but it is treated as auxiliary, and the driver is responsible for driving.

[0003] On the other hand, in Lv.3 to Lv.5 of autonomous driving, the autonomous driving system of the automobile basically performs driving, and the responsible entity for driving is the system. In Lv..4 autonomous driving, the driver is not required under specific conditions, and it is possible to reduce accidents caused by mistakes when humans drive, but it is considered that accidents due to system errors may also occur.

[0004] Therefore, as a means to deal with system errors, it has been considered to remotely monitor the video from the monitoring camera of autonomous driving, and the operator of the monitoring center remotely operates the automobile according to the situation (Non-Patent Document 1). Also, in the guidelines of the Ministry of Land, Infrastructure, Transport and Tourism (Non-Patent Document 2), when performing autonomous driving, it is necessary to ensure the safety of operation by appropriately grasping various risks and the like that occur during operation and setting appropriate routes and areas. Therefore, when the operator of the monitoring center monitors the monitoring video, which is the video transmitted from the monitoring camera of autonomous driving, it is necessary to appropriately respond when a dangerous scene occurs during driving. For example, in the case where an object (person, animal, etc.) jumps out onto the road or an object that will collide with the road is placed, it is necessary for the operator to appropriately grasp such objects that hinder driving and perform control such as remotely operating the automobile to stop it.

[0005] On the other hand, in situations where a vehicle is operating autonomously and transmitting surveillance video to a monitoring center in real time, there is a challenge in that the surveillance video may be affected by fluctuations in wireless communication bandwidth, potentially degrading the video quality. If the video quality of the surveillance video deteriorates, the operator at the monitoring center may not be able to recognize objects that could interfere with driving, potentially leading to an accident. Therefore, in autonomous driving, if the video quality does not meet a certain standard, it is considered necessary to take measures such as stopping the vehicle in advance. In order to determine the video quality standard for object recognition, it is necessary to check what the object recognition rate (the percentage of monitors who were actually able to recognize an object when multiple monitors viewed the surveillance video) will be when parameters that affect the quality, such as the video bitrate, are input. As a method for estimating the quality perceived by users from the bitrate of the distributed video, the ITU-T has established Recommendation P.1204 (Non-Patent Literature 3). On the other hand, for object recognition in autonomous driving, the video quality perceived by users of general distributed video is insufficient; a video quality necessary for operators to recognize objects from the surveillance video is required.

[0006] Points to consider at this time include the degradation of image quality due to changes in the video encoding rate and resolution caused by bandwidth fluctuations, and the loss of video data due to packet loss during video data transmission. The degradation of image quality makes the display of objects coarser, making it difficult to recognize objects with the naked eye. In addition, since the lost video data cannot be played back, it is expected that block noise will occur in the missing parts, or the frames containing the missing parts will not be played back, leading to the video becoming a frozen frame, depending on the decoding implementation. If the aforementioned block noise or freeze occurs at the moment an object jumps out, it will become impossible to recognize the object with the naked eye.

[0007] As described above, a technique for estimating object recognition rates that takes into account the degradation of surveillance video quality and the loss of playback video data is necessary.

[0008] Supplementary explanatory materials for the final evaluation of research and development and demonstration projects toward the social implementation of advanced autonomous driving and MaaS, 2021, ITS and Autonomous Driving Promotion Office, Automobile Division, Manufacturing Industries Bureau, Ministry of Industry; Guidelines for passenger transport operators to ensure safety and convenience in unmanned autonomous driving mobility services in limited areas, Road Transport Bureau, Ministry of Land, Infrastructure, Transport and Tourism, p.1204: Video quality assessment of streaming services over reliable transport for resolutions up to 4K, ITU-T, 2020

[0009] When transmitting video in real time, it is expected that the bitrate and resolution will be dynamically varied in response to changes in the wireless bandwidth. Furthermore, it is anticipated that transmission packets of video data may be lost due to a decrease in radio wave propagation strength. Ideally, the number of affected frames should be counted when outputting the playback video during decoding. However, in systems incorporating commercially available decoders, such counting may be difficult. In such cases, it is necessary to consider parameters that indicate the video transmission status, such as the packet loss rate. Additionally, in autonomous driving, the vehicle's speed is not constant but fluctuates. The faster the vehicle's speed, the faster the background and other objects in the surveillance video will move. When there is large (fast) movement in the video, encoding becomes difficult. Therefore, even when transmitting video from the vehicle to the monitoring center at the same bitrate, the video quality may deteriorate, potentially reducing the object recognition rate. Furthermore, at high vehicle speeds, the distance the vehicle travels between object recognition and operator intervention increases. Therefore, it is necessary to recognize objects at an earlier stage, i.e., when objects are displayed smaller on the screen.

[0010] This disclosure is made in view of the circumstances described above and aims to determine the object recognition rate by operators at the monitoring center.

[0011] To achieve the above objective, this disclosure provides an object recognition rate calculation device that calculates a final object recognition rate, which indicates the percentage of people who were actually able to recognize an object when multiple people viewed the video, based on a first parameter relating to the video quality of the video acquired from a camera installed on a mobile body and a second parameter relating to the transmission status of the video to an operator terminal.

[0012] As explained above, this disclosure has the effect of enabling the determination of the object recognition rate by operators at the monitoring center.

[0013] This is an overall configuration diagram of the communication system according to the first embodiment. This is a diagram showing an example of the display screen on the operator terminal of the monitoring center. This is a diagram showing an example of the first object recognition rate model. This is a diagram showing an example of the second object recognition rate model. This is a diagram showing an example of a replacement model which is part of the second object recognition rate model. This is a diagram showing an example of the third object recognition rate model. This is an electrical hardware configuration diagram of each device and terminal according to the embodiment. This is a flowchart showing the process of estimating and presenting the object recognition rate according to the first embodiment. This is a flowchart showing the process of estimating and presenting the object recognition rate according to the first embodiment. This is an overall configuration diagram of the communication system according to the second embodiment. This is an overall configuration diagram of the communication system according to the third embodiment. This is a flowchart showing the process of estimating and presenting the object recognition rate according to the third embodiment. This is a flowchart showing the process of estimating and presenting the object recognition rate according to the third embodiment. This is an overall configuration diagram of the communication system according to the fourth embodiment.

[0014] Embodiments of the present invention will be described below with reference to the drawings.

[0015] ●First Embodiment [System Configuration of the First Embodiment] The general configuration of the communication system 1a of the embodiment will be described using Figure 1. Figure 1 is an overall configuration diagram of the communication system according to the first embodiment. Note that the values ​​shown in Figure 1 (1.2Mbps, 1280x720, 30fps, 5%, 35km / h, 70%) are examples and are not limited to these.

[0016] As shown in Figure 1, the communication system 1a is constructed from devices mounted on a mobile body A capable of automatic movement, such as a vehicle (imaging device 2, speed detection device 3, object recognition rate calculation device 4, and communication device 5), and devices located on the monitoring center B side (operator terminal 6, communication device 8). The imaging device 2, speed detection device 3, object recognition rate calculation device 4, and communication device 5, as well as the operator terminal 6 and communication device 8, each have a communication unit (transmitter unit, receiver unit) for transmitting or receiving data (information) with other devices. Figure 2 shows an example of a display screen from the operator terminal at the monitoring center.

[0017] While mobile unit A primarily operates autonomously, the operator terminal 6 at monitoring center B displays video from one mobile unit A, or from multiple mobile units A as shown in Figure 2. This allows operators at monitoring center B to monitor the movement status of each mobile unit and remotely control a specific mobile unit according to its movement status. This enables operators to remotely avoid accidents if an object or person that could obstruct driving suddenly appears while viewing video from a designated mobile unit.

[0018] Furthermore, there may be multiple instances of each of the following: the imaging device 2, the speed detection device 3, the object recognition rate calculation device 4, the communication device 5, the operator terminal 6, and the communication device 8. In particular, there may be multiple imaging devices 2 to capture images of the front, right side, left side, rear, and interior of the moving vehicle A, respectively.

[0019] In this embodiment, the mobile unit A is assumed to be, for example, a vehicle (automobile) having communication and autonomous driving functions. However, the mobile unit A is not limited to automobiles, and may also be agricultural machinery such as a tractor having communication and autonomous driving functions. Furthermore, the mobile unit may be a ship, aircraft, or the like, in addition to a vehicle.

[0020] The imaging device 2 has a surveillance camera mounted on the mobile body A and an encoder for compressing video data. It encodes video data obtained by imaging the outside or inside of the mobile body A and transmits it to the communication device 5. Furthermore, the imaging device 2 has a function for extracting video transmission parameters such as bitrate, resolution, and frame rate. It extracts the bitrate, resolution, and frame rate of the video from the encoded video data and transmits bitrate information indicating the bitrate, resolution information indicating the resolution, and frame rate information indicating the frame rate, respectively, to the object recognition rate calculation device 4 every unit time t (for example, 1 second). Note that the imaging device 2 and the encoder (device) may be separate entities.

[0021] The velocity detection device 3 detects the speed of the moving object A (for example, h in kilometers per hour) and transmits velocity information indicating the detected speed to the object recognition rate calculation device 4. This velocity information is also transmitted every unit of time t, similar to the bitrate information, resolution information, and frame rate information.

[0022] The object recognition rate calculation device 4 has a calculation unit 9 that calculates the object recognition rate based on the information acquired from the imaging device 2 (bitrate information, resolution information, and frame rate information), packet loss rate information acquired from the communication device 8 (transmission information acquisition unit 7), which will be described later, and speed information acquired from the speed detection device 3. This calculation unit 9 is a function that is realized by the execution of a program. In the object recognition rate calculation device 4, the calculation is performed using information transmitted from the imaging device 2, the speed detection device 3, and the operator terminal 6 (transmission information acquisition unit 7) at the same time. For example, the calculation unit 9 adds timestamp information to the transmitted information so that even if a delay occurs in the communication network 100, the object recognition rate can be calculated based on information at the same time.

[0023] The "object recognition rate" indicates the percentage of supervisors who were able to actually recognize an object when multiple supervisors viewed the surveillance video. For example, if there are 100 supervisors with varying levels of visual acuity, and 50 of them were able to recognize an object shown in the surveillance video, the object recognition rate is 50%. The object recognition rate calculation device 4 then transmits the calculated object recognition rate information to the operator terminal 6 at the monitoring center B via the communication device 5. As a result, the operator terminal 6 displays the video from each moving object (surveillance video) along with the object recognition rate information (e.g., 70%) for each moving object, as shown in Figure 2. The method for calculating the object recognition rate will be explained in detail later.

[0024] Communication device 5 can communicate data with operator terminal 6 via communication network 100 and communication device 8. Communication network 100 is constructed using mobile networks, the internet, etc. The internet also includes space internet, which is accessed via space using artificial satellites, etc.

[0025] Operator terminal 6 is a PC (Personal Computer) or similar terminal used by operators at monitoring center B. Operator terminal 6 has a decoder for decoding encoded video data.

[0026] The transmission information acquisition unit 7, a function of the communication device 8, calculates the packet loss rate as an example of a transmission status parameter that shows the impact of data loss on the video transmitted to the operator terminal 6 installed in the monitoring center B. This packet loss rate information is calculated every unit time t and transmitted from the communication device 8 to the object recognition rate calculation device 4 via the communication device 5.

[0027] The communication device 8 can communicate data with the object recognition rate calculation device 4 via the communication network 100 and the communication device 5.

[0028] With the above configuration, bitrate information, resolution information, and frame rate information transmitted from the imaging device 2, as well as speed information acquired by the speed detection device 3, are sent to the object recognition rate calculation device 4. Furthermore, the video data transmitted from the imaging device 2 is sent to the operator terminal 6 via the communication devices 5 and 8. The packet loss rate information transmitted from the communication device 8 (transmission information acquisition unit 7) is sent to the object recognition rate calculation device 4 via the communication device 5. The object recognition rate information, which indicates the object recognition rate calculated by the object recognition rate calculation device 4, is transmitted from the object recognition rate calculation device 4 and sent to the operator terminal 6 via the communication devices 5 and 8.

[0029] [Detailed Calculation Method for Object Recognition Rate] Next, Figures 3 to 6 illustrate the method for calculating the object recognition rate calculated by the object recognition rate calculation device 4. Here, we will explain a method for determining the object recognition rate Z using the bitrate b, resolution r, frame rate f, speed v, and packet loss rate p of the video input to the device.

[0030] <Relationship between bitrate, resolution, frame rate and object recognition rate> Video quality is correlated with bitrate; the higher the bitrate, the higher the video quality and the higher the object recognition rate. The effects of resolution and frame rate on the same bitrate vary depending on the bitrate and encoding method. Generally, at high bitrates, lower resolution results in higher video quality and a higher object recognition rate. For the same bitrate, higher frame rates result in smoother motion, but the amount of data allocated per frame decreases, which tends to lower the image quality of each frame. Generally, at low bitrates, lower frame rates tend to result in a higher object recognition rate. Figure 3 shows the relationship between bitrate, resolution, frame rate and object recognition rate.

[0031] Figure 3 shows the first object recognition rate model. The first object recognition rate model shows the relationship between the object recognition rate and the first parameters related to the video quality of the video (bitrate, resolution, and frame rate, etc.). Specifically, the first object recognition rate model shows the relationship between the bitrate and the object recognition rate for each resolution and frame rate at the same speed of the moving object A. Although three curves are shown in Figure 3, this is only due to the constraints of the drawing, and in reality, the first object recognition rate model has all possible patterns of resolution and frame rate. Furthermore, bitrate, resolution, and frame rate are examples of the first parameters related to the video transmission parameters or video quality of the video.

[0032] While the object recognition rate increases with higher bitrates, once a certain bitrate is reached, the video quality almost reaches its upper limit, and the object recognition rate hardly changes even at higher bitrates. Therefore, the object recognition rate calculation device 4 calculates the object recognition rate using an object recognition rate model in which the object recognition rate increases with increasing bitrate, but the object recognition rate hardly changes above a predetermined bitrate.

[0033] The object recognition rate calculation device 4 has a calculation unit (first calculation unit) that calculates the object recognition rate based on video transmission parameters using a first object recognition rate model.

[0034] <Relationship between packet loss rate and object recognition rate for missing frames> The more missing frames there are, the later it is possible to recognize an object, and the lower the object recognition rate becomes. Figure 4 shows the relationship between the number of missing frames and the object recognition rate.

[0035] Figure 4 shows the second object recognition rate model. The second object recognition rate model shows the relationship between the first parameter (bitrate, resolution, or frame rate) and the second parameter (number of missing frames, etc.) and the object recognition rate. Specifically, the second object recognition rate model shows the relationship between the number of missing frames (horizontal axis), which is the second parameter related to the video playback status on the operator terminal 6, and the object recognition rate (vertical axis) for each frame rate, with the same parameters (same bitrate and resolution, and the same travel speed of the moving object A).

[0036] Note that while Figure 4 shows two curves, this is merely a representation due to the constraints of the drawing; in reality, the second object recognition rate model encompasses all possible frame rate patterns.

[0037] Furthermore, the second object recognition rate model may show the relationship between the number of missing frames (horizontal axis) and the object recognition rate (vertical axis) for each bitrate, under the same parameters (same frame rate and resolution, and the same speed of moving object A). Alternatively, the second object recognition rate model may show the relationship between the number of missing frames (horizontal axis) and the object recognition rate (vertical axis) for each resolution, under the same parameters (same bit rate and frame rate, and the same speed of moving object A). Note that the relationship between the vertical and horizontal axes may be reversed.

[0038] Furthermore, the object recognition rate begins to decrease when the number of missing frames exceeds a certain level, and the object recognition rate converges to 0 when the number of missing frames and the number of frames per unit time (= frame rate × t) are equal, i.e., when all frames are missing. Therefore, the object recognition rate calculation device 4 calculates the object recognition rate using an object recognition rate model (second object recognition rate model) in which the object recognition rate decreases as the number of missing frames increases and becomes 0 when the packet loss rate and the number of frames per unit time are the same value. In this case, the lower the frame rate, the longer the time length for a single missing frame, so for the same number of missing frames, the object recognition rate decreases as the frame rate decreases.

[0039] Note that the number of missing frames is an example of a replacement parameter in which the reproduction status parameter of the video or the second parameter related to the reproduction status of the video is replaced.

[0040] The object recognition rate calculation device 4 has a calculation unit 9 (second calculation unit) that calculates the object recognition rate based on the reproduction status parameter information using the second object recognition rate model.

[0041] (Relationship between packet loss rate and number of missing frames) The number of missing frames increases as the packet loss rate increases. As a result, the object recognition rate decreases as shown in FIG. 4. The relationship between the packet loss rate and the number of missing frames is shown in FIG. 5. FIG. 5 is a diagram showing an example of a replacement model that is part of the second object recognition rate model (included in the second object recognition rate model). The replacement model shows the relationship between the packet loss rate and the number of missing frames at the same bit rate, resolution, and frame rate. As the packet loss rate increases, the number of missing frames increases. When the packet loss rate is at 100%, all frames of the video are missing, so the number of missing frames is the same as the number of frames per unit time (= frame rate × t). Note that the replacement model may be referred to as a conversion model.

[0042] Note that the packet loss rate is an example of a transmission status parameter related to the transmission status of the video or a second parameter related to the transmission status of the video.

[0043] As described above, the object recognition rate calculation device 4 has a calculation unit 9 (second calculation unit) that calculates the object recognition rate based on the packet loss rate information using the second object recognition rate model including the replacement model. This calculation unit 9 is a function realized by executing a program.

[0044] <Relationship with Object Recognition Rate with Respect to Speed>Also, by taking the speed information of the moving body A as an input, the object recognition rate calculation device 4 can also handle outputs with different recognition rates when the speed of the moving body A is different. FIG. 6 is a diagram showing a third object recognition rate model. The third object recognition rate model shows the relationship between the first parameter (bit rate, resolution, or frame rate) and the speed of the moving body A and the object recognition rate. Specifically, the third object recognition rate model is a diagram showing an example of a model (third object recognition rate model) that shows the relationship between the bit rate (horizontal axis) and the object recognition rate (vertical axis) for each speed of the moving body A at the same resolution and frame rate. When the speed of the moving body A is high, the movement of the background and objects in the video becomes large (fast), making video encoding difficult. Therefore, even when the video is delivered at the same bit rate, the quality of the video deteriorates and the object recognition rate is considered to decrease. Also, when the speed of the moving body A per hour is high, the distance that the moving body A moves before a person such as an operator recognizes an object becomes long. Therefore, since the object must be recognized from a farther distance, this also affects the decrease in the object recognition rate. The object recognition rate calculation device 4 calculates the object recognition rate using these properties.

[0045] [[ID=ģ]] Note that in FIG. 6, although three curves are shown, due to drawing constraints, only the three curves are explicitly shown. In reality, the third object recognition rate model has every pattern of hourly speed.

[0046] Also, the third object recognition rate model may show the relationship between the frame rate (horizontal axis) and the object recognition rate (vertical axis) for each speed of the moving body A at the same parameter (same bit rate and resolution). Alternatively, the third object recognition rate model may show the relationship between the resolution (horizontal axis) and the object recognition rate (vertical axis) for each speed of the moving body A at the same parameter (same frame rate and bit rate).

[0047] The object recognition rate calculation device 4 has a calculation unit 9 (third calculation unit) that calculates the object recognition rate based on the speed (moving speed of the moving body A) using the third object recognition rate model.

[0048] Based on the above, the object recognition rate calculation device 4 uses a first object recognition rate model that shows the relationship between the first parameter and the object recognition rate, and calculates the object recognition rate based on the first parameter. Furthermore, the object recognition rate calculation device 4 uses a second object recognition rate model that shows the relationship between the first parameter, the second parameter and the object recognition rate, and calculates the object recognition rate based on the first parameter and the second parameter. Furthermore, the object recognition rate calculation device 4 uses a third object recognition rate model that shows the relationship between the first parameter and velocity and the object recognition rate, and calculates the object recognition rate based on the first parameter and velocity.

[0049] <Method for Calculating Object Recognition Rate by Object Recognition Rate Calculation Device> This section describes the method for calculating the object recognition rate performed by the object recognition rate calculation device 4, specifically the method for determining the object recognition rate Z when using the video bitrate b, resolution r (=height × width), frame rate f, packet loss rate p, and speed v. The formula for calculating the object recognition rate Z is expressed as an equation f(b, r, f, p, v) with bitrate information, resolution information, frame rate information, packet loss rate information, and speed information as inputs. As shown in Figure 6, the function f(b, r, f, p, v) has the characteristic that the output object recognition rate decreases as the speed increases. Taking this into account, f(b, r, f, p, v) is calculated using one of the formulas (Equation 1) to (Equation 4). Note that f(b, r, f, p, v) is defined in the range of 0 to 1, and in each of the following equations (Equation 1) to (Equation 4), if f(b, r, f, p, v) is 0 or less, the object recognition rate is defined as 0, and if f(b, r, f, p, v) is 1 or greater, the object recognition rate is defined as 1.

[0050]

[0051]

[0052]

[0053] Here, in (Equation 1) to (Equation 4), c1 to c 13is a coefficient that differs for each formula, and t is the unit time [seconds] for calculating the information. Also, log is a logarithm with base e, which is the base of the natural logarithm. Here, by setting the above coefficients to values ​​according to conditions such as the encoding scheme, GoP length (and GoP structure), decoding implementation in case of data loss (indicating whether it is block noise or freeze), size of the object, size of the display, size of the object displayed on the display, weather conditions during driving, coefficient of friction of the road surface, and data transmission delay between mobile body A and monitoring center B, it is possible to estimate the object according to the driving conditions and monitoring environment.

[0054] [Hardware Configuration] Next, the electrical hardware configuration of the object recognition rate calculation device 4, communication device 5, operator terminal 6, and communication device 8 will be described using Figure 7. Figure 7 is an electrical hardware configuration diagram of each device and terminal according to the embodiment.

[0055] The object recognition rate calculation device 4, as a computer, includes a CPU 101, ROM 102, RAM 103, SSD 104, external device connection interface 105, network interface 106, display 107, input device 108, media interface 109, and bus line 110, as shown in Figure 7.

[0056] Of these, the CPU 101 controls the operation of the entire object recognition rate calculation device 4. The ROM 102 stores programs used to drive the CPU 101, such as the IPL. The RAM 103 is used as the work area for the CPU 101.

[0057] The SSD 104 reads or writes various types of data according to the control of the CPU 101. Alternatively, an HDD (Hard Disk Drive) may be used instead of the SSD 104.

[0058] External device connection I / F 105 is an interface for connecting various external devices. These external devices include displays, speakers, keyboards, mice, USB memory sticks, and printers.

[0059] The network interface 106 is an interface for data communication via the communication network 100.

[0060] The display 107 is a type of display means, such as a liquid crystal display or an organic EL (electroluminescence) display, which displays various images.

[0061] The input device 108 is an example of an input means for receiving input operations such as characters, numbers, and various instructions, and may be a button, switch, keyboard, or pointing device.

[0062] The media interface 109 controls the reading or writing (storage) of data to or from a recording medium 109m, such as flash memory. The recording medium 109m includes DVDs, Blu-ray Discs (registered trademarks), and the like.

[0063] The bus line 110 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 101 shown in Figure 7.

[0064] Since the communication device 5, operator terminal 6, and communication device 8 have the same configuration as the object recognition rate calculation device 4, their explanation will be omitted.

[0065] [Processing or Operation of the Communication System According to the First Embodiment] Next, the processing or operation of each device in the communication system 1a will be explained using Figures 8 and 9. Figures 8 and 9 are flowcharts illustrating the process of estimating and presenting the object recognition rate according to the first embodiment.

[0066] S11: The imaging device 2 captures video data from the outside or inside of the moving object A, then compresses the video data using an encoder and transmits it to the communication device 5.

[0067] S12: The communication device 5 transmits video data to the operator terminal 6 via the communication network 100 and the communication device 8.

[0068] S13: The imaging device 2 uses the encoding information of the video data to extract the bitrate, resolution, and frame rate of the video, and transmits the bitrate information (e.g., 5Mbps, 10Mbps), resolution information (e.g., 640x360, 1280x720), and frame rate information (e.g., 10fps, 30fps) to the object recognition rate calculation device 4.

[0069] S14: The speed detection device 3 detects the speed of the moving object A and transmits speed information indicating this speed to the object recognition rate calculation device 4.

[0070] S15: The transmission information acquisition unit 7 extracts the packet loss rate during video transmission to the operator terminal 6, that is, the packet loss rate (for example, 0%, 5%) for the video transmission information to be sent to the operator terminal 6.

[0071] S16: The communication device 8 transmits packet loss rate information, which is extracted by the transmission information acquisition unit 7, to the object recognition rate calculation device 4 via the communication network 100 and the communication device 5.

[0072] S17: The object recognition rate calculation device 4 uses the first to third object recognition rate models and calculates the final object recognition rate (for example, 70%) based on bitrate information, resolution information, frame rate information, packet loss rate information, and speed information.

[0073] Here, we will specifically explain how to calculate the object recognition rate using the first to third object recognition rate models. For example, the calculation unit 9 uses the first object recognition rate model (Figure 3) at a reference speed (km / h) of 30 km / h to calculate a predetermined object recognition rate based on a bitrate of 1.2 Mbps at a resolution (1280 × 720) and a frame rate of 30 fps. Then, if the speed indicated by the speed information is 35 km / h, the calculation unit 9 uses the third object recognition rate model (Figure 6) to adjust the predetermined object recognition rate by selecting a curve for 35 km / h that is closer to 50 km / h from the curve for the reference speed of 30 km / h. After that, the calculation unit 9 uses a substitution model (Figure 5) to substitute the packet loss rate with the number of missing frames, then uses the second object recognition rate model (Figure 4) to select a curve for a frame rate of 30 fps, and calculates the final object recognition rate based on the number of missing frames (5 frames) indicated by the number of missing frames information. The object recognition rate calculation device 4 then transmits object recognition rate information indicating the final object recognition rate to the communication device 5.

[0074] S18: The communication device 5 transmits the object recognition rate information received from the object recognition rate calculation device 4 to the operator terminal 6 via the communication network 100 and the communication device 8.

[0075] As a result, as shown in Figure 2, the operator terminal 6 decodes the video data sent from each mobile object by processing S12, associates it with the object recognition rate information related to the video of each mobile object sent from the communication device 5 by processing S18, and displays it on the display by splitting the screen. In order to associate the video with the object recognition rate information, identification information such as an ID may be included in both the video data and the object recognition rate information. In this case, the identification information is added to the video data and bitrate information output by the shooting device 2. The object recognition rate calculation device 4 then adds the identification information added to the bitrate information acquired from the shooting device 2 to the object recognition rate information and outputs it.

[0076] S19: If the movement of mobile body A is not completed (NO), the process returns to S11. On the other hand, if the movement of mobile body A is completed (YES), the processes shown in Figures 8 and 9 are completed.

[0077] [Effects of the First Embodiment] As described above, according to the first embodiment, the object recognition rate by the operator of the monitoring center B can be determined and presented. As a result, the operator terminal 6 of the monitoring center B displays the monitoring video from each moving object in association with the object recognition rate information of this monitoring video, as shown in Figure 2, thus eliminating the situation in which the operator does not notice that the video quality of the monitoring video has deteriorated.

[0078] ●Second Embodiment [System Configuration of the Second Embodiment] Next, the communication system 1b according to the second embodiment will be described using Figure 10. Figure 10 is an overall configuration diagram of the communication system according to the second embodiment. The second embodiment differs from the configuration of the first embodiment in that it does not use a speed detection device 3. That is, the second embodiment shows an example where the speed of the mobile body A is fixed (constant). Note that the values ​​shown in Figure 10 (1.2 Mbps, etc.) are just examples, as with Figure 1, and are not limiting. Also, devices or terminals that are the same as in Figure 1 are denoted by the same reference numerals and their descriptions are omitted.

[0079] <Method for Calculating Object Recognition Rate by Object Recognition Rate Calculation Device> The method for calculating the object recognition rate performed by the object recognition rate calculation device 4 is explained below. Here, we will explain a method for determining the object recognition rate Z using the video bitrate b, resolution r (=height × width), frame rate f, and packet loss rate p. The formula for calculating the object recognition rate Z is expressed as an equation f(b, r, f, p) with bitrate information, resolution information, frame rate information, and packet loss rate information as inputs. The function f(b, r, f, p) has the characteristics of the model shown in Figures 3 and 4. Based on this, f(b, r, f, p) is calculated using one of the following equations (Equation 5) to (Equation 8). Note that f(b, r, f, p) is defined in the range of 0 to 1, and in each of the following equations, if f(b, r, f, p) is 0 or less, the object recognition rate is defined as 0, and if f(b, r, f, p) is 1 or greater, the object recognition rate is defined as 1. Under conditions where resolution and frame rate remain constant, formulas that do not take the relevant parameters as inputs may be used. Furthermore, the formulas used to calculate the object recognition rate are not limited to those shown below.

[0080]

[0081]

[0082]

[0083] Here, in (Equation 5) to (Equation 8), c1 to c 10 is a coefficient that differs for each formula, and t is the unit time [seconds] for calculating the information. Also, log is a logarithm with base e, which is the base of the natural logarithm. Here, by setting the above coefficients to values ​​according to conditions such as the encoding scheme, GoP length (and GoP structure), decoding implementation in case of data loss (indicating whether it is block noise or freeze), size of the object, size of the display, size of the object displayed on the display, weather conditions during driving, coefficient of friction of the road surface, and data transmission delay between mobile body A and monitoring center B, it is possible to estimate the object according to the driving conditions and monitoring environment.

[0084] [Processing or Operation of the Communication System According to the Second Embodiment] In the second embodiment, as a difference from the flowchart showing the processing of the first embodiment shown in Figures 8 and 9, S14 is not performed, and in S17, the object recognition rate calculation device 4 uses the first to third models to calculate the object recognition rate based on bitrate information, resolution information, frame rate information, and packet loss rate information. In the second embodiment, the fourth model is not used, so the calculation unit 9 calculates the final object recognition rate using the second object recognition rate model without adjusting the object recognition rate, for example, while keeping the reference speed at 30 km / h.

[0085] [Effects of the Second Embodiment] As described above, according to the second embodiment, even when the travel speed is constant or unknown, the recognition rate of objects by the operator of the monitoring center B can be determined and presented.

[0086] ●Third Embodiment [System Configuration of the Third Embodiment] Next, the communication system 1c according to the third embodiment will be described with reference to Figures 11 to 13. Figure 11 is an overall configuration diagram of the communication system according to the third embodiment. Note that the values ​​shown in Figure 11 (1.2 Mbps, etc.) are examples, as with Figure 1, and are not limited to these. Also, the same devices or terminals as in Figure 1 are denoted by the same reference numerals and their descriptions are omitted.

[0087] In the communication system 1c of Figure 11, the object recognition rate calculation device 4, which was installed on the mobile body A side in the first embodiment (Figure 1), is installed on the monitoring center B side. The video data encoded by the imaging device 2 is transmitted to the operator terminal 6 via the communication devices 5 and 8. However, in this embodiment, the operator terminal 6, not the imaging device 2, has the bitrate extraction function. In addition, the speed information indicating the speed of mobile body A detected by the mobile body A speed detection device 3 is transmitted to the object recognition rate calculation device 4 via the communication devices 5 and 8. This transmission is performed every unit time t (for example, 1 second).

[0088] The calculation unit 9 of the object recognition rate calculation device 4 calculates the current (final) object recognition rate based on bitrate information, resolution information, frame rate information, packet loss rate information, and speed information, similar to the first embodiment. The object recognition rate calculation device 4 then transmits this object recognition rate information to the operator terminal 6.

[0089] [Processing or Operation of the Communication System According to the Third Embodiment] Next, the processing or operation of each device in the communication system 1c will be explained using Figures 12 and 13. Figures 12 and 13 are flowcharts illustrating the process of estimating and presenting the object recognition rate according to the third embodiment.

[0090] S21: The imaging device 2 takes a picture of the outside or inside of the moving object A to obtain video data, then compresses the video data using an encoder and transmits it to the communication device 5.

[0091] S22: The speed detection device 3 detects the current speed of the moving object A and transmits speed information indicating this speed to the communication device 5.

[0092] S23: The communication device 5 transmits video data to the operator terminal 6 via the communication network 100 and the communication device 8, and also transmits speed information to the object recognition rate calculation device 4.

[0093] S24: The operator terminal 6 uses the encoding information of the video data to extract the bitrate, resolution, and frame rate of the video, and transmits the bitrate information (e.g., 5Mbps, 10Mbps), resolution information (e.g., 640x360, 1280x720), and frame rate information (e.g., 10fps, 30fps) to the object recognition rate calculation device 4.

[0094] S25: The transmission information acquisition unit 7 extracts the packet loss rate during video transmission to the operator terminal 6, that is, the packet loss rate for the video transmission information received by the operator terminal 6. The communication device 8 then transmits this packet loss rate information (for example, 0%, 5%) to the object recognition rate calculation device 4.

[0095] S26: The object recognition rate calculation device 4 uses the models shown in Figures 3 and 4 to calculate the object recognition rate (for example, 70%) based on bitrate information, resolution information, frame rate information, packet loss rate information, and speed information. The object recognition rate calculation device 4 then transmits object recognition rate information to the operator terminal 6.

[0096] As a result, as shown in Figure 2, the operator terminal 6 decodes the video data sent from each moving object by processing S23, associates it with the object recognition rate information related to the video of each moving object sent from the object recognition rate calculation device 4 by processing S26, and displays it on the display by splitting the screen. In this case as well, identification information such as an ID may be included in both the video data and the object recognition rate information in order to associate the video with the object recognition rate information. In this case, the shooting device 2 adds the identification information to the video data it outputs. Then, when the operator terminal 6 extracts the bitrate, it adds the same identification information to the bitrate information. The object recognition rate calculation device 4 adds the identification information added to the bitrate information obtained from the operator terminal 6 to the object recognition rate information and transmits it to the operator terminal 6.

[0097] S27: If the movement of mobile body A is not completed (NO), the process returns to S21. On the other hand, if the movement of mobile body A is completed (YES), the processes shown in Figures 12 and 13 are completed.

[0098] [Effects of the Third Embodiment] As described above, the third embodiment provides the same effects as the first embodiment. Furthermore, the third embodiment requires fewer devices on the mobile body A side compared to the first embodiment, thus reducing the initial costs for receiving remote monitoring and remote control services from this communication system 1c.

[0099] ●Fourth Embodiment Next, the communication system 1d according to the fourth embodiment will be described using Figure 14. Figure 14 is an overall configuration diagram of the communication system according to the fourth embodiment. Note that the values ​​shown in Figure 14 (1.2 Mbps, etc.) are examples, as with Figure 1, and are not limited to these. Also, the same reference numerals are used for devices or terminals as in Figure 1, and their descriptions are omitted.

[0100] In the fourth embodiment, as a difference from the flowchart showing the processing of the third embodiment shown in Figures 12 and 13, processing S22 is not performed. Also, in processing S26, the object recognition rate calculation device 4 uses the first object recognition rate model (Figure 3) and the second object recognition rate model (Figure 4), which includes the substitution model (Figure 5), but, similar to the second embodiment, it does not use the third object recognition rate model (Figure 6) and calculates the object recognition rate based on bitrate information, resolution information, frame rate information, and packet loss rate information. The fourth embodiment differs from the third embodiment (Figure 11) in that the speed detection device 3 is not used. That is, the fourth embodiment shows an example where the speed of the moving object A is fixed (constant).

[0101] [Effects of the Fourth Embodiment] As described above, the fourth embodiment provides the same effects as the second embodiment. Furthermore, in the fourth embodiment, since fewer devices are required on the mobile body A side compared to the second embodiment, the initial cost for receiving remote monitoring and remote control services using this communication system 1d can be reduced.

[0102] [Supplement] The above embodiment is not limited to the above-described embodiment, and various modifications and applications are possible, for example, as shown below.

[0103] (1) In the above embodiment, the imaging device 2 or operator terminal 6 extracts the bitrate from the video data and transmits the bitrate information, and the object recognition rate calculation device 4 calculates the object recognition rate using the bitrate information, but it is not limited to this. For example, the imaging device 2 or operator terminal 6 uses bitrate information, resolution information, and frame rate information, but it may use only some of this information to calculate the object recognition rate. Note that bitrate information, resolution information, and frame rate information are examples of video quality parameters, and other parameters may be used. Also, packet loss rate is an example of a parameter of video transmission status, and other parameters may be used.

[0104] (2) Each device can be realized by a computer and a program, and this program can be recorded on a (non-temporary) recording medium or provided via a communication network such as the Internet.

[0105] (3) The CPU 101 may be a single unit or multiple units.

[0106] 1a, 1b, 1c, 1d Communication system A Mobile object B Monitoring center 2 Imaging device 3 Speed ​​detection device 4 Object recognition rate calculation device 5 Communication device 6 Operator terminal 7 Transmission information acquisition unit 8 Communication device 9 Calculation unit 100 Communication network

Claims

1. An object recognition rate calculation device that calculates a final object recognition rate, which indicates the percentage of people who were actually able to recognize an object when multiple people viewed the video, based on a first parameter relating to the video quality of the video acquired from a camera installed on a mobile body, and a second parameter relating to the transmission status of the video to an operator terminal.

2. An object recognition rate calculation device according to claim 1, which uses a first object recognition rate model showing the relationship between the first parameter and the object recognition rate, and a second object recognition rate model showing the relationship between the second parameter and the object recognition rate, and calculates the final object recognition rate based on the first parameter and the second parameter.

3. An object recognition rate calculation device according to claim 2, comprising: using the first object recognition rate model to calculate a predetermined object recognition rate based on the first parameters; and using the second object recognition rate model to calculate the final object recognition rate based on the predetermined object recognition rate and the second parameters.

4. The object recognition rate calculation device according to claim 3, wherein the second parameter is the packet loss rate during transmission of the video to the operator terminal, and the packet loss rate is replaced with the number of missing frames using a substitution model that shows the substitution relationship between the packet loss rate and the number of missing frames relating to the playback status of the video at the operator terminal, and the final object recognition rate is calculated based on the predetermined object recognition rate and the number of missing frames using the second object recognition rate model that shows the relationship between the number of missing frames and the object recognition rate.

5. An object recognition rate calculation device according to claim 3 or 4, comprising adjusting the predetermined object recognition rate based on the speed of the moving body obtained from a speed detection device provided on the moving body, and calculating the final object recognition rate based on the adjusted predetermined object recognition rate and the second parameter using the second object recognition rate model.

6. An object recognition rate calculation device according to claim 5, which adjusts the predetermined object recognition rate using the first parameter and a third object recognition rate model that shows the relationship between the speed and the object recognition rate.

7. An object recognition rate calculation method that calculates a final object recognition rate, which indicates the percentage of people who were actually able to recognize an object when multiple people viewed the video, based on a first parameter relating to the video quality of the video acquired from a camera installed on a mobile body and a second parameter relating to the transmission status of the video to an operator terminal.

8. A program that causes a computer to perform the method described in claim 7.

Images