Object recognition rate calculation device, device, object recognition rate calculation method, and program

Abstract

The purpose of the present disclosure is to determine an object recognition rate by an operator of a monitoring center.　To achieve this purpose, the present disclosure provides an object recognition rate calculation device that calculates a final object recognition rate on the basis of a first parameter relating to video quality of a video acquired from an imaging device provided on a moving body and a second parameter relating to a playback status of the video acquired from an operator terminal that displays the video, the final object recognition rate indicating, for a case in which a plurality of people have watched the video, the ratio of people who can recognize an object within a time period in which the moving body can be prevented from colliding with the object when the object has appeared a target distance from the moving body.

Description

Object recognition rate calculation device, device, object recognition rate calculation method, and program

[0001] The present disclosure relates to a technique for calculating the recognition rate of an object by an operator of a monitoring center that monitors autonomous driving.

[0002] With the development of communication technology, the improvement of sensor accuracy attached to automobiles, and the improvement of autonomous driving control technology, etc., the market scale of autonomous driving of automobiles is expanding. In autonomous driving technology, according to the level of autonomous driving, it is classified into levels from Lv.1 to Lv.5. In Lv.1 and Lv.2, the responsible entity for driving is the driver, and the autonomous driving system executes functions such as lane keep assist and braking, but it is treated as auxiliary, and the driver is responsible for driving.

[0003] On the other hand, in Lv.3 to Lv.5 of autonomous driving, the autonomous driving system of the automobile basically performs driving, and the responsible entity for driving is the system. In Lv.4 autonomous driving, the driver is not required under specific conditions, and it is possible to reduce accidents caused by mistakes when humans drive, but it is considered that accidents due to system errors may also occur.

[0004] Therefore, as a means to deal with system errors, it has been considered to remotely monitor the video from the monitoring camera of autonomous driving, and the operator of the monitoring center remotely operates the automobile according to the situation (Non-Patent Document 1). Also, in the guidelines of the Ministry of Land, Infrastructure, Transport and Tourism (Non-Patent Document 2), it is described that when performing autonomous driving, it is necessary to ensure the safety of operation by appropriately grasping various risks and the like that occur during operation and setting appropriate routes and areas. Therefore, when the operator of the monitoring center monitors the monitoring video, which is the video transmitted from the monitoring camera of autonomous driving, it is necessary to appropriately respond when a dangerous scene occurs during driving. For example, in the case where an object (person, animal, etc.) jumps out onto the road or an object that will collide with the road is placed, etc., it is necessary for the operator to appropriately grasp such objects that hinder driving and perform control such as remotely operating the automobile to stop it.

[0005] On the other hand, in situations where a vehicle is operating autonomously and transmitting surveillance video to a monitoring center in real time, there is a challenge in that the surveillance video may be affected by fluctuations in wireless communication bandwidth, potentially degrading the video quality. If the video quality of the surveillance video deteriorates, the operator at the monitoring center may not be able to recognize objects that could interfere with driving, potentially leading to an accident. Therefore, in autonomous driving, it is considered necessary to take measures such as stopping the vehicle in advance if the video quality does not meet a certain standard. In order to determine the video quality standard for object recognition, it is necessary to check what the object recognition rate will be when parameters that affect the quality, such as the video bitrate, are used as input. As a method for estimating the quality perceived by the user from the bitrate of the distributed video, the ITU-T has established Recommendation P.1204 (Non-Patent Literature 3). On the other hand, the video quality required for object recognition in autonomous driving differs from the video quality perceived by the user from general distributed video, so the video quality required when an operator recognizes objects from surveillance video is also required.

[0006] Points to consider at this time include the degradation of image quality due to changes in the video encoding rate and resolution caused by bandwidth fluctuations, the occurrence of packet loss during video data transmission, and the loss of playback video data due to buffer overflow of received video data caused by increased transmission delay. The degradation of image quality makes the display of objects coarser, making it difficult to recognize objects with the naked eye. In addition, the loss of playback video data is expected to lead to the occurrence of block noise in the missing portion, or a freeze where the frame information containing the missing portion is discarded and the previous frame is displayed as is, depending on the decoding implementation. If the aforementioned block noise or freeze occurs at the moment an object jumps out, it will become impossible to recognize the object with the naked eye.

[0007] As described above, a technique for estimating object recognition rates that takes into account the degradation of surveillance video quality and the loss of playback video data is necessary.

[0008] Supplementary explanatory materials for the final evaluation of research and development and demonstration projects toward the social implementation of advanced autonomous driving and MaaS, 2021, ITS and Autonomous Driving Promotion Office, Automobile Division, Manufacturing Industries Bureau, Ministry of Industry; Guidelines for passenger transport operators to ensure safety and convenience in unmanned autonomous driving mobility services in limited areas, Road Transport Bureau, Ministry of Land, Infrastructure, Transport and Tourism, p.1204: Video quality assessment of streaming services over reliable transport for resolutions up to 4K, ITU-T, 2020

[0009] When transmitting video in real time, it is expected that the bitrate and resolution will be dynamically varied in response to changes in the wireless bandwidth. Furthermore, it is anticipated that transmission packets of video data may be lost due to a decrease in radio wave propagation strength, or that some playback data may be discarded due to an increase in transmission delay, resulting in frame loss. In addition, in autonomous driving, the speed of the vehicle is not always constant but fluctuates, and the faster the vehicle's speed, the faster the movement of the background and other objects in the surveillance video becomes. When there is a lot of (fast) movement in the video, encoding the video becomes difficult, so even if the video is transmitted from the vehicle to the monitoring center at the same bitrate, the video quality may decrease and the object recognition rate may decrease. Moreover, when the vehicle is moving at a high speed, the distance the vehicle travels between the time the vehicle recognizes an object and the time the operator takes action increases, so it is necessary to recognize the object at an earlier stage, that is, when the object is displayed as smaller on the screen.

[0010] This disclosure is made in view of the circumstances described above and aims to determine the object recognition rate by operators at the monitoring center.

[0011] To achieve the above objective, this disclosure provides an object recognition rate calculation device that calculates a final object recognition rate, which indicates the percentage of multiple people who can recognize an object within a time limit that prevents the moving body from colliding with an object when the object appears at a target distance from the moving body, based on a first parameter relating to the image quality of the image acquired from a camera installed on a moving body and a second parameter relating to the playback status of the image acquired from an operator terminal that displays the image.

[0012] As explained above, this disclosure has the effect of enabling the determination of the object recognition rate by operators at the monitoring center.

[0013] This is an overall configuration diagram of the communication system according to the first embodiment. This is a diagram showing an example of the display screen on the operator terminal of the monitoring center. This is a diagram showing an example of the first object recognition rate model. This is a diagram showing an example of the second object recognition rate model. This is a diagram showing an example of the third object recognition rate model. This is an electrical hardware configuration diagram of each device and terminal according to the embodiment. This is a flowchart showing the process of estimating and presenting the object recognition rate according to the first embodiment. This is a flowchart showing the process of estimating and presenting the object recognition rate according to the first embodiment. This is an overall configuration diagram of the communication system according to the second embodiment. This is an overall configuration diagram of the communication system according to the third embodiment. This is a flowchart showing the process of estimating and presenting the object recognition rate according to the third embodiment. This is a flowchart showing the process of estimating and presenting the object recognition rate according to the third embodiment. This is an overall configuration diagram of the communication system according to the fourth embodiment.

[0014] Embodiments of the present invention will be described below with reference to the drawings.

[0015] ●First Embodiment [System Configuration of the First Embodiment] The general configuration of the communication system 1a of the embodiment will be described using Figure 1. Figure 1 is an overall configuration diagram of the communication system according to the first embodiment. Note that the values ​​shown in Figure 1 (1.2Mbps, 1280x720, 30fps, 5frames, 35km / h, 70%) are examples and are not limited to these.

[0016] As shown in Figure 1, the communication system 1a is constructed from devices mounted on a mobile body A capable of automatic movement, such as a vehicle (imaging device 2, speed detection device 3, object recognition rate calculation device 4, and communication device 5), and devices located on the monitoring center B side (operator terminal 6, communication device 8). The imaging device 2, speed detection device 3, object recognition rate calculation device 4, and communication device 5, as well as the operator terminal 6 and communication device 8, each have a communication unit (transmitter unit, receiver unit) for transmitting or receiving data (information) with other devices. Figure 2 shows an example of a display screen from the operator terminal at the monitoring center.

[0017] While mobile unit A primarily operates autonomously, the operator terminal 6 at monitoring center B displays video from one mobile unit A, or from multiple mobile units A as shown in Figure 2. This allows operators at monitoring center B to monitor the movement status of each mobile unit and remotely control a specific mobile unit according to its movement status. As a result, when viewing video from mobile unit A, operators can recognize an object appearing at a target distance from mobile unit A within the time it takes to prevent a collision between mobile unit A and the object, thereby avoiding an accident.

[0018] Furthermore, there may be multiple instances of each of the following: the imaging device 2, the speed detection device 3, the object recognition rate calculation device 4, the communication device 5, the operator terminal 6, and the communication device 8. In particular, there may be multiple imaging devices 2 to photograph the front, right side, left side, rear, and interior of the moving object A.

[0019] In this embodiment, the mobile unit A is assumed to be, for example, a vehicle (automobile) having communication and autonomous driving functions. However, the mobile unit A is not limited to automobiles, and may also be agricultural machinery such as a tractor having communication and autonomous driving functions. Furthermore, the mobile unit A may be a ship, aircraft, or the like, in addition to a vehicle.

[0020] The imaging device 2 has a surveillance camera mounted on the mobile body A and an encoder for compressing video data. It encodes video data obtained by imaging the outside or inside of the mobile body A and transmits it to the communication device 5. Furthermore, the imaging device 2 has a function for extracting video quality parameters such as bitrate, resolution, and frame rate. It extracts the bitrate, resolution, and frame rate of the video from the encoded video data and transmits bitrate information indicating the bitrate, resolution information indicating the resolution, and frame rate information indicating the frame rate, respectively, to the object recognition rate calculation device 4 every unit time t (for example, 1 second). Note that the imaging device 2 and the encoder (device) may be separate entities.

[0021] The velocity detection device 3 detects the speed of the moving object A (for example, h in kilometers per hour) and transmits velocity information indicating the detected speed to the object recognition rate calculation device 4. This velocity information is also transmitted every unit of time t, similar to the bitrate information, resolution information, and frame rate information.

[0022] The object recognition rate calculation device 4 has a calculation unit 9 that calculates the object recognition rate based on the information acquired from the imaging device 2 (bitrate information, resolution information, and frame rate information), the number of missing frames acquired from the operator terminal 6 (playback information acquisition unit 7), which will be described later, and the speed information acquired from the speed detection device 3. This calculation unit 9 is a function that is realized by the execution of a program. In the object recognition rate calculation device 4, the calculation is performed using information transmitted from the imaging device 2, the speed detection device 3, and the operator terminal 6 (playback information acquisition unit 7) at the same time. For example, the calculation unit 9 adds timestamp information to the transmitted information so that even if a delay occurs in the communication network 100, the object recognition rate can be calculated based on information at the same time.

[0023] Furthermore, the "object recognition rate" indicates the percentage of people who were able to actually recognize an object when multiple people viewed the aforementioned video. More specifically, it indicates the percentage of people who, when multiple monitors or other personnel viewed surveillance footage, were able to recognize an object within the time required to prevent the autonomous vehicle A from colliding with the object when the object appeared at a target distance from the autonomous vehicle A.

[0024] Here, object recognition refers to recognizing the appearance of an object, not identifying its type. The target distance refers to the distance from the camera 2 of the mobile body A, which is capturing surveillance footage, to the appearing object, and is the distance at which mobile body A can stop safely without colliding with the object. For example, it can be calculated from at least one of the following: glass-to-glass delay (the delay until the captured surveillance footage is displayed), reaction delay (the delay until the monitor recognizes the object and reacts, such as pressing a brake command), control delay (the delay from when a brake command is executed until mobile body A actually stops), the speed of mobile body A, and the coefficient of friction of the road surface. The time within which a collision can be prevented refers to the reaction delay. The coefficient of friction of the road surface may be a value predetermined by the region or weather, or it may be a value estimated by AI (artificial intelligence) based on the footage captured by the camera 2.

[0025] For example, if there are 100 monitors with varying levels of visual acuity, and 50 of them are able to recognize an object displayed in the surveillance video at a predetermined target distance ahead of moving object A within a time limit (a set reaction delay time) to prevent a collision, then the object recognition rate is 50%. The object recognition rate calculation device 4 then transmits the calculated object recognition rate information to the operator terminal 6 at the monitoring center B via the communication device 5. As a result, the operator terminal 6 displays the video from each moving object (surveillance video) along with the object recognition rate information (e.g., 70%) for each moving object, as shown in Figure 2. Note that the method of notifying the operator is not limited to displaying on the screen; it may also include voice notifications or alert notifications when a certain threshold is exceeded. The method of calculating the object recognition rate will be explained in detail later.

[0026] Communication device 5 can communicate data with operator terminal 6 via communication network 100 and communication device 8. Communication network 100 is constructed using mobile networks, the internet, etc. The internet also includes space internet, which is accessed via space using artificial satellites, etc.

[0027] Operator terminal 6 is a PC (Personal Computer) or similar terminal used by operators at monitoring center B. Operator terminal 6 has a decoder for decoding encoded video data.

[0028] The playback information acquisition unit 7, a function of the operator terminal 6, calculates the number of missing frames as an example of playback status parameters indicating whether the video transmitted to the operator terminal 6 in the monitoring center B could be displayed properly on the operator terminal 6 even if there was a delay in the transmitted data and missing frames. This missing frame count information is calculated every unit time t and transmitted from the operator terminal 6 to the object recognition rate calculation device 4 via the communication device 8 and the communication device 5.

[0029] The communication device 8 can communicate data with the object recognition rate calculation device 4 via the communication network 100 and the communication device 5.

[0030] With the above configuration, bitrate information, resolution information, and frame rate information transmitted from the imaging device 2, as well as speed information acquired by the speed detection device 3, are sent to the object recognition rate calculation device 4. Furthermore, the video data transmitted from the imaging device 2 is sent to the operator terminal 6 via the communication devices 5 and 8. The missing frame count information transmitted from the operator terminal 6 (playback information acquisition unit 7) is sent to the object recognition rate calculation device 4 via the communication devices 8 and 5. The object recognition rate information, which indicates the object recognition rate calculated by the object recognition rate calculation device 4, is transmitted from the object recognition rate calculation device 4 and sent to the operator terminal 6 via the communication devices 5 and 8.

[0031] [Detailed Calculation Method for Object Recognition Rate] Next, Figures 3 to 5 illustrate the method for calculating the object recognition rate calculated by the object recognition rate calculation device 4. Here, we will explain a method for determining the object recognition rate Z using the bitrate b, resolution r, frame rate f, speed v, and number of missing frames l of the video input to the device.

[0032] <Relationship between bitrate, resolution, frame rate and object recognition rate> Video quality is correlated with bitrate; the higher the bitrate, the higher the video quality and the higher the object recognition rate. The effects of resolution and frame rate on the same bitrate vary depending on the bitrate and encoding method. Generally, at high bitrates, higher resolution results in higher video quality and a higher object recognition rate. For the same bitrate, higher frame rates result in smoother motion, but the image quality of each frame tends to decrease due to the reduced amount of data allocated per frame. Generally, at low bitrates, higher frame rates tend to result in a lower object recognition rate. Figure 3 shows the relationship between bitrate, resolution, frame rate and object recognition rate.

[0033] Figure 3 shows the first object recognition rate model. The first object recognition rate model shows the relationship between the object recognition rate and the first parameters related to the image quality of the video (such as bitrate, resolution, and frame rate). Specifically, the first object recognition rate model shows the relationship between bitrate and object recognition rate for each resolution and frame rate at the same speed of the moving object A. Although three curves are shown in Figure 3, this is only due to the constraints of the drawing, and in reality, the first object recognition rate model has all possible patterns of resolution and frame rate. Furthermore, bitrate, resolution, and frame rate are examples of the image quality parameters of the video or the first parameters related to the image quality of the video.

[0034] While the object recognition rate increases with higher bitrates, once a certain bitrate is reached, the video quality almost reaches its upper limit, and the object recognition rate hardly changes even at higher bitrates. Therefore, the object recognition rate calculation device 4 calculates the object recognition rate using an object recognition rate model in which the object recognition rate increases with increasing bitrate, but the object recognition rate hardly changes above a predetermined bitrate.

[0035] The object recognition rate calculation device 4 has a calculation unit 9 (first calculation unit) that calculates the object recognition rate based on video quality parameters using a first object recognition rate model.

[0036] <Relationship between the number of missing frames and object recognition rate> The more missing frames there are, the later the timing of object recognition is, and the lower the object recognition rate becomes. Figure 4 shows the relationship between the number of missing frames and the object recognition rate.

[0037] Figure 4 shows the second object recognition rate model. The second object recognition rate model shows the relationship between the object recognition rate and the first parameter (bitrate, resolution, and frame rate) and the second parameter (number of missing frames, etc.). Specifically, the second object recognition rate model shows the relationship between the number of missing frames (horizontal axis), which is the second parameter related to the video playback status on the operator terminal 6, and the object recognition rate (vertical axis) for each frame rate, with the same parameters (same bitrate and resolution, and the same travel speed of the moving object A).

[0038] Note that although two curves are shown in Figure 4, this is only due to the constraints of the drawing; in reality, the second object recognition rate model encompasses all possible frame rate patterns.

[0039] Furthermore, the second object recognition rate model may show the relationship between the number of missing frames (horizontal axis) and the object recognition rate (vertical axis) for each bitrate, under the same parameters (same frame rate and resolution, and the same speed of moving object A). Alternatively, the second object recognition rate model may show the relationship between the number of missing frames (horizontal axis) and the object recognition rate (vertical axis) for each resolution, under the same parameters (same bit rate and frame rate, and the same speed of moving object A). Note that the relationship between the vertical and horizontal axes may be reversed.

[0040] Furthermore, the object recognition rate begins to decrease when the number of missing frames exceeds a certain level, and the object recognition rate converges to 0 when the number of missing frames and the number of frames per unit time (= frame rate × t) are equal, i.e., when all frames are missing. Therefore, the object recognition rate calculation device 4 calculates the object recognition rate using an object recognition rate model (second object recognition rate model) in which the object recognition rate decreases as the number of missing frames increases and becomes 0 when the number of missing frames and the number of frames per unit time are the same. In this case, the lower the frame rate, the longer the duration of the missing frames for each missing frame, so for the same number of missing frames, the object recognition rate decreases as the frame rate decreases.

[0041] The number of missing frames is an example of a video playback status parameter or a second parameter related to the video playback status.

[0042] The object recognition rate calculation device 4 has a calculation unit 9 (second calculation unit) that calculates the object recognition rate based on playback status parameter information using a second object recognition rate model.

[0043] <Relationship with Object Recognition Rate with Respect to Speed>Also, the object recognition rate calculation device 4 can handle different output recognition rates when the speed of the moving body A is different, with respect to the object recognition rate obtained using the first object recognition rate model by inputting the speed information of the moving body A. FIG. 5 is a diagram showing the third object recognition rate model. The third object recognition rate model shows the relationship between the first parameters (bit rate, resolution, frame rate, etc.) and the speed of the moving body A and the object recognition rate. Specifically, the third object recognition rate model is a diagram showing an example of a model (the third object recognition rate model) that shows the relationship between the bit rate (horizontal axis) and the object recognition rate (vertical axis) for each speed of the moving body A with the same parameters (the same resolution and frame rate). When the speed of the moving body A is high, the movement of the background and objects in the video becomes large (fast), making video encoding difficult. Therefore, even when the video is delivered at the same bit rate, the video quality deteriorates and the object recognition rate is considered to decrease. Also, when the hourly speed of the moving body A is high, the distance that the moving body A moves before a human such as an operator recognizes an object becomes long. Therefore, since the object must be recognized from a farther distance, this also affects the decrease in the object recognition rate. The object recognition rate calculation device 4 calculates the object recognition rate using these properties.

[0044] Note that in FIG. 5, although three curves are shown, due to drawing constraints, only the three curves are explicitly shown. In reality, the third object recognition rate model has all patterns of hourly speeds.

[0045] Furthermore, the third object recognition rate model has a relationship with each of the first parameters or combinations used in the first object recognition rate model for each speed. Specifically, the relationship between frame rate (horizontal axis) and object recognition rate (vertical axis) may be shown for each speed of moving object A, with the same parameters (same bitrate and resolution). Alternatively, the third object recognition rate model may show the relationship between resolution (horizontal axis) and object recognition rate (vertical axis) for each speed of moving object A, with the same parameters (same frame rate and bitrate). Also, due to the constraints of the figure, the relationship is shown for the case where two parameters are the same when the first parameters are the same, but the same relationship may be shown even when one of the first parameters is the same.

[0046] The object recognition rate calculation device 4 has a calculation unit 9 (third calculation unit) that calculates the object recognition rate based on the speed (movement speed) of the moving object A using a third object recognition rate model.

[0047] As described above, the object recognition rate calculation device 4 calculates the object recognition rate based on the first parameter using the first object recognition rate model showing the relationship between the first parameter and the object recognition rate. Further, the object recognition rate calculation device 4 uses the first object recognition rate model showing the relationship between the first parameter and the object recognition rate and the second object recognition rate model showing the relationship between the second parameter and the object recognition rate, and calculates the object recognition rate based on the first parameter and the second parameter. Specifically, a predetermined object recognition rate is calculated based on the first parameter using the first object recognition rate model, and the object recognition rate is calculated based on the predetermined object recognition rate and the second parameter using the second object recognition rate model. Further, the object recognition rate calculation device 4 calculates the object recognition rate based on the first parameter and the speed using the third object recognition model showing the relationship between the first parameter, the speed, and the object recognition rate. Further, the object recognition rate calculation device 4 uses the second object recognition rate model showing the relationship between the second parameter and the object recognition rate and the third object recognition rate model showing the relationship between the first parameter, the speed, and the object recognition rate, and calculates the object recognition rate based on the first parameter, the second parameter, and the speed. Specifically, the object recognition rate is calculated based on the first parameter and the speed using the third object recognition rate model, and the object recognition rate is calculated based on the predetermined object recognition rate and the second parameter using the second object recognition rate model.

[0048] <Method for Calculating Object Recognition Rate by Object Recognition Rate Calculation Device> This section describes the method for calculating the object recognition rate performed by the object recognition rate calculation device 4, specifically the method for determining the object recognition rate Z when using the video bitrate b, resolution r (=height × width), frame rate f, number of missing frames l, and speed v. The formula for calculating the object recognition rate Z is expressed as an equation f(b, r, f, l, v) with bitrate information, resolution information, frame rate information, number of missing frames information, and speed information as inputs. As shown in Figure 5, the function f(b, r, f, l, v) has the characteristic that the output object recognition rate decreases as the speed increases. Taking this into account, f(b, r, f, l, v) is calculated using one of the formulas (Equation 1) to (Equation 4). Note that f(b, r, f, l, v) is defined in the range of 0 to 1, and in each of the following equations (Equation 1) to (Equation 4), if f(b, r, f, l, v) is 0 or less, the object recognition rate is defined as 0, and if f(b, r, f, l, v) is 1 or greater, the object recognition rate is defined as 1.

[0049]

[0050]

[0051]

[0052] Here, in (Equation 1) to (Equation 4), c1 to c 10 is a coefficient that differs for each formula, and t is the unit time [seconds] for calculating the information. Also, log is a logarithm with base e, which is the base of the natural logarithm. Here, by setting the above coefficients to values ​​according to conditions such as encoding conditions (codec, GoP length (and GoP structure), profile, etc.), decoding implementation in case of data loss (indicating whether it is block noise or freeze), size and color of the object, display resolution, size of the object displayed on the display, weather conditions during driving, driving time (day / night, etc.), road surface friction coefficient, glass-to-glass delay, control delay, reaction delay, and shooting conditions of the shooting device 2 (camera mode, focal length, Field of View, image stabilization, white balance delay, lens flare, etc.), it is possible to estimate the object according to the driving conditions and monitoring environment.

[0053] [Hardware Configuration] Next, the electrical hardware configuration of the object recognition rate calculation device 4, communication device 5, operator terminal 6, and communication device 8 will be described using Figure 6. Figure 6 is an electrical hardware configuration diagram of each device and terminal according to this embodiment.

[0054] The object recognition rate calculation device 4, as a computer, includes a CPU 101, ROM 102, RAM 103, SSD 104, external device connection interface 105, network interface 106, display 107, input device 108, media interface 109, and bus line 110, as shown in Figure 6.

[0055] Of these, the CPU 101 controls the operation of the entire object recognition rate calculation device 4. The ROM 102 stores programs used to drive the CPU 101, such as the IPL. The RAM 103 is used as the work area for the CPU 101.

[0056] The SSD 104 reads or writes various types of data according to the control of the CPU 101. Alternatively, an HDD (Hard Disk Drive) may be used instead of the SSD 104.

[0057] External device connection I / F 105 is an interface for connecting various external devices. These external devices include displays, speakers, keyboards, mice, USB memory sticks, and printers.

[0058] The network interface 106 is an interface for data communication via the communication network 100.

[0059] The display 107 is a type of display means, such as a liquid crystal display or an organic EL (electroluminescence) display, which displays various images.

[0060] The input device 108 is an example of an input means for receiving input operations such as characters, numbers, and various instructions, and may include buttons, switches, keyboards, pointing devices, etc.

[0061] The media interface 109 controls the reading or writing (storage) of data to or from a recording medium 109m, such as flash memory. The recording medium 109m includes DVDs, Blu-ray Discs (registered trademarks), and the like.

[0062] The bus line 110 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 101 shown in Figure 6.

[0063] Since the communication device 5, operator terminal 6, and communication device 8 have the same configuration as the object recognition rate calculation device 4, their explanation will be omitted.

[0064] [Processing or Operation of the Communication System According to the First Embodiment] Next, the processing or operation of each device in the communication system 1a will be explained using Figures 7 and 8. Figures 7 and 8 are flowcharts illustrating the process of estimating and presenting the object recognition rate according to the first embodiment.

[0065] S11: The imaging device 2 captures video data from the outside or inside of the moving object A, then compresses the video data using an encoder and transmits it to the communication device 5.

[0066] S12: The communication device 5 transmits video data to the operator terminal 6 via the communication network 100 and the communication device 8.

[0067] S13: The imaging device 2 uses the encoding information of the video data to extract the bitrate, resolution, and frame rate of the video, and transmits the bitrate information (e.g., 5Mbps, 10Mbps), resolution information (e.g., 640x360, 1280x720), and frame rate information (e.g., 10fps, 30fps) to the object recognition rate calculation device 4.

[0068] S14: The speed detection device 3 detects the speed of the moving object A and transmits speed information indicating this speed to the object recognition rate calculation device 4.

[0069] S15: The playback information acquisition unit 7 extracts the number of missing frames during playback from the operator terminal 6, and the operator terminal 6 transmits this missing frame number information (for example, 0 frames, 5 frames) to the communication device 8.

[0070] S16: The communication device 8 transmits the missing frame count information received from the operator terminal 6 to the object recognition rate calculation device 4 via the communication network 100 and the communication device 5.

[0071] S17: The calculation unit 9 of the object recognition rate calculation device 4 uses the first to third object recognition rate models and calculates the final object recognition rate (for example, 70%) based on bitrate information, resolution information, frame rate information, number of missing frames information, and velocity information.

[0072] Here, we will specifically explain how to calculate the object recognition rate using the first to third object recognition rate models. For example, the calculation unit 9 uses the first object recognition rate model (Figure 3) at a reference speed of 30 km / h to calculate a predetermined object recognition rate based on a bitrate of 1.2 Mbps at a resolution of 1280 × 720 and a frame rate of 30 fps. Then, if the speed indicated by the speed information is 35 km / h, the calculation unit 9 uses the third object recognition model (Figure 5) to adjust the predetermined object recognition rate by selecting a curve for 35 km / h that is closer to 50 km / h from the curve for a reference speed of 30 km / h. After that, the calculation unit 9 uses the second object recognition rate model (Figure 4) to select a curve for a frame rate of 30 fps and calculates the final object recognition rate based on the number of missing frames, 5 frames, indicated by the number of missing frames information. The object recognition rate calculation device 4 then transmits object recognition rate information indicating the final object recognition rate to the communication device 5.

[0073] S18: The communication device 5 transmits the object recognition rate information received from the object recognition rate calculation device 4 to the operator terminal 6 via the communication network 100 and the communication device 8.

[0074] As a result, as shown in Figure 2, the operator terminal 6 decodes the video data sent from each mobile object by processing S12, associates it with the object recognition rate information related to the video of each mobile object sent from the communication device 5 by processing S18, and displays it on the display by splitting the screen. In order to associate the video with the object recognition rate information, identification information such as an ID may be included in both the video data and the object recognition rate information. In this case, the identification information is added to the video data and bitrate information output by the shooting device 2. The object recognition rate calculation device 4 then adds the identification information added to the bitrate information acquired from the shooting device 2 to the object recognition rate information and outputs it.

[0075] S19: If the movement of mobile body A is not completed (NO), the process returns to S11. On the other hand, if the movement of mobile body A is completed (YES), the processes shown in Figures 7 and 8 are completed.

[0076] [Effects of the First Embodiment] As described above, according to the first embodiment, the object recognition rate by the operator of the monitoring center B can be determined and presented. As a result, the operator terminal 6 of the monitoring center B displays the monitoring video from each moving object in association with the object recognition rate information of this monitoring video, as shown in Figure 2, thus eliminating the situation in which the operator does not notice that the video quality of the monitoring video has deteriorated.

[0077] ●Second Embodiment [System Configuration of the Second Embodiment] Next, the communication system 1b according to the second embodiment will be described with reference to Figure 9. Figure 9 is an overall configuration diagram of the communication system according to the second embodiment. The second embodiment differs from the configuration of the first embodiment in that it does not use the speed detection device 3. That is, the second embodiment shows an example where the speed of the mobile body A is unknown. Note that the values ​​shown in Figure 9 (1.2 Mbps, etc.) are just examples, as with Figure 1, and are not limiting. Also, devices or terminals identical to those in Figure 1 are denoted by the same reference numerals and their descriptions are omitted.

[0078] <Method for Calculating Object Recognition Rate by Object Recognition Rate Calculation Device> This section explains the method for calculating the object recognition rate performed by the object recognition rate calculation device 4. Here, we will explain a method for obtaining the object recognition rate Z using the video bitrate b, resolution r (= height × width), frame rate f, and number of missing frames l. The formula for calculating the object recognition rate Z is expressed as an expression f(b, r, f, l) with bitrate information, resolution information, frame rate information, and number of missing frames as inputs. The function f(b, r, f, l) has the characteristics of the model shown in Figures 3 and 4. Based on this, f(b, r, f, l) is calculated using one of the following formulas (Equation 5) to (Equation 8). Note that f(b, r, f, l) is defined in the range of 0 to 1, and in each of the following formulas, if f(b, r, f, l) is 0 or less, the object recognition rate is defined as 0, and if f(b, r, f, l) is 1 or greater, the object recognition rate is defined as 1. Under conditions where resolution and frame rate remain constant, formulas that do not take the relevant parameters as inputs may be used. Furthermore, the formulas used to calculate the object recognition rate are not limited to those shown below.

[0079]

[0080]

[0081]

[0082] Here, in equations 5 to 8, c1 to c8 are different coefficients for each equation, and t is the unit time [seconds] for calculating the information. Also, log is a logarithm with base e, which is the base of the natural logarithm. By setting the above coefficients to values ​​corresponding to conditions such as encoding conditions (codec, GoP length (and GoP structure), profile, etc.), decoding implementation in case of data loss (indicating whether it is block noise or freeze), size and color of the object, display resolution, size of the object displayed on the display, weather conditions during driving, driving time (day / night, etc.), road surface friction coefficient, glass-to-glass delay, control delay, reaction delay, and shooting conditions of the shooting device 2 (camera mode, focal length, Field of View, image stabilization, white balance delay, lens flare, etc.), it is possible to estimate the object according to the driving conditions and monitoring environment.

[0083] [Processing or Operation of the Communication System According to the Second Embodiment] In the second embodiment, as a difference from the flowchart showing the processing of the first embodiment shown in Figures 7 and 8, S14 is not performed, and in S17, the object recognition rate calculation device 4 uses the first and second object recognition rate models to calculate the object recognition rate based on bitrate information, resolution information, frame rate information, and missing frame number information. In the second embodiment, since the third object recognition model is not used, the calculation unit 9 uses, for example, only the second object recognition model to calculate the final object recognition rate.

[0084] [Effects of the Second Embodiment] As described above, according to the second embodiment, even when the travel speed is constant or unknown, the recognition rate of objects by the operator of the monitoring center B can be determined and presented.

[0085] ●Third Embodiment [System Configuration of the Third Embodiment] Next, the communication system 1c according to the third embodiment will be described with reference to Figures 10 to 12. Figure 10 is an overall configuration diagram of the communication system according to the third embodiment. Note that the values ​​shown in Figure 10 (1.2 Mbps, etc.) are examples, as with Figure 1, and are not limited to these. Also, the same reference numerals are used for devices or terminals as in Figure 1, and their descriptions are omitted.

[0086] In the communication system 1c of Figure 10, the object recognition rate calculation device 4, which was installed on the mobile body A side in the first embodiment (Figure 1), is installed on the monitoring center B side. The video data encoded by the imaging device 2 is transmitted to the operator terminal 6 via the communication devices 5 and 8. However, in this embodiment, the operator terminal 6, not the imaging device 2, has the bitrate extraction function. In addition, speed information indicating the speed of mobile body A detected by the mobile body A speed detection device 3 is transmitted to the object recognition rate calculation device 4 via the communication devices 5 and 8. This transmission is performed every unit time t (for example, 1 second).

[0087] The calculation unit 9 of the object recognition rate calculation device 4 calculates the current (final) object recognition rate based on bitrate information, resolution information, frame rate information, number of missing frames information, and speed information, similar to the first embodiment. The object recognition rate calculation device 4 then transmits this object recognition rate information to the operator terminal 6.

[0088] [Processing or Operation of the Communication System According to the Third Embodiment] Next, the processing or operation of each device in the communication system 1c will be explained using Figures 11 and 12. Figures 11 and 12 are flowcharts illustrating the process of estimating and presenting the object recognition rate according to the third embodiment.

[0089] S21: The imaging device 2 takes a picture of the outside or inside of the moving object A to obtain video data, then compresses the video data using an encoder and transmits it to the communication device 5.

[0090] S22: The speed detection device 3 detects the current speed of the moving object A and transmits speed information indicating this speed to the communication device 5.

[0091] S23: The communication device 5 transmits video data to the operator terminal 6 via the communication network 100 and the communication device 8, and also transmits speed information to the object recognition rate calculation device 4.

[0092] S24: The operator terminal 6 uses the encoding information of the video data to extract the bitrate, resolution, and frame rate of the video, and transmits the bitrate information (e.g., 5Mbps, 10Mbps), resolution information (e.g., 640x360, 1280x720), and frame rate information (e.g., 10fps, 30fps) to the object recognition rate calculation device 4.

[0093] S25: The playback information acquisition unit 7 extracts the number of missing frames during playback from the operator terminal 6. The operator terminal 6 then transmits this missing frame information (for example, 0 frames, 5 frames) to the object recognition rate calculation device 4.

[0094] S26: The object recognition rate calculation device 4 uses the models shown in Figures 3, 4, and 5 to calculate the object recognition rate (for example, 70%) based on bitrate information, resolution information, frame rate information, number of missing frames information, and velocity information. The object recognition rate calculation device 4 then transmits the object recognition rate information to the operator terminal 6.

[0095] As a result, as shown in Figure 2, the operator terminal 6 decodes the video data sent from each moving object by processing S23, associates it with the object recognition rate information related to the video of each moving object sent from the object recognition rate calculation device 4 by processing S26, and displays it on the display by splitting the screen. In this case as well, identification information such as an ID may be included in both the video data and the object recognition rate information in order to associate the video with the object recognition rate information. In this case, the shooting device 2 adds the identification information to the video data it outputs. Then, when the operator terminal 6 extracts the bitrate, it adds the same identification information to the bitrate information. The object recognition rate calculation device 4 adds the identification information added to the bitrate information obtained from the operator terminal 6 to the object recognition rate information and transmits it to the operator terminal 6.

[0096] S27: If the movement of mobile body A is not completed (NO), the process returns to S21. On the other hand, if the movement of mobile body A is completed (YES), the processes shown in Figures 11 and 12 are completed.

[0097] [Effects of the Third Embodiment] As described above, the third embodiment provides the same effects as the first embodiment. Furthermore, the third embodiment requires fewer devices on the mobile body A side compared to the first embodiment, thus reducing the initial costs for receiving remote monitoring and remote control services from this communication system 1c.

[0098] ●Fourth Embodiment Next, the communication system 1d according to the fourth embodiment will be described using Figure 13. Figure 13 is an overall configuration diagram of the communication system according to the fourth embodiment. Note that the values ​​shown in Figure 13 (1.2 Mbps, etc.) are examples, as with Figure 1, and are not limited to these. Also, the same reference numerals are used for devices or terminals as in Figure 1, and their descriptions are omitted.

[0099] In the fourth embodiment, as a difference from the flowchart showing the processing of the third embodiment shown in Figures 11 and 12, processing S22 is not performed. Also, in processing S26, the object recognition rate calculation device 4 uses the first object recognition rate model (Figure 3) and the second object recognition rate model (Figure 4), but similar to the second embodiment, it does not use the third object recognition rate model (Figure 5) and calculates the object recognition rate based on bitrate information, resolution information, frame rate information, and missing frame count information. The fourth embodiment differs from the third embodiment (Figure 10) in that the velocity detection device 3 is not used. That is, the fourth embodiment shows an example where the velocity of the moving object A is fixed (constant).

[0100] [Effects of the Fourth Embodiment] As described above, the fourth embodiment provides the same effects as the second embodiment. Furthermore, in the fourth embodiment, since fewer devices are required on the mobile body A side compared to the second embodiment, the initial cost for receiving remote monitoring and remote control services using this communication system 1d can be reduced.

[0101] [Relationship between each object recognition rate model and formula] First object recognition rate model: Expressed by the formulas X, A1, A2, A3 in any one of the formulas (Formula 5), ​​(Formula 6), (Formula 7), (Formula 8) Second object recognition rate model: Expressed in the denominator of the formula Z in any one of the formulas (Formula 1), (Formula 2), (Formula 3), (Formula 4), (Formula 5), ​​(Formula 6), (Formula 7), (Formula 8) Third object recognition rate model: Expressed by the formulas X, A1, A2, A3 in any one of the formulas (Formula 1), (Formula 2), (Formula 3), (Formula 4) Object recognition rate: The value derived from the formula Z in (Formula 5) [Note] The above embodiment is not limited to the above embodiment, and various modifications and applications are possible, for example, as shown below.

[0102] (1) In the above embodiment, the shooting device 2 or operator terminal 6 extracts the bitrate from the video data and transmits the bitrate information, and the object recognition rate calculation device 4 calculates the object recognition rate using the bitrate information, but it is not limited to this. For example, the shooting device 2 or operator terminal 6 uses bitrate information, resolution information, and frame rate information, but it may calculate the object recognition rate using only some of this information. Note that bitrate information, resolution information, and frame rate information are examples of video quality parameters, and other parameters may be used. Also, the number of missing frames is an example of a parameter of the video playback status, and other parameters may be used.

[0103] (2) Each device can be implemented using a computer and a program, and this program can be recorded on a (non-temporary) recording medium or provided via a communication network such as the Internet. Furthermore, by storing the first to third object recognition rate models in a device (storage device) such as memory, a PC, or a server, the device can have at least one of the first to third object recognition rate models.

[0104] (3) The CPU 101, which is a hardware processor, may be single or multiple.

[0105] [Other] Each of the above embodiments can also be represented by the following supplementary information.

[0106] [Addendum 1] An object recognition rate calculation device that calculates a final object recognition rate, which indicates the percentage of people who were actually able to recognize an object when multiple people viewed the video, based on a first parameter relating to the video quality of the video acquired from a shooting device installed on a moving object, and a second parameter relating to the playback status of the video acquired from an operator terminal that displays the video.

[0107] [Appendix 2] An object recognition rate calculation device according to Appendix 1, which uses a first object recognition rate model showing the relationship between the first parameter and the object recognition rate, and a second object recognition rate model showing the relationship between the second parameter and the object recognition rate, and calculates the final object recognition rate based on the first parameter and the second parameter.

[0108] [Appendix 3] An object recognition rate calculation device according to Appendix 2, which uses the first object recognition rate model to calculate a predetermined object recognition rate based on the first parameters, and uses the second object recognition rate model to calculate the final object recognition rate based on the predetermined object recognition rate and the second parameters.

[0109] [Appendix 4] An object recognition rate calculation device according to Appendix 3, which adjusts the predetermined object recognition rate based on the speed of the moving body obtained from a speed detection device provided on the moving body, and calculates the final object recognition rate using the second object recognition rate model based on the adjusted predetermined object recognition rate and the second parameter.

[0110] [Appendix 5] An object recognition rate calculation device according to Appendix 4, which adjusts the predetermined object recognition rate using the first parameter and a third object recognition model that shows the relationship between the speed and the object recognition rate.

[0111] [Appendix 6] An object recognition rate calculation method that calculates a final object recognition rate indicating the percentage of people who were actually able to recognize an object when multiple people viewed the video, based on a first parameter relating to the video quality of the video acquired from a shooting device installed on a moving object, and a second parameter relating to the playback status of the video acquired from an operator terminal that displays the video.

[0112] [Appendix 7] A program that causes a computer to perform the method described in Appendix 6.

[0113] [Relationship with the Basic Application] This patent application claims priority based on the international patent application PCT / JP2024 / 032930 filed on September 13, 2024, and the entire contents of the international patent application PCT / JP2024 / 032930 are incorporated herein by reference.

[0114] 1a, 1b, 1c, 1d Communication system A Mobile object B Monitoring center 2 Imaging device 3 Speed ​​detection device 4 Object recognition rate calculation device 5 Communication device 6 Operator terminal 7 Playback information acquisition unit 8 Communication device 9 Calculation unit 100 Communication network

Claims

1. An object recognition rate calculation device that calculates a final object recognition rate, which indicates the percentage of multiple people who, when viewing the video, can recognize an object within the time required to prevent the moving body from colliding with the object when the object appears at a target distance from the moving body, based on a first parameter relating to the video quality of the video acquired from a camera installed on a moving body and a second parameter relating to the playback status of the video acquired from an operator terminal displaying the video.

2. An object recognition rate calculation device according to claim 1, which uses a first object recognition rate model showing the relationship between the first parameter and the object recognition rate, and a second object recognition rate model showing the relationship between the second parameter and the object recognition rate, and calculates the final object recognition rate based on the first parameter and the second parameter.

3. An object recognition rate calculation device according to claim 2, comprising: using the first object recognition rate model to calculate a predetermined object recognition rate based on the first parameters; and using the second object recognition rate model to calculate the final object recognition rate based on the predetermined object recognition rate and the second parameters.

4. The object recognition rate calculation device according to claim 1, which calculates the final object recognition rate based on the speed of the moving body obtained from a speed detection device provided on the moving body.

5. An object recognition rate calculation device according to claim 4, which calculates the final object recognition rate based on the first parameter, the speed, and the second parameter, using a third object recognition rate model showing the relationship between the first parameter and the speed and the object recognition rate, and a second object recognition rate model showing the relationship between the second parameter and the object recognition rate.

6. An object recognition rate calculation device according to claim 5, comprising: using the third object recognition rate model to calculate a predetermined object recognition rate based on the first parameters and speed; and using the second object recognition rate model to calculate the final object recognition rate based on the predetermined object recognition rate and the second parameters.

7. A device having a first object recognition rate model that shows the relationship between a first parameter relating to the image quality of an image acquired from a camera installed on a moving object, and an object recognition rate that indicates the proportion of multiple persons who, when viewing the image, can recognize an object within a time that prevents the moving object from colliding with the object when the object appears at a target distance from the moving object.

8. A device having a second object recognition rate model that shows the relationship between a second parameter relating to the playback status of the video acquired from an operator terminal that displays the video acquired from a shooting device installed on a moving body, and an object recognition rate that indicates the percentage of people who, when multiple people view the video, can recognize an object within a time that prevents the moving body from colliding with the object when the object appears at a target distance from the moving body.

9. A device having a third object recognition rate model that shows the relationship between a first parameter relating to the image quality of images acquired from a camera installed on a moving body, the speed of the moving body, and an object recognition rate that indicates the proportion of multiple persons who, when viewing images acquired from a camera installed on the moving body, can recognize an object within a time that prevents the moving body from colliding with the object when the object appears at a target distance from the moving body.

10. An object recognition rate calculation method that calculates a final object recognition rate, which indicates the percentage of people who, when multiple people view the video, can recognize an object within a time that prevents the moving body from colliding with the object when the object appears at a target distance from the moving body, based on a first parameter relating to the video quality of the video acquired from a shooting device installed on a moving body, and a second parameter relating to the playback status of the video acquired from an operator terminal that displays the video.

11. A program that causes a computer to perform the method described in claim 10.

Images