Object recognition rate calculation device, device, object recognition rate calculation method, and program

Abstract

An objective of the present disclosure is to obtain an object recognition rate via an operator of a monitoring center.　Therefore, the present disclosure relates to an object recognition rate calculation device that calculates a final object recognition rate on the basis of a first parameter relating to the video quality of a video acquired from an imaging device provided on a moving body and a second parameter relating to the transmission status of the video to an operator terminal, wherein the final object recognition rate indicates the ratio of people who can recognize an object within a time period in which it is possible to prevent the moving body from colliding with the object when the object appears at a target distance from the moving body when a plurality of people watch the video.

Description

Object recognition rate calculation device, device, object recognition rate calculation method, and program

[0001] The present disclosure relates to a technique for calculating the recognition rate of an object by an operator of a monitoring center that monitors autonomous driving.

[0002] With the development of communication technology, the improvement of sensor accuracy attached to automobiles, and the improvement of autonomous driving control technology, etc., the market scale of autonomous driving of automobiles is expanding. In autonomous driving technology, according to the level of autonomous driving, it is classified into levels from Lv.1 to Lv.5. In Lv.1 and Lv.2, the main responsible party for driving is the driver. The autonomous driving system executes functions such as lane keep assist and braking, but it is treated as auxiliary, and the driver is responsible for driving.

[0003] On the other hand, in Lv.3 to Lv.5 of autonomous driving, the autonomous driving system of the automobile basically conducts driving, and the main responsible party for driving is the system. In Lv.4 autonomous driving, a driver is not required under specific conditions, and it is possible to reduce accidents caused by mistakes when humans drive, but it is considered that accidents due to system errors may also occur.

[0004] Therefore, as a means to deal with system errors, it has been considered to remotely monitor the video from the monitoring camera of autonomous driving, and for the operator of the monitoring center to remotely operate the automobile according to the situation (Non-Patent Document 1). Also, in the guidelines of the Ministry of Land, Infrastructure, Transport and Tourism (Non-Patent Document 2), it is described that when conducting autonomous driving, it is necessary to ensure the safety of operation by surely grasping various risks that occur during operation and setting appropriate routes and areas. Therefore, when the operator of the monitoring center monitors the monitoring video, which is the video transmitted from the monitoring camera of autonomous driving, it is necessary to appropriately respond when a dangerous scene for driving occurs. For example, in cases where an object (person, animal, etc.) jumps out onto the road or an object that will collide with the vehicle is placed on the road, etc., it is necessary for the operator to appropriately recognize the objects that hinder driving and perform control such as remotely operating the vehicle to stop it.

[0005] On the other hand, in situations where a vehicle is operating autonomously and transmitting surveillance video to a monitoring center in real time, there is a challenge in that the surveillance video may be affected by fluctuations in wireless communication bandwidth, potentially degrading the video quality. If the video quality of the surveillance video deteriorates, the operator at the monitoring center may not be able to recognize objects that could interfere with driving, potentially leading to an accident. Therefore, in autonomous driving, it is considered necessary to take measures such as stopping the vehicle in advance if the video quality does not meet a certain standard. In order to determine the video quality standard for object recognition, it is necessary to check what the object recognition rate will be when parameters that affect the quality, such as the video bitrate, are used as input. As a method for estimating the quality perceived by the user from the bitrate of the distributed video, the ITU-T has established Recommendation P.1204 (Non-Patent Literature 3). On the other hand, the video quality required for object recognition in autonomous driving differs from the video quality perceived by the user from general distributed video, so the video quality required when an operator recognizes objects from surveillance video is also required.

[0006] Points to consider at this time include the degradation of image quality due to changes in the video encoding rate and resolution caused by bandwidth fluctuations, and the loss of video data due to packet loss during video data transmission. The degradation of image quality makes the display of objects coarser, making it difficult to recognize objects with the naked eye. In addition, since the lost video data cannot be played back, it is expected that block noise will occur in the missing parts, or the frames containing the missing parts will not be played back, leading to the video becoming a frozen frame, depending on the decoding implementation. If the aforementioned block noise or freeze occurs at the moment an object jumps out, it will become impossible to recognize the object with the naked eye.

[0007] As described above, a technique for estimating object recognition rates that takes into account the degradation of surveillance video quality and the loss of playback video data is necessary.

[0008] Supplementary explanatory materials for the final evaluation of research and development and demonstration projects toward the social implementation of advanced autonomous driving and MaaS, 2021, ITS and Autonomous Driving Promotion Office, Automobile Division, Manufacturing Industries Bureau, Ministry of Industry; Guidelines for passenger transport operators to ensure safety and convenience in unmanned autonomous driving mobility services in limited areas, Road Transport Bureau, Ministry of Land, Infrastructure, Transport and Tourism, p.1204: Video quality assessment of streaming services over reliable transport for resolutions up to 4K, ITU-T, 2020

[0009] When transmitting video in real time, it is expected that the bitrate and resolution will be dynamically varied in response to changes in the wireless bandwidth. Furthermore, it is anticipated that transmission packets of video data may be lost due to a decrease in radio wave propagation strength. Ideally, the number of affected frames should be counted when outputting the playback video during decoding. However, in systems incorporating commercially available decoders, such counting may be difficult. In such cases, it is necessary to consider parameters that indicate the video transmission status, such as the packet loss rate. Additionally, in autonomous driving, the vehicle's speed is not constant but fluctuates. The faster the vehicle's speed, the faster the background and other objects in the surveillance video will move. When there is large (fast) movement in the video, encoding becomes difficult. Therefore, even when transmitting video from the vehicle to the monitoring center at the same bitrate, the video quality may deteriorate, potentially reducing the object recognition rate. Furthermore, at high vehicle speeds, the distance the vehicle travels between object recognition and operator intervention increases. Therefore, it is necessary to recognize objects at an earlier stage, i.e., when objects are displayed smaller on the screen.

[0010] This disclosure is made in view of the circumstances described above and aims to determine the object recognition rate by operators at the monitoring center.

[0011] To achieve the above objective, this disclosure provides an object recognition rate calculation device that calculates a final object recognition rate, which indicates the percentage of people who, when multiple people view the video, can recognize an object within a time that prevents the moving body from colliding with the object when the object appears at a target distance from the moving body, based on a first parameter relating to the video quality of the video acquired from a camera installed on a moving body and a second parameter relating to the transmission status of the video to an operator terminal.

[0012] As explained above, this disclosure has the effect of enabling the determination of the object recognition rate by operators at the monitoring center.

[0013] This is an overall configuration diagram of the communication system according to the first embodiment. This is a diagram showing an example of the display screen on the operator terminal of the monitoring center. This is a diagram showing an example of the first object recognition rate model. This is a diagram showing an example of the second object recognition rate model. This is a diagram showing an example of a replacement model which is part of the second object recognition rate model. This is a diagram showing an example of the third object recognition rate model. This is an electrical hardware configuration diagram of each device and terminal according to the embodiment. This is a flowchart showing the process of estimating and presenting the object recognition rate according to the first embodiment. This is a flowchart showing the process of estimating and presenting the object recognition rate according to the first embodiment. This is an overall configuration diagram of the communication system according to the second embodiment. This is an overall configuration diagram of the communication system according to the third embodiment. This is a flowchart showing the process of estimating and presenting the object recognition rate according to the third embodiment. This is a flowchart showing the process of estimating and presenting the object recognition rate according to the third embodiment. This is an overall configuration diagram of the communication system according to the fourth embodiment.

[0014] Embodiments of the present invention will be described below with reference to the drawings.

[0015] ●First Embodiment [System Configuration of the First Embodiment] The general configuration of the communication system 1a of the embodiment will be described using Figure 1. Figure 1 is an overall configuration diagram of the communication system according to the first embodiment. Note that the values ​​shown in Figure 1 (1.2Mbps, 1280x720, 30fps, 5%, 35km / h, 70%) are examples and are not limited to these.

[0016] As shown in Figure 1, the communication system 1a is constructed from devices mounted on a mobile body A capable of automatic movement, such as a vehicle (imaging device 2, speed detection device 3, object recognition rate calculation device 4, and communication device 5), and devices located on the monitoring center B side (operator terminal 6, communication device 8). The imaging device 2, speed detection device 3, object recognition rate calculation device 4, and communication device 5, as well as the operator terminal 6 and communication device 8, each have a communication unit (transmitter unit, receiver unit) for transmitting or receiving data (information) with other devices. Figure 2 shows an example of a display screen from the operator terminal at the monitoring center.

[0017] While mobile unit A primarily operates autonomously, the operator terminal 6 at monitoring center B displays video from one mobile unit A, or from multiple mobile units A as shown in Figure 2. This allows the operator at monitoring center B to monitor the movement status of each mobile unit and remotely control a specific mobile unit according to its movement status. As a result, when viewing video from a predetermined mobile unit A, if an object appears at a target distance from the mobile unit, the operator can recognize the object within the time required to prevent a collision between mobile unit A and the object, thereby avoiding an accident.

[0018] Furthermore, there may be multiple instances of each of the following: the imaging device 2, the speed detection device 3, the object recognition rate calculation device 4, the communication device 5, the operator terminal 6, and the communication device 8. In particular, there may be multiple imaging devices 2 to capture images of the front, right side, left side, rear, and interior of the moving object A.

[0019] In this embodiment, the mobile unit A is assumed to be, for example, a vehicle (automobile) having communication and autonomous driving functions. However, the mobile unit A is not limited to automobiles, and may also be agricultural machinery such as a tractor having communication and autonomous driving functions. Furthermore, the mobile unit A may be a ship, aircraft, or the like, in addition to a vehicle.

[0020] The imaging device 2 has a surveillance camera mounted on the mobile body A and an encoder for compressing video data. It encodes video data obtained by imaging the outside or inside of the mobile body A and transmits it to the communication device 5. Furthermore, the imaging device 2 has a function for extracting video transmission parameters such as bitrate, resolution, and frame rate. It extracts the bitrate, resolution, and frame rate of the video from the encoded video data and transmits bitrate information indicating the bitrate, resolution information indicating the resolution, and frame rate information indicating the frame rate, respectively, to the object recognition rate calculation device 4 every unit time t (for example, 1 second). Note that the imaging device 2 and the encoder (device) may be separate entities.

[0021] The velocity detection device 3 detects the speed of the moving object A (for example, h in kilometers per hour) and transmits velocity information indicating the detected speed to the object recognition rate calculation device 4. This velocity information is also transmitted every unit of time t, similar to the bitrate information, resolution information, and frame rate information.

[0022] The object recognition rate calculation device 4 has a calculation unit 9 that calculates the object recognition rate based on the information acquired from the imaging device 2 (bitrate information, resolution information, and frame rate information), packet loss rate information acquired from the communication device 8 (transmission information acquisition unit 7), which will be described later, and speed information acquired from the speed detection device 3. This calculation unit 9 is a function that is realized by the execution of a program. In the object recognition rate calculation device 4, the calculation is performed using information transmitted from the imaging device 2, the speed detection device 3, and the operator terminal 6 (transmission information acquisition unit 7) at the same time. For example, the calculation unit 9 adds timestamp information to the transmitted information so that even if a delay occurs in the communication network 100, the object recognition rate can be calculated based on information at the same time.

[0023] Furthermore, the "object recognition rate" indicates the percentage of people who were able to actually recognize an object when multiple people viewed the aforementioned video. More specifically, it indicates the percentage of people who, when multiple monitors or other personnel viewed surveillance footage, were able to recognize an object within the time required to prevent the autonomous vehicle A from colliding with the object when the object appeared at a target distance from the vehicle A.

[0024] Here, object recognition refers to recognizing the appearance of an object, not identifying its type. The target distance refers to the distance from the camera 2 of the mobile body A, which is capturing surveillance footage, to the appearing object, and is the distance at which mobile body A can stop safely without colliding with the object. For example, this can be calculated in advance from at least one of the following: glass-to-glass delay (the delay until the captured surveillance footage is displayed), reaction delay (the delay until the monitor recognizes the object and reacts, such as pressing a brake command), control delay (the delay from when a brake command is executed until mobile body A actually stops), the speed of mobile body A, and the coefficient of friction of the road surface. The time within which a collision can be prevented refers to the reaction delay. The coefficient of friction of the road surface may be a value predetermined by the region or weather, or it may be a value estimated by AI (artificial intelligence) based on the footage captured by the camera 2.

[0025] For example, if there are 100 monitors with varying levels of visual acuity, and 50 of them are able to recognize an object displayed in the surveillance video at a predetermined target distance ahead of moving object A within a time limit (a set reaction delay time) to prevent a collision, then the object recognition rate is 50%. The object recognition rate calculation device 4 then transmits the calculated object recognition rate information to the operator terminal 6 at the monitoring center B via the communication device 5. As a result, the operator terminal 6 displays the video from each moving object (surveillance video) along with the object recognition rate information (e.g., 70%) for each moving object, as shown in Figure 2. Note that the method of notifying the operator is not limited to displaying on the screen; it may also include voice notifications or alert notifications when a certain threshold is exceeded. The method of calculating the object recognition rate will be explained in detail later.

[0026] Communication device 5 can communicate data with operator terminal 6 via communication network 100 and communication device 8. Communication network 100 is constructed using mobile networks, the internet, etc. The internet also includes space internet, which is accessed via space using artificial satellites, etc.

[0027] Operator terminal 6 is a PC (Personal Computer) or similar terminal used by operators at monitoring center B. Operator terminal 6 has a decoder for decoding encoded video data.

[0028] The transmission information acquisition unit 7, a function of the communication device 8, calculates the packet loss rate as an example of a transmission status parameter that shows the impact of data loss on the video transmitted to the operator terminal 6 installed in the monitoring center B. This packet loss rate information is calculated every unit time t and transmitted from the communication device 8 to the object recognition rate calculation device 4 via the communication device 5.

[0029] The communication device 8 can communicate data with the object recognition rate calculation device 4 via the communication network 100 and the communication device 5.

[0030] With the above configuration, bitrate information, resolution information, and frame rate information transmitted from the imaging device 2, as well as speed information acquired by the speed detection device 3, are sent to the object recognition rate calculation device 4. Furthermore, the video data transmitted from the imaging device 2 is sent to the operator terminal 6 via the communication devices 5 and 8. The packet loss rate information transmitted from the communication device 8 (transmission information acquisition unit 7) is sent to the object recognition rate calculation device 4 via the communication device 5. The object recognition rate information, which indicates the object recognition rate calculated by the object recognition rate calculation device 4, is transmitted from the object recognition rate calculation device 4 and sent to the operator terminal 6 via the communication devices 5 and 8.

[0031] [Detailed Calculation Method for Object Recognition Rate] Next, Figures 3 to 6 illustrate the method for calculating the object recognition rate calculated by the object recognition rate calculation device 4. Here, we will explain a method for determining the object recognition rate Z using the bitrate b, resolution r, frame rate f, speed v, and packet loss rate p of the video input to the device.

[0032] <Relationship between bitrate, resolution, frame rate and object recognition rate> Video quality is correlated with bitrate; the higher the bitrate, the higher the video quality and the higher the object recognition rate. The effects of resolution and frame rate on the same bitrate vary depending on the bitrate and encoding method. Generally, at high bitrates, lower resolution results in higher video quality and a higher object recognition rate. For the same bitrate, higher frame rates result in smoother motion, but the amount of data allocated per frame decreases, which tends to lower the image quality of each frame. Generally, at low bitrates, lower frame rates tend to result in a higher object recognition rate. Figure 3 shows the relationship between bitrate, resolution, frame rate and object recognition rate.

[0033] Figure 3 shows the first object recognition rate model. The first object recognition rate model shows the relationship between the object recognition rate and the first parameters related to the video quality of the video (bitrate, resolution, and frame rate, etc.). Specifically, the first object recognition rate model shows the relationship between the bitrate and the object recognition rate for each resolution and frame rate at the same speed of the moving object A. Although three curves are shown in Figure 3, this is only due to the constraints of the drawing, and in reality, the first object recognition rate model has all possible patterns of resolution and frame rate. Furthermore, bitrate, resolution, and frame rate are examples of the first parameters related to the video transmission parameters or video quality of the video.

[0034] While the object recognition rate increases with higher bitrates, once a certain bitrate is reached, the video quality almost reaches its upper limit, and the object recognition rate hardly changes even at higher bitrates. Therefore, the object recognition rate calculation device 4 calculates the object recognition rate using an object recognition rate model in which the object recognition rate increases with increasing bitrate, but the object recognition rate hardly changes above a predetermined bitrate.

[0035] The object recognition rate calculation device 4 has a calculation unit (first calculation unit) that calculates the object recognition rate based on video transmission parameters using a first object recognition rate model.

[0036] <Relationship between packet loss rate and object recognition rate for missing frames> The more missing frames there are, the later it is possible to recognize an object, and the lower the object recognition rate becomes. Figure 4 shows the relationship between the number of missing frames and the object recognition rate.

[0037] Figure 4 shows the second object recognition rate model. The second object recognition rate model shows the relationship between the object recognition rate and the first parameter (bitrate, resolution, and frame rate) and the second parameter (number of missing frames, etc.). Specifically, the second object recognition rate model shows the relationship between the number of missing frames (horizontal axis), which is the second parameter related to the video playback status on the operator terminal 6, and the object recognition rate (vertical axis) for each frame rate, with the same parameters (same bitrate and resolution, and the same travel speed of the moving object A).

[0038] Note that although two curves are shown in Figure 4, this is only due to the constraints of the drawing; in reality, the second object recognition rate model encompasses all possible frame rate patterns.

[0039] Furthermore, the second object recognition rate model may show the relationship between the number of missing frames (horizontal axis) and the object recognition rate (vertical axis) for each bitrate, under the same parameters (same frame rate and resolution, and the same speed of moving object A). Alternatively, the second object recognition rate model may show the relationship between the number of missing frames (horizontal axis) and the object recognition rate (vertical axis) for each resolution, under the same parameters (same bit rate and frame rate, and the same speed of moving object A). Note that the relationship between the vertical and horizontal axes may be reversed.

[0040] Incidentally, when the number of missing frames reaches a certain level or more, the object recognition rate begins to decrease. When the number of missing frames is the same as the number of frames per unit time (= frame rate × t), that is, when all frames are missing, the object recognition rate converges to 0. Therefore, as the number of missing frames increases, the object recognition rate calculation device 4 uses a model of the object recognition rate (second object recognition rate model) that decreases as the number of missing frames increases and becomes 0 when the number of missing frames is the same as the number of frames per unit time, to calculate the object recognition rate. At this time, the lower the frame rate, the longer the length of time missing for each missing frame. Therefore, for the same number of missing frames, the lower the frame rate, the lower the object recognition rate.

[0041] Incidentally, the number of missing frames is an example of a replacement parameter in which a reproduction status parameter of the video or a second parameter related to the reproduction status of the video is replaced.

[0042] The object recognition rate calculation device 4 has a calculation unit (second calculation unit), the calculation unit 9, which uses the second object recognition rate model and calculates the object recognition rate based on the reproduction status parameter information.

[0043] (Relationship between packet loss rate and number of missing frames) The number of missing frames increases as the packet loss rate increases. As a result, as shown in FIG. 4, the object recognition rate decreases. The relationship between the packet loss rate and the number of missing frames is shown in FIG. 5. FIG. 5 is a diagram showing an example of a replacement model that is part of the second object recognition rate model (included in the second object recognition rate model). The replacement model shows the relationship between the packet loss rate and the number of missing frames at the same bit rate, resolution, and frame rate. As the packet loss rate increases, the number of missing frames increases. When the packet loss rate is 100%, all frames of the video are missing, so the number of missing frames is the same as the number of frames per unit time (= frame rate × t). Incidentally, the replacement model may be referred to as a conversion model.

[0044] Incidentally, the packet loss rate is an example of a transmission status parameter related to the transmission status of the video or a second parameter related to the transmission status of the video.

[0045] As described above, the object recognition rate calculation apparatus 4 has a calculation unit 9 (second calculation unit) that calculates the object recognition rate based on the packet loss rate information using the second object recognition rate model including the substitution model. This calculation unit 9 is a function realized by executing a program.

[0046] <Relationship between the object recognition rate and the speed> Also, the object recognition rate calculation apparatus 4 can handle different output recognition rates when the speed of the moving body A is different, with respect to the object recognition rate obtained using the first object recognition rate model by inputting the speed information of the moving body A. FIG. 6 is a diagram showing the third object recognition rate model. The third object recognition rate model shows the relationship between the first parameters (bit rate, resolution, and frame rate) and the speed of the moving body A and the object recognition rate. Specifically, the third object recognition rate model is a diagram showing an example of a model (third object recognition rate model) that shows the relationship between the bit rate (horizontal axis) and the object recognition rate (vertical axis) for each speed of the moving body A at the same resolution and frame rate. When the speed of the moving body A is high, the movement of the background and objects in the video becomes large (fast), making video encoding difficult. Therefore, even when the video is delivered at the same bit rate, the video quality deteriorates and the object recognition rate is considered to decrease. Also, when the speed of the moving body A per hour is high, the distance that the moving body A moves before a person such as an operator recognizes an object becomes long. Therefore, since the object must be recognized from a farther distance, this also affects the decrease in the object recognition rate. The object recognition rate calculation apparatus 4 calculates the object recognition rate using these properties.

[0047] In FIG. 6, although three curves are shown, due to the limitations of the drawing, only the three curves are explicitly shown. In reality, the third object recognition rate model has all patterns of hourly speeds.

[0048] Furthermore, the third object recognition rate model has a relationship with each of the first parameters or combinations used in the first object recognition rate model for each speed. Specifically, the relationship between frame rate (horizontal axis) and object recognition rate (vertical axis) may be shown for each speed of moving object A, with the same parameters (same bitrate and resolution). Alternatively, the third object recognition rate model may show the relationship between resolution (horizontal axis) and object recognition rate (vertical axis) for each speed of moving object A, with the same parameters (same frame rate and bitrate). Also, due to the constraints of the figure, the relationship when two parameters are the same is shown, but the same relationship may be shown even when one of the first parameters is the same.

[0049] The object recognition rate calculation device 4 has a calculation unit 9 (third calculation unit) that calculates the object recognition rate based on the speed (movement speed) of the moving object A using a third object recognition rate model.

[0050] Based on the above, the object recognition rate calculation device 4 uses a first object recognition rate model that shows the relationship between the first parameter and the object recognition rate, and calculates the object recognition rate based on the first parameter. Furthermore, the object recognition rate calculation device 4 uses a first object recognition rate model that shows the relationship between the first parameter and the object recognition rate, and a second object recognition rate model that shows the relationship between the second parameter and the object recognition rate, and calculates the object recognition rate based on the first parameter and the second parameter. Specifically, it uses the first object recognition rate model to calculate a predetermined object recognition rate based on the first parameter, and uses the second object recognition rate model to replace the second parameter with the number of missing frames, and calculates the object recognition rate based on the predetermined object recognition rate and the replaced number of missing frames. Furthermore, the object recognition rate calculation device 4 uses a third object recognition rate model that shows the relationship between the first parameter and velocity and the object recognition rate, and calculates the object recognition rate based on the first parameter and velocity. Furthermore, the object recognition rate calculation device 4 uses a second object recognition rate model that shows the relationship between the second parameter and the object recognition rate, and a third object recognition rate model that shows the relationship between the first parameter and velocity and the object recognition rate, to calculate the object recognition rate based on the first parameter, the second parameter and velocity. Specifically, it uses the third object recognition rate model to calculate the object recognition rate based on the first parameter and velocity, and uses the second object recognition rate model to replace the second parameter with the number of missing frames, and calculates the object recognition rate based on a predetermined object recognition rate and the replaced number of missing frames.

[0051] <Method for Calculating Object Recognition Rate by Object Recognition Rate Calculation Device> This section describes the method for calculating the object recognition rate performed by the object recognition rate calculation device 4, specifically the method for determining the object recognition rate Z when using the video bitrate b, resolution r (=height × width), frame rate f, packet loss rate p, and speed v. The formula for calculating the object recognition rate Z is expressed as an equation f(b, r, f, p, v) with bitrate information, resolution information, frame rate information, packet loss rate information, and speed information as inputs. As shown in Figure 6, the function f(b, r, f, p, v) has the characteristic that the output object recognition rate decreases as the speed increases. Taking this into account, f(b, r, f, p, v) is calculated using one of the formulas (Equation 1) to (Equation 4). Note that f(b, r, f, p, v) is defined in the range of 0 to 1, and in each of the following equations (Equation 1) to (Equation 4), if f(b, r, f, p, v) is 0 or less, the object recognition rate is defined as 0, and if f(b, r, f, p, v) is 1 or greater, the object recognition rate is defined as 1.

[0052]

[0053]

[0054]

[0055] Here, in (Equation 1) to (Equation 4), c1 to c 13 is a coefficient that differs for each formula, and t is the unit time [seconds] for calculating the information. Also, log is a logarithm with base e, which is the base of the natural logarithm. Here, by setting the above coefficients to values ​​according to conditions such as encoding conditions (codec, GoP length (and GoP structure), profile, etc.), decoding implementation in case of data loss (indicating whether it is block noise or freeze), size and color of the object, display resolution, size of the object displayed on the display, weather conditions during driving, driving time (day / night, etc.), road surface friction coefficient, glass-to-glass delay, control delay, reaction delay, and shooting conditions of the shooting device 2 (camera mode, focal length, Field of View, image stabilization, white balance delay, lens flare, etc.), it is possible to estimate the object according to the driving conditions and monitoring environment.

[0056] [Hardware Configuration] Next, the electrical hardware configuration of the object recognition rate calculation device 4, communication device 5, operator terminal 6, and communication device 8 will be described using Figure 7. Figure 7 is an electrical hardware configuration diagram of each device and terminal according to the embodiment.

[0057] The object recognition rate calculation device 4, as a computer, includes a CPU 101, ROM 102, RAM 103, SSD 104, external device connection interface 105, network interface 106, display 107, input device 108, media interface 109, and bus line 110, as shown in Figure 7.

[0058] Of these, the CPU 101 controls the operation of the entire object recognition rate calculation device 4. The ROM 102 stores programs used to drive the CPU 101, such as the IPL. The RAM 103 is used as the work area for the CPU 101.

[0059] The SSD 104 reads or writes various types of data according to the control of the CPU 101. Alternatively, an HDD (Hard Disk Drive) may be used instead of the SSD 104.

[0060] External device connection I / F 105 is an interface for connecting various external devices. These external devices include displays, speakers, keyboards, mice, USB memory sticks, and printers.

[0061] The network interface 106 is an interface for data communication via the communication network 100.

[0062] The display 107 is a type of display means, such as a liquid crystal display or an organic EL (electroluminescence) display, which displays various images.

[0063] The input device 108 is an example of an input means for receiving input operations such as characters, numbers, and various instructions, and may include buttons, switches, keyboards, pointing devices, etc.

[0064] The media interface 109 controls the reading or writing (storage) of data to or from a recording medium 109m, such as flash memory. The recording medium 109m includes DVDs, Blu-ray Discs (registered trademarks), and the like.

[0065] The bus line 110 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 101 shown in Figure 7.

[0066] Since the communication device 5, operator terminal 6, and communication device 8 have the same configuration as the object recognition rate calculation device 4, their explanation will be omitted.

[0067] [Processing or Operation of the Communication System According to the First Embodiment] Next, the processing or operation of each device in the communication system 1a will be explained using Figures 8 and 9. Figures 8 and 9 are flowcharts illustrating the process of estimating and presenting the object recognition rate according to the first embodiment.

[0068] S11: The imaging device 2 captures video data from the outside or inside of the moving object A, then compresses the video data using an encoder and transmits it to the communication device 5.

[0069] S12: The communication device 5 transmits video data to the operator terminal 6 via the communication network 100 and the communication device 8.

[0070] S13: The imaging device 2 uses the encoding information of the video data to extract the bitrate, resolution, and frame rate of the video, and transmits the bitrate information (e.g., 5Mbps, 10Mbps), resolution information (e.g., 640x360, 1280x720), and frame rate information (e.g., 10fps, 30fps) to the object recognition rate calculation device 4.

[0071] S14: The speed detection device 3 detects the speed of the moving object A and transmits speed information indicating this speed to the object recognition rate calculation device 4.

[0072] S15: The transmission information acquisition unit 7 extracts the packet loss rate during video transmission to the operator terminal 6, that is, the packet loss rate (for example, 0%, 5%) for the video transmission information to be sent to the operator terminal 6.

[0073] S16: The communication device 8 transmits packet loss rate information, which is extracted by the transmission information acquisition unit 7, to the object recognition rate calculation device 4 via the communication network 100 and the communication device 5.

[0074] S17: The object recognition rate calculation device 4 uses the first to third object recognition rate models and calculates the final object recognition rate (for example, 70%) based on bitrate information, resolution information, frame rate information, packet loss rate information, and speed information.

[0075] Here, we will specifically explain how to calculate the object recognition rate using the first to third object recognition rate models. For example, the calculation unit 9 uses the first object recognition rate model (Figure 3) at a reference speed of 30 km / h to calculate a predetermined object recognition rate based on a bitrate of 1.2 Mbps at a resolution of 1280 × 720 and a frame rate of 30 fps. Then, if the speed indicated by the speed information is 35 km / h, the calculation unit 9 uses the third object recognition rate model (Figure 6) to adjust the predetermined object recognition rate by selecting a curve for 35 km / h that is closer to 50 km / h from the curve for a reference speed of 30 km / h. After that, the calculation unit 9 uses a substitution model (Figure 5) to substitute the packet loss rate with the number of missing frames, then uses the second object recognition rate model (Figure 4) to select a curve for a frame rate of 30 fps, and calculates the final object recognition rate based on the number of missing frames (5 frames) indicated by the number of missing frames information. The object recognition rate calculation device 4 then transmits object recognition rate information indicating the final object recognition rate to the communication device 5.

[0076] S18: The communication device 5 transmits the object recognition rate information received from the object recognition rate calculation device 4 to the operator terminal 6 via the communication network 100 and the communication device 8.

[0077] As a result, as shown in Figure 2, the operator terminal 6 decodes the video data sent from each mobile object by processing S12, associates it with the object recognition rate information related to the video of each mobile object sent from the communication device 5 by processing S18, and displays it on the display by splitting the screen. In order to associate the video with the object recognition rate information, identification information such as an ID may be included in both the video data and the object recognition rate information. In this case, the identification information is added to the video data and bitrate information output by the shooting device 2. The object recognition rate calculation device 4 then adds the identification information added to the bitrate information acquired from the shooting device 2 to the object recognition rate information and outputs it.

[0078] S19: If the movement of mobile body A is not completed (NO), the process returns to S11. On the other hand, if the movement of mobile body A is completed (YES), the processes shown in Figures 8 and 9 are completed.

[0079] [Effects of the First Embodiment] As described above, according to the first embodiment, the object recognition rate by the operator of the monitoring center B can be determined and presented. As a result, the operator terminal 6 of the monitoring center B displays the monitoring video from each moving object in association with the object recognition rate information of this monitoring video, as shown in Figure 2, thus eliminating the situation in which the operator does not notice that the video quality of the monitoring video has deteriorated.

[0080] ●Second Embodiment [System Configuration of the Second Embodiment] Next, the communication system 1b according to the second embodiment will be described using Figure 10. Figure 10 is an overall configuration diagram of the communication system according to the second embodiment. The second embodiment differs from the configuration of the first embodiment in that it does not use the speed detection device 3. That is, the second embodiment shows an example where the speed of the mobile body A is unknown. Note that the values ​​shown in Figure 10 (1.2 Mbps, etc.) are just examples, as with Figure 1, and are not limiting. Also, devices or terminals that are the same as in Figure 1 are denoted by the same reference numerals and their descriptions are omitted.

[0081] <Method for Calculating Object Recognition Rate by Object Recognition Rate Calculation Device> The method for calculating the object recognition rate performed by the object recognition rate calculation device 4 is explained below. Here, we will explain a method for determining the object recognition rate Z using the video bitrate b, resolution r (=height × width), frame rate f, and packet loss rate p. The formula for calculating the object recognition rate Z is expressed as an equation f(b, r, f, p) with bitrate information, resolution information, frame rate information, and packet loss rate information as inputs. The function f(b, r, f, p) has the characteristics of the model shown in Figures 3 and 4. Based on this, f(b, r, f, p) is calculated using one of the following formulas (Equation 5) to (Equation 8). Note that f(b, r, f, p) is defined in the range of 0 to 1, and in each of the following formulas, if f(b, r, f, p) is 0 or less, the object recognition rate is defined as 0, and if f(b, r, f, p) is 1 or greater, the object recognition rate is defined as 1. Under conditions where resolution and frame rate remain constant, formulas that do not take the relevant parameters as inputs may be used. Furthermore, the formulas used to calculate the object recognition rate are not limited to those shown below.

[0082]

[0083]

[0084]

[0085] Here, in (Equation 5) to (Equation 8), c1 to c 10is a coefficient that differs for each formula, and t is the unit time [seconds] for calculating the information. Also, log is a logarithm with base e, which is the base of the natural logarithm. Here, by setting the above coefficients to values ​​corresponding to conditions such as encoding conditions (codec, GoP length (and GoP structure), profile, etc.), decoding implementation in case of data loss (indicating whether it is block noise or freeze), size and color of the object, display resolution, size of the object displayed on the display, weather conditions during driving, driving time (day / night, etc.), road surface friction coefficient, glass-to-glass delay, control delay, reaction delay, and shooting conditions of the shooting device 2 (camera mode, focal length, Field of View, image stabilization, white balance delay, lens flare, etc.), it is possible to estimate the object according to the driving conditions and monitoring environment.

[0086] [Processing or Operation of the Communication System According to the Second Embodiment] In the second embodiment, as a difference from the flowchart showing the processing of the first embodiment shown in Figures 8 and 9, S14 is not performed, and in S17, the object recognition rate calculation device 4 uses the first to third models to calculate the object recognition rate based on bitrate information, resolution information, frame rate information, and packet loss rate information. In the second embodiment, since the third object recognition model is not used, the calculation unit 9 uses, for example, only the second object recognition rate model to calculate the final object recognition rate.

[0087] [Effects of the Second Embodiment] As described above, according to the second embodiment, even when the travel speed is constant or unknown, the recognition rate of objects by the operator of the monitoring center B can be determined and presented.

[0088] ●Third Embodiment [System Configuration of the Third Embodiment] Next, the communication system 1c according to the third embodiment will be described with reference to Figures 11 to 13. Figure 11 is an overall configuration diagram of the communication system according to the third embodiment. Note that the values ​​shown in Figure 11 (1.2 Mbps, etc.) are examples, as with Figure 1, and are not limited to these. Also, the same devices or terminals as in Figure 1 are denoted by the same reference numerals and their descriptions are omitted.

[0089] In the communication system 1c of Figure 11, the object recognition rate calculation device 4, which was installed on the mobile body A side in the first embodiment (Figure 1), is installed on the monitoring center B side. The video data encoded by the imaging device 2 is transmitted to the operator terminal 6 via the communication devices 5 and 8. However, in this embodiment, the operator terminal 6, not the imaging device 2, has the bitrate extraction function. In addition, the speed information indicating the speed of mobile body A detected by the mobile body A speed detection device 3 is transmitted to the object recognition rate calculation device 4 via the communication devices 5 and 8. This transmission is performed every unit time t (for example, 1 second).

[0090] The calculation unit 9 of the object recognition rate calculation device 4 calculates the current (final) object recognition rate based on bitrate information, resolution information, frame rate information, packet loss rate information, and speed information, similar to the first embodiment. The object recognition rate calculation device 4 then transmits this object recognition rate information to the operator terminal 6.

[0091] [Processing or Operation of the Communication System According to the Third Embodiment] Next, the processing or operation of each device in the communication system 1c will be explained using Figures 12 and 13. Figures 12 and 13 are flowcharts illustrating the process of estimating and presenting the object recognition rate according to the third embodiment.

[0092] S21: The imaging device 2 takes a picture of the outside or inside of the moving object A to obtain video data, then compresses the video data using an encoder and transmits it to the communication device 5.

[0093] S22: The speed detection device 3 detects the current speed of the moving object A and transmits speed information indicating this speed to the communication device 5.

[0094] S23: The communication device 5 transmits video data to the operator terminal 6 via the communication network 100 and the communication device 8, and also transmits speed information to the object recognition rate calculation device 4.

[0095] S24: The operator terminal 6 uses the encoding information of the video data to extract the bitrate, resolution, and frame rate of the video, and transmits the bitrate information (e.g., 5Mbps, 10Mbps), resolution information (e.g., 640x360, 1280x720), and frame rate information (e.g., 10fps, 30fps) to the object recognition rate calculation device 4.

[0096] S25: The transmission information acquisition unit 7 extracts the packet loss rate during video transmission to the operator terminal 6, that is, the packet loss rate for the video transmission information received by the operator terminal 6. The communication device 8 then transmits this packet loss rate information (for example, 0%, 5%) to the object recognition rate calculation device 4.

[0097] S26: The object recognition rate calculation device 4 uses the models shown in Figures 3, 4, 5, and 6 to calculate the object recognition rate (for example, 70%) based on bitrate information, resolution information, frame rate information, packet loss rate information, and speed information. The object recognition rate calculation device 4 then transmits the object recognition rate information to the operator terminal 6.

[0098] As a result, as shown in Figure 2, the operator terminal 6 decodes the video data sent from each moving object by processing S23, associates it with the object recognition rate information related to the video of each moving object sent from the object recognition rate calculation device 4 by processing S26, and displays it on the display by splitting the screen. In this case as well, identification information such as an ID may be included in both the video data and the object recognition rate information in order to associate the video with the object recognition rate information. In this case, the shooting device 2 adds the identification information to the video data it outputs. Then, when the operator terminal 6 extracts the bitrate, it adds the same identification information to the bitrate information. The object recognition rate calculation device 4 adds the identification information added to the bitrate information obtained from the operator terminal 6 to the object recognition rate information and transmits it to the operator terminal 6.

[0099] S27: If the movement of mobile body A is not completed (NO), the process returns to S21. On the other hand, if the movement of mobile body A is completed (YES), the processes shown in Figures 12 and 13 are completed.

[0100] [Effects of the Third Embodiment] As described above, the third embodiment provides the same effects as the first embodiment. Furthermore, the third embodiment requires fewer devices on the mobile body A side compared to the first embodiment, thus reducing the initial costs for receiving remote monitoring and remote control services from this communication system 1c.

[0101] ●Fourth Embodiment Next, the communication system 1d according to the fourth embodiment will be described using Figure 14. Figure 14 is an overall configuration diagram of the communication system according to the fourth embodiment. Note that the values ​​shown in Figure 14 (1.2 Mbps, etc.) are examples, as with Figure 1, and are not limited to these. Also, the same reference numerals are used for devices or terminals as in Figure 1, and their descriptions are omitted.

[0102] In the fourth embodiment, as a difference from the flowchart showing the processing of the third embodiment shown in Figures 12 and 13, processing S22 is not performed. Also, in processing S26, the object recognition rate calculation device 4 uses the first object recognition rate model (Figure 3) and the second object recognition rate model (Figure 4), which includes the substitution model (Figure 5), but, similar to the second embodiment, it does not use the third object recognition rate model (Figure 6) and calculates the object recognition rate based on bitrate information, resolution information, frame rate information, and packet loss rate information. The fourth embodiment differs from the third embodiment (Figure 11) in that the speed detection device 3 is not used. That is, the fourth embodiment shows an example where the speed of the moving object A is fixed (constant).

[0103] [Effects of the Fourth Embodiment] As described above, the fourth embodiment provides the same effects as the second embodiment. Furthermore, in the fourth embodiment, since fewer devices are required on the mobile body A side compared to the second embodiment, the initial cost for receiving remote monitoring and remote control services using this communication system 1d can be reduced.

[0104] [Relationship between each object recognition rate model and the formula] First object recognition rate model: Expressed by the formulas X, A1, A2, A3 in any one of the formulas (Formula 5), ​​(Formula 6), (Formula 7), (Formula 8) Second object recognition rate model: Expressed in the denominator of the formula Z in any one of the formulas (Formula 1), (Formula 2), (Formula 3), (Formula 4), (Formula 5), ​​(Formula 6), (Formula 7), (Formula 8) Third object recognition rate model: Expressed by the formulas X, A1, A2, A3 in any one of the formulas (Formula 1), (Formula 2), (Formula 3), (Formula 4) Object recognition rate: The value derived from the formula Z in (Formula 5) [Note] The above embodiment is not limited to the above embodiment, and various modifications and applications are possible, for example, as shown below.

[0105] (1) In the above embodiment, the imaging device 2 or operator terminal 6 extracts the bitrate from the video data and transmits the bitrate information, and the object recognition rate calculation device 4 calculates the object recognition rate using the bitrate information, but it is not limited to this. For example, the imaging device 2 or operator terminal 6 uses bitrate information, resolution information, and frame rate information, but it may calculate the object recognition rate using only some of this information. Note that bitrate information, resolution information, and frame rate information are examples of video quality parameters, and other parameters may be used. Also, packet loss rate is an example of a parameter of video transmission status, and other parameters may be used.

[0106] (2) Each device can be implemented using a computer and a program, and this program can be recorded on a (non-temporary) recording medium or provided via a communication network such as the Internet. Furthermore, by storing the first to third object recognition rate models in a device (storage device) such as memory, a PC, or a server, the device can have at least one of the first to third object recognition rate models.

[0107] (3) The CPU 101, which is a hardware processor, may be single or multiple.

[0108] [Other] Each of the above embodiments can also be represented by the following supplementary information.

[0109] [Addendum 1] An object recognition rate calculation device that calculates a final object recognition rate, which indicates the percentage of people who were actually able to recognize an object when multiple people viewed the video, based on a first parameter relating to the video quality of the video acquired from a camera installed on a mobile body, and a second parameter relating to the transmission status of the video to the operator terminal.

[0110] [Appendix 2] An object recognition rate calculation device according to Appendix 1, which uses a first object recognition rate model showing the relationship between the first parameter and the object recognition rate, and a second object recognition rate model showing the relationship between the second parameter and the object recognition rate, and calculates the final object recognition rate based on the first parameter and the second parameter.

[0111] [Appendix 3] An object recognition rate calculation device according to Appendix 2, which uses the first object recognition rate model to calculate a predetermined object recognition rate based on the first parameters, and uses the second object recognition rate model to calculate the final object recognition rate based on the predetermined object recognition rate and the second parameters.

[0112] [Appendix 4] The object recognition rate calculation device according to Appendix 3, wherein the second parameter is the packet loss rate during transmission of the video to the operator terminal, and the packet loss rate is replaced with the number of missing frames using a substitution model that shows the substitution relationship between the packet loss rate and the number of missing frames relating to the playback status of the video at the operator terminal, and the final object recognition rate is calculated based on the predetermined object recognition rate and the number of missing frames using the second object recognition rate model that shows the relationship between the number of missing frames and the object recognition rate.

[0113] [Appendix 5] An object recognition rate calculation device according to appendix 3 or 4, which adjusts the predetermined object recognition rate based on the speed of the moving body obtained from a speed detection device provided on the moving body, and calculates the final object recognition rate based on the adjusted predetermined object recognition rate and the second parameter using the second object recognition rate model.

[0114] [Appendix 6] An object recognition rate calculation device according to Appendix 5, which adjusts the predetermined object recognition rate using the first parameter and a third object recognition rate model that shows the relationship between the speed and the object recognition rate.

[0115] [Appendix 7] An object recognition rate calculation method that calculates a final object recognition rate, which indicates the percentage of people who were actually able to recognize an object when multiple people viewed the video, based on a first parameter relating to the video quality of the video acquired from a camera installed on a mobile body and a second parameter relating to the transmission status of the video to an operator terminal.

[0116] [Appendix 8] A program that causes a computer to perform the method described in Appendix 7.

[0117] [Relationship with the Basic Application] This patent application claims priority based on the international patent application PCT / JP2024 / 032933 filed on September 13, 2024, and the entire contents of the international patent application PCT / JP2024 / 032933 are incorporated herein by reference.

[0118] 1a, 1b, 1c, 1d Communication system A Mobile object B Monitoring center 2 Imaging device 3 Speed ​​detection device 4 Object recognition rate calculation device 5 Communication device 6 Operator terminal 7 Transmission information acquisition unit 8 Communication device 9 Calculation unit 100 Communication network

Claims

1. An object recognition rate calculation device that calculates a final object recognition rate, which indicates the percentage of multiple persons who, when viewing the video, can recognize an object within the time required to prevent the moving body from colliding with the object when the object appears at a target distance from the moving body, based on a first parameter relating to the video quality of the video acquired from a camera installed on a moving body and a second parameter relating to the transmission status of the video to an operator terminal.

2. An object recognition rate calculation device according to claim 1, which uses a first object recognition rate model showing the relationship between the first parameter and the object recognition rate, and a second object recognition rate model showing the relationship between the second parameter and the object recognition rate, and calculates the final object recognition rate based on the first parameter and the second parameter.

3. An object recognition rate calculation device according to claim 2, comprising: using the first object recognition rate model to calculate a predetermined object recognition rate based on the first parameters; and using the second object recognition rate model to calculate the final object recognition rate based on the predetermined object recognition rate and the second parameters.

4. The object recognition rate calculation device according to claim 3, wherein the second parameter is the packet loss rate during transmission of the video to the operator terminal, and includes a substitution model that shows the substitution relationship between the packet loss rate and the number of missing frames relating to the playback status of the video at the operator terminal, and uses the second object recognition rate model that shows the relationship between the number of missing frames and the object recognition rate by substituting the packet loss rate with the number of missing frames, to calculate the final object recognition rate based on the predetermined object recognition rate and the number of missing frames substituted from the packet loss rate.

5. The object recognition rate calculation device according to claim 1, which calculates the final object recognition rate based on the speed of the moving body obtained from a speed detection device provided on the moving body.

6. An object recognition rate calculation device according to claim 5, which calculates the final object recognition rate based on the first parameter, the speed, and the second parameter, using a third object recognition rate model showing the relationship between the first parameter and the speed and the object recognition rate, and a second object recognition rate model showing the relationship between the second parameter and the object recognition rate.

7. An object recognition rate calculation device according to claim 6, comprising: using the third object recognition rate model to calculate a predetermined object recognition rate based on the first parameters and velocity; and using the second object recognition rate model to calculate the final object recognition rate based on the predetermined object recognition rate and the second parameters.

8. The object recognition rate calculation device according to claim 7, wherein the second parameter is the packet loss rate during transmission of the video to the operator terminal, and includes a substitution model that shows a substitution relationship between the packet loss rate and the number of missing frames relating to the playback status of the video at the operator terminal, and the final object recognition rate is calculated based on the predetermined object recognition rate and the number of missing frames substituted from the packet loss rate using the second object recognition rate model that shows the relationship between the number of missing frames and the object recognition rate by substituting the packet loss rate with the number of missing frames.

9. A device having a first object recognition rate model that shows the relationship between a first parameter relating to the image quality of an image acquired from a camera installed on a moving object, and an object recognition rate that indicates the proportion of multiple persons who, when viewing the image, can recognize an object within a time that prevents the moving object from colliding with the object when the object appears at a target distance from the moving object.

10. A device having a second object recognition rate model that shows the relationship between a second parameter relating to the transmission status of the video acquired from a shooting device installed on a moving body to an operator terminal that displays the video, and an object recognition rate indicating the percentage of people who, when multiple people view the video, can recognize an object within a time that prevents the moving body from colliding with the object when the object appears at a target distance from the moving body.

11. The apparatus according to claim 10, wherein the second parameter is the packet loss rate, and includes a substitution model that shows a substitution relationship between the packet loss rate and the number of missing frames relating to the playback status of the video at the operator terminal, and has a second object recognition rate model that substitutes the packet loss rate with the number of missing frames and shows a relationship between the number of missing frames and the object recognition rate.

12. A device having a third object recognition rate model that shows the relationship between a first parameter relating to the image quality of images acquired from a camera installed on a moving body, the speed of the moving body, and an object recognition rate that indicates the proportion of multiple people who, when viewing images acquired from a camera installed on the moving body, can recognize an object within the time required to prevent the moving body from colliding with the object when the object appears at a target distance from the moving body.

13. An object recognition rate calculation method that calculates a final object recognition rate, which indicates the percentage of multiple persons who, when viewing the video, can recognize an object within the time required to prevent the moving body from colliding with the object when the object appears at a target distance from the moving body, based on a first parameter relating to the video quality of the video acquired from a camera installed on a moving body and a second parameter relating to the transmission status of the video to an operator terminal.

14. A program that causes a computer to perform the method described in claim 13.

Images