Image processing system, image processing method, and image processing program

Abstract

To provide a video processing system that is both versatile and reliable.SOLUTION: A processor 201 for a video processing system 20 is designed to process video frames Fm acquired from an infrastructure camera 10 through a network NW. The processor 201 for the video processing system 20 is configured to execute acquiring the video frames Fm in chronological order, identifying a video frame Fm for which an acquisition interval Tr has increased outside a delay tolerance range Ra as a delayed frame Fd that is delayed even more than the video frame Fm acquired on a regular interval Ti, adding delay information Id on the delayed frame Fd, and outputting time-series video data Dm of the video frames Fm that includes the delayed frame Fd to which the delay information Id has been added.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a video processing technique for processing video frames acquired from a camera over a network. [Background technology]

[0002] Patent Document 1 discloses a video processing system that processes imaging data acquired by a camera along with the imaging time. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6443145 Summary of the Invention [Problem to be solved by the invention]

[0004] Generally, when capturing image data from a camera via a network, there is a risk of delays occurring between the time the camera transmits the captured image data and the time the image data is captured by the image processing system, due to, for example, disturbances on the network. In this regard, if the camera does not have a timestamp function, such as that disclosed in Patent Document 1, which provides the capture time along with the captured image data, it is not possible to recognize the delay status of the captured image data, which could result in a decrease in system reliability. Conversely, the image processing system of Patent Document 1 assumes the use of a camera with a timestamp function to ensure reliability, which could result in a lack of versatility.

[0005] An object of the present disclosure is to provide a video processing system that is both versatile and reliable. Another object of the present disclosure is to provide a video processing method that is both versatile and reliable. Another object of the present disclosure is to provide a video processing program that is both versatile and reliable. [Means for solving the problem]

[0006] The technical means of the present disclosure for solving the problems will be described below. Note that the claims and the reference characters in parentheses in this section indicate the correspondence with the specific means described in the embodiments described later in detail, and do not limit the technical scope of the present disclosure.

[0007] A first aspect of the present disclosure is A processor (201), From camera (10, 10A) Through the network (NW, NWA) Take The obtained video frame (Fm) is 、 A video processing system (20, 20A) for processing The processor acquiring video frames in chronological order; Identifying a video frame whose acquisition interval (Tr) has increased beyond the delay tolerance range (Ra) as a delayed frame (Fd) delayed from a video frame acquired at a regular interval (Ti); adding delay information (Id) to the delayed frame; outputting time-series video data (Dm) of video frames including delay frames to which delay information has been added; is configured to execute

[0008] A second aspect of the present disclosure is From camera (10, 10A) Through the network (NW, NWA) Take Obtained video frame (Fm) , processing A video processing method executed by a processor (201) to acquiring video frames in chronological order; Identifying a video frame whose acquisition interval (Tr) has increased beyond the delay tolerance range (Ra) as a delayed frame (Fd) delayed from a video frame acquired at a regular interval (Ti); adding delay information (Id) to the delayed frame; outputting time-series video data (Dm) of video frames including delay frames to which delay information has been added; Includes.

[0009] A third aspect of the present disclosure is stored in a storage medium (202); From camera (10, 10A) Through the network (NW, NWA) Take Obtained video frame (Fm) , processing a video processing program including instructions to be executed by a processor (201) to The command is, acquiring video frames in chronological order; Identifying a video frame whose acquisition interval (Tr) has increased beyond a delay tolerance range (Ra) as a delayed frame (Fd) delayed from a video frame acquired at a regular interval (Ti); adding delay information (Id) to the delayed frame; outputting time-series video data (Dm) of video frames including delay frames to which delay information has been added; Includes.

[0010] According to these first to third aspects, video frames whose acquisition intervals have exceeded the allowable delay range are treated as delayed frames delayed relative to video frames acquired at regular intervals, and delay information is added to them. This allows time-series video data of video frames containing delayed frames with added delay information to be output, making it possible to recognize the delay status of the video frames based on the delay information added to the delayed frames, regardless of the camera specifications. This makes it possible to achieve both versatility and reliability. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic diagram showing the overall configuration of a video distribution system according to a first embodiment. [Figure 2] 1 is a block diagram showing a configuration of a video processing system according to a first embodiment. [Figure 3] 3 is a flowchart illustrating a video processing method according to the first embodiment. [Figure 4] FIG. 10 is a schematic diagram for explaining a normal state. [Figure 5] FIG. 10 is a schematic diagram for explaining a delayed frame and a semi-delayed frame. [Figure 6] 10 is a flowchart illustrating a video processing method according to a second embodiment. [Figure 7] FIG. 10 is a schematic diagram showing the overall configuration of a video distribution system according to a third embodiment. [Figure 8] FIG. 11 is a schematic diagram showing another overall configuration of a video distribution system according to the third embodiment. [Figure 9] FIG. 11 is a schematic diagram showing yet another overall configuration of the video distribution system according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, multiple embodiments of the present disclosure will be described with reference to the drawings. Note that corresponding components in each embodiment are designated by the same reference numerals, and redundant description may be omitted. Furthermore, when only a portion of the configuration is described in each embodiment, the configuration of another previously described embodiment may be applied to the remaining portions of the configuration. Furthermore, in addition to the combinations of configurations explicitly stated in the description of each embodiment, configurations of multiple embodiments may be partially combined together even if not explicitly stated, provided that there is no particular problem with the combination.

[0013] (First embodiment) Hereinafter, a first embodiment of the present disclosure will be described with reference to the drawings.

[0014] 1, a video distribution network system 1 according to the first embodiment processes a video frame Fm captured by an infrastructure camera 10, adds information to the processed video data Dm, and distributes the video data Dm to a distribution destination system 30. To achieve this, the video distribution network system 1 includes the infrastructure camera 10, a video processing system 20, the distribution destination system 30, and a communication network NW. Here, the communication network NW is an infrastructure network that combines wireless communication and wired communication.

[0015] Infrastructure cameras 10, which are fixed or movable cameras installed at fixed points around a road, capture images of road users, obstacles, and the like on the target road. The infrastructure cameras 10 transmit video frames Fm, which are captured images in time series, to the video processing system 20 via a communication network NW. The infrastructure cameras 10 transmit the video frames Fm in time series at regular transmission intervals. While multiple infrastructure cameras 10 are envisioned, the following explanation will typically focus on processing by a single infrastructure camera 10.

[0016] The video processing system 20 is constructed by at least one network server, such as a cloud server or an edge server. The video processing system 20 receives video frames Fm from the infrastructure camera 10 via a communication network NW between the infrastructure camera 10 and the video processing system 20, and processes the video frames Fm obtained by decoding. The video processing system 20 outputs video data Dm obtained by processing the video frames Fm to the destination system 30 via the communication network NW. The video processing system 20 detects a delay between the transmission of the video frames Fm by the infrastructure camera 10 and the acquisition by the system 20 itself, as a delay occurring in the communication network NW due to, for example, communication congestion. Therefore, the video processing system 20 adds information indicating the delay detection result to the video data Dm and provides the video data Dm to the destination system 30.

[0017] Here, the delivery destination system 30 is managed by a recipient user of the output destination who requires time-series video data Dm of video frames Fm output from the video processing system 20. The delivery destination system 30 uses the video data Dm delivered by the output from the video processing system 20 according to the receiving purpose of each recipient user. The delivery destination system 30 may be constructed by an ECU (Electronic Control Unit) of a vehicle that is a road user, such as an autonomous vehicle, that uses the video data Dm to control the vehicle. The delivery destination system 30 may be constructed by a network server of a service provider that is a user that uses the video data Dm for services related to transportation, for example.

[0018] The dedicated computer constituting the video processing system 20 that provides information to such a distribution destination system 30 has at least one memory 202 and one processor 201. The memory 202 is at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium, that non-temporarily stores computer-readable programs, data, etc. The processor 201 includes at least one type of core, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a RISC (Reduced Instruction Set Computer)-CPU.

[0019] The processor 201 executes a plurality of instructions included in the video processing program stored in the memory 202. As a result, the video processing system 20 constructs a plurality of functional blocks for processing the video frame Fm. In this way, in the video processing system 20, the video processing program stored in the memory 202 causes the processor 201 to execute a plurality of instructions to detect a delay in the video frame, thereby constructing a plurality of functional blocks. As shown in FIG. 2, the functional blocks constructed by the video processing system 20 include a video frame acquisition block 200, a delay determination block 210, and a video data output block 220.

[0020] The flow of the video processing method (hereinafter referred to as the video processing flow) for detecting a delay in a video frame Fm through the cooperation of the video frame acquisition block 200, delay determination block 210, and video data output block 220 will be described below with reference to Figure 3. This flow starts when the video processing system is started and ends when it is stopped. Note that in this flow, "S" refers to each of the multiple steps executed by multiple commands included in the video processing program.

[0021] In S100 of the video processing flow, the delay determination block 210 initializes the remaining number of delay frames Nd (Nd=0). The remaining number of delay frames Nd is the number of delay frames Fd that have not received delay information addition in S104, out of the number of delay frames Fd calculated in S103 (described later).

[0022] In S101, the video frame acquisition block 200 receives video frames Fm transmitted in chronological order from the infrastructure camera 10, as shown in Fig. 4. Here, the video frame acquisition block 200 periodically acquires the video frames Fm at regular intervals Ti, which are transmission periods that are steadily set in the infrastructure camera 10.

[0023] In S102 shown in FIG. 3, the delay determination block 210 determines whether the interval Tr for acquiring video frames Fm (hereinafter referred to as the acquisition interval) has exceeded the allowable delay range Ra and whether the remaining number of delayed frames Nd is equal to or greater than 1, as shown in FIG. 5, in order to identify a delayed frame Fd that is delayed relative to a video frame Fm acquired at a regular interval Ti. Here, the allowable delay range Ra may be any range that exceeds the regular interval Ti. In particular, in this embodiment, the allowable delay range Ra is set to twice the regular interval Ti (2Ti). Note that, in the first video processing flow after startup of the video processing system 20, there is no previously acquired video frame Fm that serves as the basis for the acquisition interval Tr. Therefore, in S102 in the first cycle, it is determined that the acquisition interval Tr has not exceeded the allowable delay range Ra.

[0024] 3, if it is determined that the acquisition interval Tr of the video frame Fm has increased outside the allowable delay range Ra (see FIG. 5) and / or the remaining number of delayed frames Nd is 1 or more, the delay determination block 210 in S103 identifies, as delayed frames Fd, video frames Fm with a frame number N corresponding to the ratio Td / Ti of the regular interval Ti to the delayed acquisition interval Td, which is the acquisition interval Tr outside the allowable delay range Ra. Specifically, the delay determination block 210 calculates the frame number N of the delayed frames Fd corresponding to the ratio Td / Ti according to the following Equation 1.

[0025]

number

[0026] As shown in FIG. 3, in S104 following S103, the delay determination block 210 adds delay information Id, such as a flag, indicating the delay state of communication to the delay frame Fd.

[0027] In S105 following S104, the delay determination block 210 sets the remaining number of delay frames Nd to N-1 (Nd=N-1). After S105 is executed, the video processing flow proceeds to S109.

[0028] If it is determined in S102 above that the acquisition interval Tr of the video frames Fm has not increased outside the allowable delay range Ra and the remaining number of delayed frames Nd is not 1 or greater, the video processing flow proceeds to S106. In S106, the delay determination block 210 determines whether the acquisition interval Tr of the video frames Fm has increased from within the return determination range Rr to outside the return determination range Rr. Here, the return determination range Rr may be any range that is less than the regular interval Ti. In particular, in this embodiment, the return determination range Rr is set to 1 / 2 the regular interval Ti (Ti / 2).

[0029] If it is determined in S106 that the acquisition interval Tr has increased beyond the restoration determination range Rr, the video processing flow proceeds to S107. In this case, the video frame Fm is a normal frame whose acquisition interval Tr is within the allowable delay range Ra but outside the restoration determination range Rr. In S107, the delay determination block 210 adds normal information In, such as a flag, to the normal video frame Fm, indicating a normal state in which no delay occurs. As a result, the delay determination block 210 ceases identifying the semi-delayed frame Fs, which will be described later, in response to the acquisition interval Tr of the video frame Fm increasing from within the restoration determination range Rr to outside the restoration determination range Rr within the allowable delay range Ra. After S107 is executed, the video processing flow proceeds to S109.

[0030] On the other hand, even if it is determined in S106 that the acquisition interval Tr has not increased beyond the recovery determination range Rr, the currently acquired video frame Fm is not a delayed frame Fd. However, in this case, the currently acquired video frame Fm is considered to be a semi-delayed frame Fs, acquired later than the normal acquisition time, as shown in FIG. 5. Therefore, the delay determination block 210 identifies the delayed frames Fd with the number N of frames corresponding to the ratio Td / Ti, and then identifies the video frame Fm whose acquisition interval Tr has decreased within the allowable delay range Ra but has fallen within the recovery determination range Rr as a semi-delayed frame Fs. If it is determined in S106 that the acquisition interval Tr has not increased beyond the recovery determination range Rr, the video processing flow proceeds to S108. In S108, the delay determination block 210 adds semi-delay information Is, such as a flag, to the semi-delayed frame Fs, which indicates the semi-delay state of communication, separate from the delay information Id. After S108 is executed, the video processing flow proceeds to S109.

[0031] In S109, the video data output block 220 outputs time-series video data Dm of video frames Fm to the destination system 30 as data including video frames Fm to which delay information Id, normal information In, or semi-delay information Is has been added. The video data Dm output at this time may be the video frames Fm themselves to which any of the information Id, In, or Is has been added. The video data Dm output at this time may also be video frames Fm to which any of the information Id, In, or Is has been added, and to which other information, such as recognition information obtained through recognition processing, has also been added by the video data output block 220. After S109 is executed, the video processing flow returns to S101.

[0032] By repeating steps S101 to S109 of the video processing flow described above, video data Dm obtained by processing video frames Fm received in chronological order from the infrastructure camera 10 is output in that chronological order.

[0033] (Action and effect) The effects of the first embodiment described above will be explained below.

[0034] In the first embodiment, when the acquisition interval Tr has increased beyond the allowable delay range Ra, the video frame Fm is treated as a delayed frame Fd delayed from the video frame Fm acquired at the normal interval Ti, and delay information Id is added to the video frame Fm. This allows the time-series video data Dm of the video frame Fm, which includes the delayed frame Fd with the delay information Id added, to be output, making it possible to recognize the delay status of the video frame Fm based on the delay information Id added to the delayed frame Fd, regardless of the specifications of the infrastructure camera 10. This makes it possible to achieve both versatility and reliability.

[0035] In the first embodiment, video frames Fm of a frame number N corresponding to the ratio between a normally set normal interval Ti and a delayed acquisition interval Td, which is an acquisition interval Tr outside the allowable delay range Ra, are identified as delayed frames Fd. This allows all video frames Fm of a frame number N acquired within the delayed acquisition interval Td to be identified as delayed frames Fd, and delay information Id can be added to them. Therefore, it is possible to improve reliability by determining the number of delayed frames Fd while ensuring versatility.

[0036] In the first embodiment, after identifying a delayed frame Fd with a frame number N corresponding to the ratio between the normal interval Ti and the delayed acquisition interval Td, a video frame Fm whose acquisition interval Tr has decreased to within the return determination range Rr within the allowable delay range Ra is identified as a semi-delayed frame Fs. Then, semi-delay information Is, which is distinct from the delay information Id, is added to the semi-delayed frame Fs. This makes it possible to identify a semi-delayed frame Fs that has been secondarily delayed due to the influence of the delayed frame Fd, and add the semi-delay information Is to it. Therefore, it is possible to improve reliability by taking into account the secondary delay while maintaining versatility.

[0037] In the first embodiment, as the acquisition interval Tr increases from within the restoration determination range Rr to outside the restoration determination range Rr within the allowable delay range Ra, the identification of delayed frames Fd is stopped. This allows the identification of delayed frames Fd to continue until the delay of the video frame Fm is resolved, making it possible to accurately add delay information Id only to delayed frames Fd affected by the delay. This makes it possible to improve reliability while maintaining versatility.

[0038] The restoration determination range Rr in the first embodiment is a range less than the normal interval Ti. This allows the semi-delayed frame Fs, which is delayed secondarily due to the influence of the delay frame Fd, to be accurately identified and the semi-delay information Is to be added. This makes it possible to ensure high reliability by taking into account the secondary delay.

[0039] The video processing system of the first embodiment delivers to a destination user time-series video data Dm of video frames Fm, which includes delay frames Fd with added delay information Id. This allows the destination user to use the video data Dm while taking into account the delay status of the video frames Fm based on the delay information Id of the delay frames Fd. This makes it possible to provide both versatility and reliability to the destination user.

[0040] Second Embodiment The second embodiment is a modification of the first embodiment.

[0041] In the video processing flow according to the second embodiment, as shown in Fig. 6, steps S100 to S105 similar to those in the first embodiment are executed. If it is determined in S102 of the video processing flow according to the second embodiment that the interval Tr for acquiring video frames has not increased beyond the allowable delay range Ra and that the remaining number of delay frames Nd is not 1 or greater, the video processing flow proceeds to S110. In S110, the delay determination block 210 sets the current time as the regular acquisition time To of the video frame Fm. After S110 is executed, the video processing flow proceeds to S106. Note that steps S106 to S108 are similar to those in the first embodiment.

[0042] Meanwhile, in S111 following S105, the delay determination block 210 estimates the normal acquisition time To as a time correlated with the normal interval Ti and the number of frames N of the delayed frame Fd. Specifically, the delay determination block 210 estimates the normal acquisition time To of the video frame Fm as the time calculated by adding the value (Ti×N) obtained by multiplying the normal interval Ti by the number of frames N of the delayed frame Fd to the acquisition time of the video frame Fm acquired immediately before the delayed frame Fd was identified.

[0043] After S111 is executed, the video processing flow proceeds to S112. Also, after S107 is executed, the video processing flow proceeds to S112. Furthermore, after S108 is executed, the video processing flow proceeds to S112. In S112, the delay determination block 210 adds the regular acquisition time To to the video frame Fm.

[0044] In S113 following S112, the delay determination block 210 estimates the transmission time of the video frame Fm from the infrastructure camera 10. Specifically, the delay determination block 210 estimates the transmission time of the video frame Fm to be a time that is earlier than the normal acquisition time To by the steady delay time Trd. Here, the steady delay time Trd is defined as the time that steadily occurs from the transmission of the video frame Fm by the infrastructure camera 10 to the acquisition of the video frame Fm by the video processing system 20 in a normal state where there is no influence of, for example, external disturbances in the communication network NW. Furthermore, in S113, the delay determination block 210 adds the transmission time estimated as described above to the video frame Fm. Note that S109 following S113 is the same as in the first embodiment.

[0045] Through the above video processing flow, the video processing system 20 adds the regular acquisition time To to the video frame Fm together with the delay information Id, semi-delay information Is, or normal information. Furthermore, the video processing system 20 adds a time that is earlier than the regular acquisition time To by the steady delay time Trd as the transmission time of the delayed frame Fd to the video frame Fm together with the delay information Id, semi-delay information Is, or normal information.

[0046] (Action and effect) The effects of the second embodiment described above will be explained below.

[0047] In the second embodiment, a regular acquisition time To, which correlates with the regular interval Ti and the frame number N of the delayed frames Fd, is added to the delayed frames Fd together with the delay information Id. This makes it possible to accurately recognize the delay of the delayed frames Fd based not only on the delay information Id but also on the regular acquisition time To determined by the video processing system 20. This makes it possible to improve reliability while ensuring versatility.

[0048] In the video processing system 20 of the second embodiment, a time preceding the normal acquisition time To by a steady delay time Trd that steadily occurs between the transmission of the video frame Fm by the infrastructure camera 10 and the acquisition by the system 20 itself is added to the delayed frame Fd as the transmission time of the delayed frame Fd, together with the delay information Id and the normal acquisition time To. This makes it possible to accurately recognize the delay of the delayed frame Fd based not only on the delay information Id but also on the normal acquisition time To by the video processing system 20 and the transmission time from the infrastructure camera 10. This makes it possible to ensure particularly high reliability.

[0049] (Third embodiment) The third embodiment is a modification of the first embodiment.

[0050] 7 to 9, a video transmission network system 1A according to the third embodiment processes a video frame Fm captured by an on-board camera 10A, adds information to the video data Dm, and transmits the processed video data Dm to a destination system 30A. To achieve this, the video transmission network system 1A includes an on-board camera 10A, a video processing system 20A, a destination system 30A, an on-board network NWA, and / or a communication network NW. Here, the on-board network NWA is, for example, a local area network (LAN) or the like established in a host vehicle 40A, and is connected to multiple electronic devices, such as sensors and / or actuators, other than the on-board camera 10A in the host vehicle 40A.

[0051] In the video transmission network system 1A, the in-vehicle camera 10A and the video processing system 20A are mounted on a host vehicle 40A and are interconnected via an in-vehicle network NWA. As shown in Figures 7 and 9, the destination system 30A in the video transmission network system 1A may be mounted on the host vehicle 40A (i.e., the host vehicle) and connected to the video processing system 20A via the in-vehicle network NWA. As shown in Figures 8 and 9, the destination system 30A in the video transmission network system 1A may be mounted on another vehicle other than the host vehicle 40A or constructed by a network server other than the host vehicle 40A and connected to the video processing system 20A via a communication network NW.

[0052] As shown in FIGS. 7 to 9, the vehicle-mounted camera 10A of the third embodiment, which corresponds to the infrastructure camera 10 of the first embodiment, is a fixed or movable camera relative to the host vehicle 40A. The vehicle-mounted camera 10A captures images of road users, obstacles, and the like on the road outside the host vehicle 40A. The vehicle-mounted camera 10A transmits video frames Fm, which are captured time-series images, to the video processing system 20A via the vehicle-mounted network NWA. At this time, the vehicle-mounted camera 10A transmits the video frames Fm in time-series at regular transmission intervals. While a single or multiple cameras may be used in the host vehicle 40A as the vehicle-mounted camera 10A, the following description will typically focus on processing using a single vehicle-mounted camera 10A.

[0053] The image processing system 20A of the third embodiment, which corresponds to the image processing system 20 of the first embodiment, is implemented by an ECU that processes image data Dm by controlling an in-vehicle camera 10A in a host vehicle 40A, such as an autonomous vehicle. The image processing system 20A receives image frames Fm from the in-vehicle camera 10A via an in-vehicle network NWA between the in-vehicle camera 10A and the host vehicle 40A, and processes the image frames Fm obtained by decoding. The image processing system 20A outputs the image data Dm obtained by processing the image frames Fm to a destination system 30A via the in-vehicle network NWA and / or the communication network NW. The image processing system 20A detects a delay between the transmission of the image frame Fm by the in-vehicle camera 10A and the acquisition of the image frame Fm by the image processing system 20A, which is a delay occurring in the in-vehicle network NWA due to, for example, communication congestion. The image processing system 20A then adds information indicating the delay detection result to the image data Dm and provides the resulting data to the destination system 30A.

[0054] The destination system 30A of the third embodiment, which corresponds to the distribution destination system 30 of the first embodiment, is managed by a destination user who requires time-series video data Dm of video frames Fm output from the video processing system 20A. The destination system 30A uses the video data Dm transmitted by the output from the video processing system 20A according to the receiving purpose of each recipient user. The destination system 30A may be constructed by an ECU that uses the video data Dm to control a host vehicle 40A or other vehicles, which are road users such as autonomous vehicles. The destination system 30A may be constructed by a network server of a servicer as a user that uses the video data Dm for services such as traffic-related services.

[0055] The dedicated computer constituting the video processing system 20A that provides information to the destination system 30A has at least one memory 202 and one processor 201 that conform to the first embodiment. Therefore, the multiple functional blocks configured by the video processing system 20A include a video frame acquisition block 200, a delay determination block 210, and a video data output block 220 that conform to the first embodiment of Fig. 2. However, in the video processing flow of the third embodiment, which is performed jointly by these blocks 200, 210, and 220, the infrastructure camera 10, the video processing system 20, and the destination system 30 in the flow described in the first embodiment are replaced with the vehicle-mounted camera 10A, the video processing system 20A, and the destination system 30A, respectively.

[0056] According to the third embodiment, it is possible to achieve the same effects as the first embodiment. Here, according to the third embodiment, time-series video data Dm of video frames Fm including delay frames Fd to which delay information Id has been added is output, so that the delay status of the video frames Fm can be recognized regardless of the specifications of the vehicle-mounted camera 10A, thereby achieving both versatility and reliability.

[0057] (Other embodiments) Although multiple embodiments have been described above, the present disclosure should not be construed as being limited to those embodiments, and can be applied to various embodiments and combinations within the scope that does not deviate from the gist of the present disclosure.

[0058] In a modified example, steps S111 to S113 may be omitted from the video processing flow described in the second embodiment, and the normal acquisition time To may not be added to the video frame Fm. In a modified example, the second embodiment may be applied to the third embodiment. However, the steady delay time Trd in the video processing flow of the second embodiment applied to the third embodiment may be defined as the time that steadily occurs from the transmission of the video frame Fm by the in-vehicle camera 10A to the acquisition of the video frame Fm by the video processing system 20A in a normal state where there is no influence of, for example, external disturbances in the in-vehicle network NWA.

[0059] (Disclosure of technical ideas) This specification discloses multiple technical ideas described in the following multiple clauses. Some clauses may be described in a multiple dependent form, with the subsequent clause alternatively referring to the preceding clause. Furthermore, some clauses may be described in a multiple dependent form, with the subsequent clause referring to another multiple dependent clause. These multiple dependent clauses define multiple technical ideas. Note that the reference symbols in parentheses in the disclosure section of the present ideas indicate the correspondence with the specific means described in the previously detailed embodiments, and do not limit the technical scope of the present disclosure.

[0060] (Technical thought 1) 1. A video processing system having a processor, the system processing video frames acquired from a camera via a network, The processor acquiring video frames in chronological order; Identifying a video frame whose acquisition interval has increased beyond an allowable delay range as a delayed frame delayed from a video frame acquired at a regular interval; adding delay information to the delay frame; outputting time-series video data of video frames including delay frames to which delay information has been added; A video processing system configured to perform the following:

[0061] (Technical thought 2) Identifying the delayed frame may include: A video processing system according to technical idea 1, which includes identifying as delayed frames a number of video frames corresponding to the ratio between a normal interval set to a steady state and a delayed acquisition interval, which is an acquisition interval outside the delay tolerance range.

[0062] (Technical Thought 3) Identifying the delayed frame may include: After identifying the number of delayed frames according to the ratio, video frames whose acquisition intervals have decreased to within a recovery determination range within the delay tolerance range are identified as semi-delayed frames; Adding delay information is adding semi-delay information to the semi-delayed frame, the semi-delay information being distinct from the delay information; To output video data, The video processing system according to Technical Idea 2 includes outputting video data containing a semi-delayed frame to which semi-delay information has been added.

[0063] (Technical Thought 4) Identifying the delayed frame may include: The video processing system according to Technical Idea 3 includes ceasing to identify the semi-delayed frame in response to the acquisition interval increasing from within the delay tolerance range to outside the return determination range.

[0064] (Technical Thought 5) The image processing system according to Technical Idea 3 or 4, wherein the recovery determination range is a range less than the regular interval.

[0065] (Technical Thought 6) Adding delay information is The video processing system according to any one of Technical Ideas 2 to 5, further comprising adding a regular acquisition time, which is correlated with the regular interval and the number of frames, to the delayed frame together with the delay information.

[0066] (Technical Thought 7) Adding delay information is A video processing system according to technical idea 6, which includes adding to the delayed frame, together with delay information and the normal acquisition time, a time that precedes the normal acquisition time, which is a steady delay time that occurs between the transmission of a video frame by a camera and its acquisition by the system, as the transmission time of the delayed frame.

[0067] (Technical Thought 8) Adding delay information is A video processing system described in any one of technical ideas 2 to 5, which includes adding to the delayed frame, together with delay information, the transmission time of the delayed frame, which is a time that goes back from the normal acquisition time that correlates with the normal interval and number of frames, by the steady delay time that steadily occurs between the transmission of the video frame by the camera and its acquisition by the system.

[0068] (Technical Thought 9) Outputting a video frame is The video processing system according to any one of Technical Ideas 1 to 8, including distributing or transmitting video data to a destination user.

[0069] (Technical Thought 10) 10. A video processing system according to any one of technical ideas 1 to 9, which processes video frames acquired from an infrastructure camera as a camera through an infrastructure network as a network.

[0070] (Technical Thought 11) 10. A video processing system according to any one of technical ideas 1 to 9, which processes video frames acquired from an in-vehicle camera as a camera via an in-vehicle network as a network.

[0071] The above-described technical ideas 1 to 11 may be realized in the form of a method or a program. [Explanation of symbols]

[0072] 1: video distribution network system, 1A: video transmission network system, 10: infrastructure camera, 10A: vehicle-mounted camera, 20, 20A: video processing system, 201: processor, 202: storage medium, Dm: video data, Fd: delay frame, Fm: video frame, Fs: semi-delay frame, Id: delay information, Is: semi-delay information, NW: communication network, NWA: vehicle-mounted network, Ra: delay tolerance range, Rr: recovery determination range, Td: delay acquisition interval, Ti: regular interval, To: regular acquisition time, Tr: acquisition interval, Trd: steady-state delay time

Claims

1. An image processing system (20, 20A) having a processor (201) for processing an image frame (Fm) acquired from a camera (10, 10A) through a network (NW, NWA), The processor: acquiring the video frames in chronological order; Identifying the video frame whose acquisition interval (Tr) has increased beyond the delay tolerance range (Ra) as a delayed frame (Fd) delayed from the video frame acquired at a regular interval (Ti); adding delay information (Id) to the delayed frame; outputting time-series video data (Dm) of the video frames including the delayed frames to which the delay information has been added; A video processing system configured to perform the following:

2. Identifying the delayed frame includes: The video processing system according to claim 1, further comprising: identifying as the delayed frames a number of video frames corresponding to a ratio between the normal interval that is constantly set and a delayed acquisition interval (Td) that is the acquisition interval outside the delay tolerance range.

3. Identifying the delayed frame includes: After identifying the delayed frames of the number of frames, the video frames whose acquisition intervals have decreased to within a restoration determination range (Rr) within the delay tolerance range are identified as semi-delayed frames (Fs); Adding the delay information adding sub-delay information (Is) distinct from the delay information to the sub-delay frame; outputting the video data The video processing system according to claim 2 , further comprising: outputting the video data containing the semi-delayed frame to which the semi-delay information has been added.

4. Identifying the delayed frame includes: The video processing system according to claim 3 , further comprising: discontinuing the identification of the semi-delayed frame in response to the acquisition interval increasing from within the allowable delay range to outside the return determination range.

5. The image processing system according to claim 3 , wherein the recovery determination range is a range less than the regular interval.

6. Adding the delay information The video processing system according to claim 2 , further comprising adding a regular acquisition time (To) correlated with the regular interval and the number of frames to the delayed frame together with the delay information.

7. Adding the delay information The video processing system of claim 6, further comprising: adding to the delayed frame, together with the delay information and the regular acquisition time, a time that is earlier than the regular acquisition time by a steady delay time (Trd) that steadily occurs between the transmission of the video frame by the camera and the acquisition by the system, as the transmission time of the delayed frame.

8. Adding the delay information The video processing system of claim 2, further comprising: adding to the delayed frame, together with the delay information, a time that is earlier than a normal acquisition time (To) that correlates with the normal interval and the number of frames, by a steady delay time (Trd) that steadily occurs between the transmission of the video frame by the camera and its acquisition by the system, as the transmission time of the delayed frame.

9. outputting the video frame The video processing system according to claim 1 or 2, further comprising distributing or transmitting the video data to a destination user.

10. An image processing system as described in claim 1 or 2, which processes the image frames obtained from an infrastructure camera (10) as the camera through an infrastructure network (NW) as the network.

11. An image processing system as described in claim 1 or 2, which processes the image frames acquired from an in-vehicle camera (10A) as the camera through an in-vehicle network (NWA) as the network.

12. A video processing method executed by a processor (201) to process video frames (Fm) acquired from a camera (10, 10A) through a network (NW, NWA), comprising: acquiring the video frames in chronological order; Identifying the video frame whose acquisition interval (Tr) has increased beyond the delay tolerance range (Ra) as a delayed frame (Fd) delayed from the video frame acquired at a regular interval (Ti); adding delay information (Id) to the delayed frame; outputting time-series video data (Dm) of the video frames including the delayed frames to which the delay information has been added; A video processing method including:

13. A video processing program stored in a storage medium (202) and including instructions to be executed by a processor (201) to process video frames (Fm) acquired from a camera (10, 10A) through a network (NW, NWA), The instruction: acquiring the video frames in chronological order; Identifying the video frame whose acquisition interval (Tr) has increased beyond the delay tolerance range (Ra) as a delayed frame (Fd) delayed from the video frame acquired at a regular interval (Ti); adding delay information (Id) to the delayed frame; outputting time-series video data (Dm) of the video frames including the delayed frames to which the delay information has been added; Image processing program including.

Images