Video processing system, video processing method, and video processing program
By acquiring and marking delayed image frames in a time series in the image processing system, the problem of delay recognition difficulty when acquiring camera data on the network is solved, and an image processing method that takes into account both universality and reliability is realized.
Patent Information
- Application Number
- CN202380041735.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-04-24
- Filing Date
- 2023-04-27
- Publication Date
- 2024-12-31
AI Technical Summary
When the prior art acquires camera data through the network, the lack of timestamp function leads to difficulty in identifying delays, affects system reliability, and lacks universality of cameras using timestamp functions.
The processor acquires image frames in time series order, determines that the frames whose acquisition interval increases to outside the delay allowable range are used as delay frames, and adds delay information to output image frames containing delay information to realize delay status recognition.
Taking into account both universality and reliability, it can identify the delay status of the image frame based on additional delay information to ensure the stability and applicability of the system.
Smart Images
Figure CN119234425A_ABST
Abstract
Description
[0001] Cross-references of related applications
[0002] This application is based on patent application No. 2022-93934 filed in Japan on June 9, 2022 and patent application No. 2023-71013 filed in Japan on April 24, 2023, and the contents of the underlying applications are cited by reference in their entirety. Technical Field
[0003] The invention relates to an image processing technology for processing image frames acquired from a camera through a network. Background Art
[0004] Patent Document 1 discloses an image processing system that processes image data acquired by a camera together with the image capturing time.
[0005] Prior art literature
[0006] Patent Literature
[0007] Patent Document 1: Japanese Patent No. 6443145
[0008] Generally, when acquiring video data from a camera via a network, a delay may occur from the time the camera sends the video data to the time the video processing system acquires the video data due to interference in the network, for example. In this regard, in the case of a specification without a timestamp function such as that in Patent Document 1 in which a camera provides the time of capture together with the video data, the delay of the video data cannot be identified, and the reliability of the system may be reduced. Conversely, the image processing system of Patent Document 1 is based on the use of a camera with a timestamp function in order to ensure reliability, and may lack versatility. Summary of the invention
[0009] The present invention aims to provide an image processing system that takes into account both versatility and reliability. Another object of the present invention is to provide an image processing method that takes into account both versatility and reliability. Another object of the present invention is to provide an image processing program that takes into account both versatility and reliability.
[0010] Hereinafter, technical means of the present invention for solving the problems will be described.
[0011] A first aspect of the present invention is an image processing system, which includes a processor and processes image frames acquired from a camera via a network, wherein:
[0012] The processor is configured to perform the following processing:
[0013] Acquire image frames in time series order;
[0014] determining an image frame whose acquisition interval increases to outside a delay tolerance range as a delayed frame delayed compared to an image frame acquired at a regular interval;
[0015] adding delay information to the delay frame; and
[0016] Time-series image data including image frames to which delay information is added is output.
[0017] A second embodiment of the present invention is an image processing method, which is executed by a processor to detect a delay in an image frame obtained from a camera via a network, and includes:
[0018] Acquire image frames in time series order;
[0019] determining an image frame whose acquisition interval increases to outside a delay tolerance range as a delayed frame delayed compared to an image frame acquired at a regular interval;
[0020] adding delay information to the delay frame; and
[0021] Time-series image data including image frames to which delay information is added is output.
[0022] A third aspect of the present invention is an image processing program stored in a storage medium, comprising a command for a processor to execute in order to detect a delay in an image frame acquired from a camera via a network, wherein:
[0023] The command contains:
[0024] Acquire image frames in time series order;
[0025] determining an image frame whose acquisition interval increases to outside a delay tolerance range as a delayed frame delayed compared to an image frame acquired at a regular interval;
[0026] adding delay information to the delay frame; and
[0027] Time-series image data including image frames to which delay information is added is output.
[0028] According to the first to third methods, the image frames whose acquisition interval increases to outside the delay tolerance range are added with delay information as delayed frames compared to the image frames acquired at the regular interval. Thus, by outputting time-series image data of image frames including delayed frames to which delay information is added, the delay status of the image frames can be identified based on the delay information added to the delayed frames regardless of the specifications of the camera. Therefore, both versatility and reliability can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1It is a schematic diagram showing the overall configuration of the image distribution system according to the first embodiment.
[0030] Figure 2 This is a block diagram showing the configuration of the image processing system according to the first embodiment.
[0031] Figure 3 This is a flowchart showing the image processing method according to the first embodiment.
[0032] Figure 4 This is a schematic diagram for explaining the normal state.
[0033] Figure 5 This is a schematic diagram for explaining a delayed frame and a quasi-delayed frame.
[0034] Figure 6 This is a flowchart showing the image processing method according to the second embodiment.
[0035] Figure 7 It is a schematic diagram showing the overall configuration of a video distribution system according to a third embodiment.
[0036] Figure 8 It is a schematic diagram showing another overall structure of the image distribution system according to the third embodiment.
[0037] Fig. 9 It is a schematic diagram showing another overall structure of the image distribution system according to the third embodiment. DETAILED DESCRIPTION
[0038] Hereinafter, multiple embodiments of the present invention will be described based on the accompanying drawings. In addition, in each embodiment, repeated descriptions may be omitted by marking the corresponding constituent elements with the same symbols. In addition, in each embodiment, when only a part of the structure is described, the structure of other embodiments previously described can be applied to the other parts of the structure. In addition, in addition to the combination of structures explicitly stated in the description of each embodiment, as long as the combination does not cause special obstacles, the structures of multiple embodiments can be partially combined with each other even if not explicitly stated.
[0039] (First Embodiment)
[0040] Hereinafter, a first embodiment of the present invention will be described based on the drawings.
[0041] like Figure 1As shown, the image distribution network system 1 according to the first embodiment processes the image frame Fm captured by the infrastructure camera 10 and distributes the image data Dm to which information is added to the distribution target system 30. Therefore, the image distribution network system 1 is configured to include the infrastructure camera 10 and the image processing system 20, the distribution target system 30, and the communication network NW. Here, the communication network NW is an infrastructure network formed by integrating wireless communication and wired communication.
[0042] The infrastructure camera 10, which is a fixed or movable camera installed at a fixed point around the road, captures objects such as road users and obstacles on the road as a subject. The infrastructure camera 10 transmits the captured image frames Fm as time-series images to the image processing system 20 via the communication network NW. At this time, the infrastructure camera 10 performs the transmission of the image frames Fm in a time series at a constant transmission cycle. As such an infrastructure camera 10, it is assumed that there are a plurality of cameras, but in the following description, the processing performed by one infrastructure camera 10 is representatively listed.
[0043] The image processing system 20 is constructed by at least one network server such as a cloud server or an edge server. The image processing system 20 receives the image frame Fm from the infrastructure camera 10 through the communication network NW between the infrastructure camera 10 and the image frame Fm obtained by decryption. The image processing system 20 outputs the image data Dm obtained by processing the image frame Fm to the distribution target system 30 through the communication network NW. Such an image processing system 20 detects the delay from the transmission of the image frame Fm by the infrastructure camera 10 to the acquisition by the image processing system 20 as a delay caused in the communication network NW due to, for example, communication congestion. Therefore, the image processing system 20 adds information indicating the delay detection result to the image data Dm and provides it to the distribution target system 30.
[0044] Here, the distribution target system 30 is managed by a receiving user of the output target, and the receiving user needs the time-series image data Dm of the image frame Fm output from the image processing system 20. The distribution target system 30 uses the image data Dm distributed through the output from the image processing system 20 according to the receiving purpose of each receiving user. The distribution target system 30 can be constructed, for example, by an ECU (Electronic Control Unit) of a vehicle that uses the image data Dm for control of a vehicle such as an autonomous driving vehicle as a road user. The distribution target system 30 can also be constructed, for example, by a network server of a service provider that uses the image data Dm for services such as traffic-related services as a user.
[0045] The dedicated computer constituting the image processing system 20 that provides information to such a distribution target system 30 has at least one memory 202 and a processor 201. The memory 202 is a non-transitory tangible storage medium such as a semiconductor memory, a magnetic medium, and an optical medium that non-temporarily stores programs and data that can be read by the computer. The processor 201 includes at least one type of CPU (Central Processing Unit), GPU (Graphics Processing Unit), and RISC (Reduced Instruction Set Computer)-CPU as a core.
[0046] The processor 201 executes a plurality of commands included in the image processing program stored in the memory 202. Thus, the image processing system 20 constructs a plurality of functional blocks for processing the image frame Fm. In such an image processing system 20, in order to detect the delay in the image frame, the image processing program stored in the memory 202 causes the processor 201 to execute a plurality of commands to construct a plurality of functional blocks. Figure 2 As shown, the plurality of functional blocks constructed by the image processing system 20 include an image frame acquisition block 200 , a delay determination block 210 , and an image data output block 220 .
[0047] Below, according to Figure 3 , the process of the image processing method for detecting the delay in the image frame Fm through these image frame acquisition blocks 200, delay determination blocks 210, and image data output blocks 220 (hereinafter referred to as the image processing process) is described. This process starts when the image processing system is started and ends when it is stopped. In addition, in this process, "S" respectively represents a plurality of steps executed by a plurality of commands included in the image processing program.
[0048] In S100 of the image processing flow, the delay determination block 210 initializes the number of remaining delay frames Nd (Nd=0). The number of remaining delay frames Nd is the number of delay frames Fd to which delay information is not added in S104 among the number of delay frames Fd calculated in S103 described later.
[0049] In S101, the image frame acquisition block 200 is as follows Figure 4 As shown, the image frames Fm transmitted in time series from the infrastructure camera 10 are received. Here, the image frame acquisition block 200 periodically acquires the image frames Fm using the transmission cycle set to be constant in the infrastructure camera 10 as the regular interval Ti.
[0050] exist Figure 3 In S102 shown in FIG. 1 , in order to determine the delayed frame Fd compared to the image frame Fm acquired at the regular interval Ti, the delay determination block 210 performs the following operation: Figure 5 As shown in the figure, it is determined whether the interval (hereinafter referred to as the acquisition interval) Tr of the acquired image frame Fm increases to outside the delay allowable range Ra, and whether the number of delayed frames remaining Nd is greater than 1. Here, the delay allowable range Ra is a range that exceeds the normal interval Ti. In particular, in the present embodiment, the delay allowable range Ra is set to a time (2Ti) twice the normal interval Ti. In addition, in the first image processing flow after the image processing system 20 is started, there is no image frame Fm acquired the previous time as a basis for the acquisition interval Tr. Therefore, in S102 in the first cycle, it is determined that the acquisition interval Tr has not increased to outside the delay allowable range Ra.
[0051] pass Figure 3 In S102 shown in FIG. 104 , when it is determined that the acquisition interval Tr of the image frame Fm increases to outside the delay tolerance range Ra (see FIG. 105 ), Figure 5 ) and / or the number of remaining delayed frames Nd is greater than 1, the delay determination block 210 determines the image frame Fm having the number of frames N corresponding to the ratio Td / Ti of the regular interval Ti to the delayed acquisition interval Td as the acquisition interval Tr outside the delay allowable range Ra as the delayed frame Fd. Specifically, the delay determination block 210 calculates the number of frames N of the delayed frame Fd corresponding to the ratio Td / Ti according to the following mathematical formula 1.
[0052] [Mathematical formula 1]
[0053] N=Nd+(Td / Ti)
[0054] based on Figure 5 The principle of determining the delayed frame Fd by calculating the number of frames N based on Mathematical Formula 1 in S103 is described in detail. Figure 5 In FIG, the acquisition time of the image frame Fm based on the regular interval Ti is indicated by a double-dashed line. Figure 5 In the example, the acquisition interval Tr of the image frame Fm1 is increased to 4Ti or more and less than 5Ti as the delayed acquisition interval Td. In this case, due to the delay of the image frame Fm1, the three image frames Fm following the image frame Fm1 are also acquired with a delay compared to the acquisition time based on the regular interval Ti. Therefore, the delay determination block 210 determines the three image frames Fm following the image frame Fm1 as delayed frames Fd based on the delayed acquisition interval Td of the delayed frame Fd of the delayed image frame Fm1.
[0055] like Figure 3As shown, in S104 following S103, the delay determination block 210 adds delay information Id such as a flag indicating the delay state of communication to the delay frame Fd.
[0056] In S105 following S104, the delay determination block 210 sets the number of remaining delayed frames Nd to N-1 (Nd=N-1). After S105 is executed, the image processing flow proceeds to S109.
[0057] When it is determined by the above-mentioned S102 that the acquisition interval Tr of the image frame Fm has not increased to outside the delay allowable range Ra and the number of delayed frames remaining Nd is not greater than 1, the image processing flow enters S106. In S106, the delay determination block 210 determines whether the acquisition interval Tr of the image frame Fm has increased from within the recovery determination range Rr to outside the recovery determination range Rr. Here, the recovery determination range Rr is a range smaller than the normal interval Ti. In particular, in the present embodiment, the recovery determination range Rr is set to a time of 1 / 2 of the normal interval Ti (Ti / 2).
[0058] When it is determined by S106 that the acquisition interval Tr increases to outside the recovery determination range Rr, the image processing flow enters S107. In this case, the image frame Fm is a normal frame whose acquisition interval Tr is within the delay allowable range Ra and outside the recovery determination range Rr. In S107, the delay determination block 210 adds normal information In such as a chess mark indicating a normal state in which no delay occurs to the normal image frame Fm. Thus, the delay determination block 210 terminates the determination of the quasi-delay frame Fs described later based on the fact that the acquisition interval Tr of the image frame Fm is within the delay allowable range Ra and increases from within the recovery determination range Rr to outside the recovery determination range Rr. After executing S107, the image processing flow enters S109.
[0059] On the other hand, in S106, even if it is determined that the acquisition interval Tr has not increased to outside the recovery determination range Rr, the image frame Fm acquired this time is not a delayed frame Fd. Therefore, in this case, it is considered that the image frame Fm acquired this time is Figure 5 As shown in the figure, the quasi-delayed frame Fs is acquired with a delay compared to the regular acquisition time. Therefore, after determining the delayed frame Fd corresponding to the number of frames N of the ratio Td / Ti, the delay determination block 210 determines the image frame Fm within the recovery determination range Rr within the delay allowable range Ra as the quasi-delayed frame Fs. In the case where it is determined by S106 that the acquisition interval Tr has not increased to outside the recovery determination range Rr, the image processing flow enters S108. In S108, the delay determination block 210 adds quasi-delay information Is, such as a chess mark, which indicates the quasi-delay state of the communication differently from the delay information Id, to the quasi-delayed frame Fs. After executing S108, the image processing flow enters S109.
[0060] In S109, the image data output block 220 outputs the time-series image data Dm of the image frame Fm toward the target system 30 as data including the image frame Fm to which the delay information Id, the normal information In, or the quasi-delay information Is is attached. The image data Dm output at this time may also be the image frame Fm itself to which any one of the information Id, In, Is is attached. The image data Dm output at this time may also be the image frame Fm to which other information such as identification information based on identification processing is attached by the image data output block 220 after any one of the information Id, In, Is is attached. After executing S109, the image processing flow returns to S101.
[0061] By repeating S101 to S109 of the above-described image processing flow, image data Dm obtained by processing image frames Fm received from the infrastructure camera 10 in time series is output in the time series order.
[0062] (Effect)
[0063] Hereinafter, the effects of the first embodiment described above will be described.
[0064] In the first embodiment, the image frame Fm whose acquisition interval Tr increases to outside the delay allowable range Ra is added with delay information Id as a delayed frame Fd delayed compared with the image frame Fm acquired at the regular interval Ti. Thus, by outputting the time-series image data Dm of the image frame Fm including the delayed frame Fd to which the delay information Id is added, the delay status of the image frame Fm can be identified based on the delay information Id added to the delayed frame Fd, regardless of the specifications of the infrastructure camera 10. Therefore, both versatility and reliability can be achieved.
[0065] In the first embodiment, the image frames Fm having the number N of frames corresponding to the ratio of the regular interval Ti set to be constant to the delayed acquisition interval Td as the acquisition interval Tr outside the delay allowable range Ra are determined as delayed frames Fd. Thus, all the image frames Fm having the number N of frames acquired within the delayed acquisition interval Td can be regarded as delayed frames Fd and delay information Id can be added. Therefore, versatility can be ensured, and reliability can be improved by clarifying the number of delayed frames Fd.
[0066] In the first embodiment, after the delayed frame Fd corresponding to the number of frames N corresponding to the ratio of the regular interval Ti to the delayed acquisition interval Td is determined, the image frame Fm within the recovery determination range Rr within the delay allowable range Ra is determined as a quasi-delayed frame Fs. In addition, quasi-delay information Is different from the delay information Id is added to the quasi-delayed frame Fs. Thus, the quasi-delayed frame Fs that is incidentally delayed due to the influence of the delayed frame Fd can be determined and the quasi-delay information Is can be added. Therefore, it is possible to ensure versatility and improve reliability by taking into account incidental delays.
[0067] In the first embodiment, the determination of the delayed frame Fd is stopped when the acquisition interval Tr increases from within the recovery determination range Rr to outside the recovery determination range Rr within the delay allowable range Ra. Thus, the determination of the delayed frame Fd continues until the delay of the image frame Fm is eliminated, and the delay information Id can be accurately added only to the delayed frame Fd affected by the delay. Therefore, versatility can be ensured and reliability can be improved.
[0068] The recovery determination range Rr of the first embodiment is a range smaller than the normal interval Ti. Thus, the quasi-delayed frame Fs incidentally delayed due to the influence of the delayed frame Fd can be accurately identified and the quasi-delayed information Is can be added. Therefore, high reliability can be ensured taking into account the incidental delay.
[0069] The image processing system of the first embodiment distributes time-series image data Dm including image frames Fm of delayed frames Fd to which delay information Id is added to the user of the output destination. Thus, the user of the output destination can use the image data Dm in consideration of the delay status of the image frames Fm based on the delay information Id of the delay frames Fd. Therefore, both versatility and reliability can be achieved for the user of the output destination.
[0070] (Second Embodiment)
[0071] The second embodiment is a modification of the first embodiment.
[0072] In the image processing flow based on the second embodiment, as Figure 6 As shown, the same S100 to S105 as in the first embodiment are executed. In S102 of the image processing process based on the second embodiment, when it is determined that the interval Tr for acquiring image frames has not increased to outside the delay allowable range Ra and the number of remaining delayed frames Nd is not greater than 1, the image processing process enters S110. In S110, the delay determination block 210 sets the current time as the regular acquisition time To of the image frame Fm. After executing S110, the image processing process enters S106. In addition, S106 to S108 are the same as the first embodiment.
[0073] On the other hand, in S111 following S105, the delay determination block 210 estimates the regular acquisition time To as a time related to the regular interval Ti and the number of frames N of the delay frame Fd. Specifically, the delay determination block 210 estimates the regular acquisition time To of the image frame Fm as a time calculated by adding a value (Ti×N) obtained by multiplying the regular interval Ti by the number of frames N of the delay frame Fd to the acquisition time of the image frame Fm acquired before the delay frame Fd is determined.
[0074] After S111 is executed, the image processing flow enters S112. In addition, after S107 is executed, the image processing flow also enters S112. Furthermore, after S108 is executed, the image processing flow also enters S112. In S112, the delay determination block 210 adds the regular acquisition time To to the image frame Fm.
[0075] In S113 following S112, the delay determination block 210 estimates the sending time of the image frame Fm from the infrastructure camera 10. Specifically, the delay determination block 210 estimates the sending time of the image frame Fm by tracing back the time by the constant delay time Trd from the regular acquisition time To. Here, the constant delay time Trd is defined as the time that is constantly generated from the sending of the image frame Fm by the infrastructure camera 10 to the acquisition of the image frame Fm by the image processing system 20 under normal circumstances where there is no influence such as interference in the communication network NW. Furthermore, in S113, the delay determination block 210 adds the sending time estimated as described above to the image frame Fm. In addition, S109 following S113 is the same as the first embodiment.
[0076] Through the above image processing flow, the image processing system 20 attaches the regular acquisition time To and the delay information Id, the quasi-delay information Is, or the normal information to the image frame Fm. Furthermore, the image processing system 20 attaches the time traced back by the constant delay time Trd from the regular acquisition time To as the sending time of the delay frame Fd and the delay information Id, the quasi-delay information Is, or the normal information to the image frame Fm.
[0077] (Effect)
[0078] Next, the effects of the second embodiment described above will be described.
[0079] In the second embodiment, the regular acquisition time To associated with the regular interval Ti and the number of frames N of the delayed frame Fd is added to the delayed frame Fd together with the delay information Id. Thus, in addition to the delay information Id, the delay of the delayed frame Fd can be correctly identified based on the regular acquisition time To according to the image processing system 20. Therefore, versatility can be ensured and reliability can be improved.
[0080] In the image processing system 20 of the second embodiment, the time of the constant delay time Trd generated from the regular acquisition time To to the acquisition of the image frame Fm by the infrastructure camera 10 is traced back as the transmission time of the delay frame Fd, and is added to the delay frame Fd together with the delay information Id and the regular acquisition time To. Thus, in addition to the delay information Id, the delay of the delay frame Fd can be correctly identified based on the regular acquisition time To of the image processing system 20 and the transmission time from the infrastructure camera 10. Therefore, it is possible to ensure particularly high reliability.
[0081] (Third Embodiment)
[0082] The third embodiment is a modification of the first embodiment.
[0083] like Figures 7 to 9 As shown, the image transmission network system 1A based on the third embodiment transmits the image data Dm to which the image frame Fm captured by the vehicle-mounted camera 10A is processed and information is added to the transmission target system 30A. Therefore, the image transmission network system 1A is configured to include the image processing system 20A, the transmission target system 30A, and the vehicle-mounted network NWA and / or the communication network NW together with the vehicle-mounted camera 10A. Here, the vehicle-mounted network NWA is a LAN (Local Area Network) or the like constructed in the host vehicle 40A, and is connected to a plurality of electronic devices such as sensors and / or actuators other than the vehicle-mounted camera 10A in the host vehicle 40A.
[0084] The vehicle-mounted camera 10A and the image processing system 20A in the image transmission network system 1A are mounted on a host vehicle 40A and are connected to each other via the vehicle-mounted network NWA. Figure 7 , 9 As shown in FIG. 1 , the transmission destination system 30A in the image transmission network system 1A may also be mounted on a host vehicle 40A (ie, the host vehicle) and connected to the image processing system 20A via the vehicle network NWA. Figure 8 , 9 As shown, the transmission destination system 30A in the image transmission network system 1A may be mounted on another vehicle other than the host vehicle 40A, or constructed by a network server outside the host vehicle 40A, and connected to the image processing system 20A via the communication network NW.
[0085] like Figures 7 to 9As shown, the vehicle-mounted camera 10A of the third embodiment corresponding to the infrastructure camera 10 of the first embodiment is a fixed or movable camera relative to the host vehicle 40A. The vehicle-mounted camera 10A captures objects such as road users and obstacles on the road outside the host vehicle 40A. The vehicle-mounted camera 10A sends the captured image frames Fm as time-series images to the image processing system 20A via the vehicle network NWA. At this time, the vehicle-mounted camera 10A sends the image frames Fm in a time series at a constant sending cycle. As such a vehicle-mounted camera 10A, a single camera or a plurality of cameras are assumed in the host vehicle 40A, but in the following description, the processing based on a single vehicle-mounted camera 10A is representatively listed.
[0086] The image processing system 20A of the third embodiment corresponding to the image processing system 20 of the first embodiment is constructed by, for example, an ECU that processes image data Dm by controlling an onboard camera 10A in a host vehicle 40A such as an autonomous vehicle. The image processing system 20A receives an image frame Fm from the onboard camera 10A via the onboard network NWA between the onboard camera 10A and processes the image frame Fm obtained by decryption. The image processing system 20A outputs the image data Dm obtained by processing the image frame Fm to the destination system 30A via the onboard network NWA and / or the communication network NW. Such an image processing system 20A detects a delay from the transmission of the image frame Fm by the onboard camera 10A to the acquisition by the image processing system 20A as a delay generated in the onboard network NWA due to, for example, communication congestion. Therefore, the image processing system 20A adds information indicating the delay detection result to the image data Dm and provides it to the destination system 30A.
[0087] The delivery target system 30A of the third embodiment corresponding to the distribution target system 30 of the first embodiment is managed by the receiving user of the output target who needs the time-series image data Dm of the image frame Fm output from the image processing system 20A. The delivery target system 30A uses the image data Dm sent by the output from the image processing system 20A according to the receiving purpose of each receiving user. The delivery target system 30A can be constructed by an ECU that uses the image data Dm for controlling the host vehicle 40A or other vehicles that are road users such as autonomous driving vehicles. The delivery target system 30A can also be constructed by a network server that is a service provider of the user that uses the image data Dm for services such as traffic-related services.
[0088] The dedicated computer constituting the image processing system 20A that provides information to the destination system 30A includes at least one memory 202 and one processor 201 according to the first embodiment. Therefore, the plurality of functional blocks constructed by the image processing system 20A are configured in accordance with the first embodiment. Figure 2 The first embodiment includes an image frame acquisition block 200, a delay determination block 210, and an image data output block 220. However, in the image processing flow of the third embodiment performed by these blocks 200, 210, and 220, the infrastructure camera 10, the image processing system 20, and the distribution target system 30 in the flow described in the first embodiment are replaced with the vehicle camera 10A, the image processing system 20A, and the distribution target system 30A, respectively.
[0089] According to such a third embodiment, the effect of the first embodiment can be exerted. Here, through the third embodiment, it is also possible to identify the delay status of the image frame Fm regardless of the specifications of the vehicle-mounted camera 10A by outputting the time-series image data Dm of the image frame Fm including the delay frame Fd to which the delay information Id is added, thereby achieving both versatility and reliability.
[0090] (Other embodiments)
[0091] Although a plurality of embodiments have been described above, the present invention is not limited to these embodiments and can be applied to various embodiments and combinations within a scope not departing from the gist of the present invention.
[0092] In a modified example, S111 to S113 may be omitted in the image processing flow described in the second embodiment, and the regular acquisition time To is not added to the image frame Fm. In a modified example, the second embodiment may be applied to the third embodiment. However, the constant delay time Trd in the image processing flow of the second embodiment applied to the third embodiment may be defined as the time that is constantly generated from the image frame Fm being sent by the vehicle-mounted camera 10A to the image frame Fm being acquired by the image processing system 20A under normal circumstances where there is no influence such as interference in the vehicle-mounted network NWA.
[0093] (Disclosure of technical ideas)
[0094] This specification discloses multiple technical ideas recorded in the multiple items listed below. Several items may be recorded by selectively citing a multiple dependent form of a previous item in a subsequent item. Moreover, several items may be recorded by citing a multiple dependent form of an item in other multiple dependent forms (a multiple dependent form referring to another multiple dependent form). These items recorded in multiple dependent forms define multiple technical ideas. In addition, the symbols in brackets recorded in the disclosure column of this idea represent the corresponding relationship with the specific means recorded in the previously detailed implementation method, and do not limit the technical scope of the present invention.
[0095] (Technical Thought 1)
[0096] An image processing system has a processor and processes image frames obtained from a camera via a network, wherein:
[0097] The processor is configured to perform the following processing:
[0098] Acquire image frames in time series order;
[0099] determining an image frame whose acquisition interval increases to outside a delay tolerance range as a delayed frame delayed compared to an image frame acquired at a regular interval;
[0100] adding delay information to the delay frame; and
[0101] Time-series image data including image frames to which delay information is added is output.
[0102] (Technical Thoughts 2)
[0103] In the image processing system described in the technical concept 1,
[0104] Determine the delay frame contains:
[0105] The video frames having a number corresponding to the ratio of the regular interval set to be constant to the delayed acquisition interval, which is the acquisition interval outside the delay allowable range, are determined as delayed frames.
[0106] (Technical Thoughts 3)
[0107] In the image processing system described in Technical Idea 2,
[0108] Determine the delay frame contains:
[0109] The image frames within the recovery determination range whose acquisition interval is reduced to the delay tolerance range after the delayed frames of the determined number of frames are determined as quasi-delayed frames.
[0110] Additional latency information includes:
[0111] The quasi-delay frame is added with quasi-delay information different from the delay information.
[0112] The output image data includes:
[0113] The image data including the quasi-delay frame to which the quasi-delay information is added is output.
[0114] (Technical Thoughts 4)
[0115] In the image processing system described in Technical Concept 3,
[0116] Determine the delay frame contains:
[0117] The determination of the quasi-delayed frame is stopped in accordance with the acquisition interval increasing from within the delay allowable range from within the recovery determination range to outside the recovery determination range.
[0118] (Technical Thoughts 5)
[0119] In the image processing system described in technical idea 3 or 4,
[0120] The reply judgment range is a range smaller than the normal interval.
[0121] (Technical Thought 6)
[0122] In the image processing system described in any one of technical ideas 2 to 5,
[0123] Additional latency information includes:
[0124] The regular acquisition time associated with the regular interval and number of frames is attached to the delay frame together with the delay information.
[0125] (Technical Thoughts 7)
[0126] In the image processing system described in technical idea 6,
[0127] Additional latency information includes:
[0128] The time traced back from the regular acquisition time by a constant delay time is used as the sending time of the delayed frame and is attached to the delayed frame together with the delay information and the regular acquisition time. The constant delay time is the time that is constantly generated relative to the time when the image frame is sent by the camera to the time when the image frame is acquired by this image processing system.
[0129] (Technical Thoughts 8)
[0130] In the image processing system described in any one of technical ideas 2 to 5,
[0131] Additional latency information includes:
[0132] The time at which a constant delay time is traced back from the regular acquisition time related to the regular interval and number of frames is used as the sending time of the delayed frame and is attached to the delayed frame together with the delay information. The constant delay time is the time that is constantly generated relative to the time when the image frame is sent by the camera to the time when the image frame is acquired by this image processing system.
[0133] (Technical Thoughts 9)
[0134] In the image processing system described in any one of technical concepts 1 to 8,
[0135] The output image frame contains:
[0136] Publish or send image data to users of output destinations.
[0137] (Technical Thoughts 10)
[0138] In the image processing system described in any one of technical concepts 1 to 9,
[0139] An image frame acquired from an infrastructure camera as a camera through an infrastructure network as a network is processed.
[0140] (Technical Thoughts 11)
[0141] In the image processing system described in any one of technical concepts 1 to 9,
[0142] An image frame acquired from an in-vehicle camera as a camera through an in-vehicle network as a network is processed.
[0143] Furthermore, the above-mentioned technical concepts 1 to 11 can be implemented in various forms of methods and programs.
Claims
1. An image processing system, the image processing system (20, 20A) having a processor (201) and processing an image frame (Fm) acquired from a camera (10, 10A) via a network (NW, NWA), characterized in that: The processor is configured to perform the following processing: Acquire the image frames in time series order; determining the image frame whose acquisition interval (Tr) increases to outside the delay allowable range (Ra) as a delayed frame (Fd) delayed compared with the image frame acquired at the regular interval (Ti); adding delay information (Id) to the delay frame; and The time-series video data (Dm) of the video frames including the delayed frames to which the delay information is added is output.
2. The image processing system according to claim 1, characterized in that: Determining the delayed frame includes: The image frames having a number corresponding to a ratio of the regular interval set to be constant to the delayed acquisition interval (Td) as the acquisition interval outside the delay allowable range are determined as the delayed frames.
3. The image processing system according to claim 2, characterized in that: Determining the delayed frame includes: The image frames within the recovery determination range (Rr) in which the acquisition interval is reduced to within the delay tolerance range after the delayed frames of the determined number of frames are determined as quasi-delayed frames (Fs), The additional delay information includes: Adding quasi-delay information (Is) different from the delay information to the quasi-delay frame, Outputting the image data includes: The video data including the quasi-delay frame to which the quasi-delay information is added is output.
4. The image processing system according to claim 3, characterized in that: Determining the delayed frame includes: The determination of the quasi-delayed frame is stopped in accordance with the acquisition interval increasing from the recovery determination range to outside the recovery determination range within the delay allowable range.
5. The image processing system according to claim 3, characterized in that: The reply determination range is a range smaller than the regular interval.
6. The image processing system according to claim 2, characterized in that: The additional delay information includes: A regular acquisition time (To) associated with the regular interval and the number of frames is added to the delay frame together with the delay information.
7. The image processing system according to claim 6, characterized in that: The additional delay information includes: The time traced back by a constant delay time (Trd) from the regular acquisition time is used as the sending time of the delay frame and is attached to the delay frame together with the delay information and the regular acquisition time. The constant delay time is the time that is constantly generated relative to the time when the camera sends the image frame to the time when the image frame is acquired by this image processing system.
8. The image processing system according to claim 2, characterized in that: The additional delay information includes: The time traced back by a constant delay time (Trd) from the regular acquisition time (To) related to the regular interval and the number of frames is used as the sending time of the delayed frame and is attached to the delayed frame together with the delay information. The constant delay time is the time that is constantly generated relative to the time when the image frame is sent by the camera to the time when the image frame is acquired by this image processing system.
9. The image processing system according to claim 1 or 2, characterized in that: Outputting the image frame includes: The image data is published or sent to a user of an output destination.
10. The image processing system according to claim 1 or 2, characterized in that: The image frame acquired from the infrastructure camera (10) as the camera through the infrastructure network (NW) as the network is processed.
11. The image processing system according to claim 1 or 2, characterized in that: The image frame acquired from the vehicle-mounted camera (10A) as the camera through the vehicle-mounted network (NWA) as the network is processed.
12. An image processing method, executed by a processor (201) for detecting a delay in an image frame (Fm) acquired from a camera (10, 10A) via a network (NW, NWA), characterized in that: include: Acquire the image frames in time series order; determining the image frame whose acquisition interval (Tr) increases to outside the delay allowable range (Ra) as a delayed frame (Fd) delayed compared with the image frame acquired at the regular interval (Ti); adding delay information (Id) to the delay frame; and The time-series video data (Dm) of the video frames including the delayed frames to which the delay information is added is output.
13. An image processing program, stored in a storage medium (202), comprising instructions for causing a processor (201) to execute in order to detect a delay in an image frame (Fm) acquired from a camera (10, 10A) via a network (NW, NWA), characterized in that: The command contains: Acquire the image frames in time series order; determining the image frame whose acquisition interval (Tr) increases to outside the delay allowable range (Ra) as a delayed frame (Fd) delayed compared with the image frame acquired at the regular interval (Ti); adding delay information (Id) to the delay frame; and The time-series video data (Dm) of the video frames including the delayed frames to which the delay information is added is output.