An image processing method, apparatus, device and storage medium

Abstract

Embodiments of the present application disclose an image processing method, device and equipment and a storage medium. The method comprises: if a new candidate image frame transmitted through any image transmission channel is received or processing of a previous target image frame is completed, determining a processing duration and a remaining duration of the candidate image frame; wherein the candidate image frame is an unprocessed image frame transmitted through at least two image transmission channels; determining a target image frame from the candidate image frame according to the processing duration and the remaining duration of the candidate image frame; processing the target image frame, and updating the processing duration and the remaining duration of the candidate image frame and the target image frame according to a processed duration of the target image frame. The technical solution can process video images according to actual frame rates of the video images on one data link, and can better balance image processing quality and resource cost, thereby better meeting actual application requirements.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] With the development of image processing technology, people have placed higher demands on image presentation quality. When processing video images at multiple frame rates simultaneously, existing technologies typically employ two methods: The first method involves buffering video images at their respective frame rates into storage media, then setting a fixed frame rate to read the buffered video images at each frame rate from the storage media, thus achieving video image processing through a single data link. The second method uses multiple data links to buffer, read, and process video images according to their respective frame rates.

[0003] The first approach uses a fixed frame rate to read video images at different frame rates. When the input video frame rate is higher than the fixed reading frame rate, frame drops occur, resulting in discontinuous display. When the input video frame rate is lower than the fixed reading frame rate, video duplication occurs, leading to the transmission of identical video and wasting bandwidth. The second approach uses multiple data links, requiring each video to have its own independent data link for processing, increasing resource costs. Therefore, neither of these approaches can process video images at the actual frame rate of each video image on a single data link, failing to balance image processing quality and resource cost, and thus failing to meet practical application requirements. Summary of the Invention

[0004] This invention provides an image processing method, apparatus, device, and storage medium that can process video images on a single data link according to the actual frame rate of each video image, effectively balancing image processing quality and resource cost, and better meeting practical application needs.

[0005] According to one aspect of the present invention, an image processing method is provided, the method comprising:

[0006] If a new candidate image frame is received through any image transmission channel, or if the processing of the previous target image frame is completed, the processing time and remaining time of the candidate image frame are determined.

[0007] Wherein, the candidate image frame is an unprocessed image frame transmitted through at least two image transmission channels, the processing time is used to represent the time required to process the unprocessed pixels in the candidate image frame, and the remaining time is used to represent the time from the current moment to the receipt of the next candidate image frame transmitted through the image transmission channel corresponding to the candidate image frame, and the difference between the remaining time and the processing time of the candidate image frame.

[0008] The target image frame is determined from the candidate image frames based on the processing time and remaining time of the candidate image frames.

[0009] The target image frame is processed, and the processing time and remaining time of the candidate image frame and the target image frame are updated according to the processing time of the target image frame; wherein, the processing time is the time from the start of processing the target image frame to the current time.

[0010] According to another aspect of the present invention, an image processing apparatus is provided, comprising:

[0011] The duration determination module is used to determine the processing duration and remaining duration of the candidate image frame if a new candidate image frame is received through any image transmission channel, or if the processing of the previous target image frame is completed.

[0012] Wherein, the candidate image frame is an unprocessed image frame transmitted through at least two image transmission channels, the processing time is used to represent the time required to process the unprocessed pixels in the candidate image frame, and the remaining time is used to represent the time from the current moment to the receipt of the next candidate image frame transmitted through the image transmission channel corresponding to the candidate image frame, and the difference between the remaining time and the processing time of the candidate image frame.

[0013] The target image frame determination module is used to determine the target image frame from the candidate image frames based on the processing time and remaining time of the candidate image frames;

[0014] The duration update module is used to process the target image frame and update the processing duration and remaining duration of the candidate image frame and the target image frame according to the processing duration of the target image frame; wherein, the processing duration is the duration from the start of processing the target image frame to the current time.

[0015] According to another aspect of the present invention, an image processing electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the image processing method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the image processing method according to any embodiment of the present invention.

[0020] The technical solution of this invention involves determining the processing time and remaining time of a candidate image frame if a new candidate image frame is received via any image transmission channel, or if processing of the previous target image frame is completed. The candidate image frame is an unprocessed image frame transmitted via at least two image transmission channels. The processing time represents the time required to process unprocessed pixels in the candidate image frame, and the remaining time represents the difference between the current time and the processing time of the next candidate image frame transmitted via the image transmission channel corresponding to the received candidate image frame. Based on the processing time and remaining time of the candidate image frame, a target image frame is determined from the candidate image frames. The target image frame is processed, and the processing time and remaining time of both the candidate image frame and the target image frame are updated based on the processed time of the target image frame. The processed time is the time from the start of processing the target image frame to the current time. This technical solution enables video images to be processed according to the actual frame rate of each video image on a single data link, which better balances image processing quality and resource cost, ensures the continuity of video display, saves video processing bandwidth, and reduces resource consumption, thereby better meeting the needs of practical applications.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of an image processing method provided according to Embodiment 1 of the present invention;

[0024] Figure 2 This is a schematic diagram of an image processing method provided according to Embodiment 1 of the present invention;

[0025] Figure 3 This is a flowchart of an image processing method provided according to Embodiment 2 of the present invention;

[0026] Figure 4 This is a schematic diagram of an image processing system provided according to Embodiment 2 of the present invention;

[0027] Figure 5 This is a schematic diagram of the structure of an image processing device according to Embodiment 3 of the present invention;

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device that implements an image processing method according to an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] Example 1

[0032] Figure 1 This is a flowchart of an image processing method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations that balance image processing quality and resource cost. The method can be executed by an image processing device, which can be implemented in hardware and / or software, and can be configured in an electronic device with data processing capabilities. Figure 1 As shown, the method includes:

[0033] S110: If a new candidate image frame is received through any image transmission channel, or the processing of the previous target image frame is completed, the processing time and remaining time of the candidate image frame are determined.

[0034] The image transmission channel can be an image acquisition channel, i.e., the channel through which the image acquisition device transmits real-time acquired image frames. The image transmission channel can also be a channel for transmitting candidate image frames from a storage area to an electronic device executing the present application. A candidate image frame can refer to an unprocessed image frame transmitted through at least two image transmission channels. The candidate image frame can include a newly received candidate image frame and other candidate image frames received before the new candidate image frame was received. The previous target image frame can refer to an image frame that was being processed before the new candidate image frame was received. The processing time can be used to represent the time required to process unprocessed pixels in the candidate image frame. The remaining time can be used to represent the time from the current moment until the next candidate image frame is transmitted through the image transmission channel corresponding to the received candidate image frame, and the difference between the remaining time and the processing time of the candidate image frame.

[0035] In this embodiment, if a new candidate image frame is received through any image transmission channel, or if the processing of the previous target image frame is completed, it is necessary to determine the processing time and remaining time of the candidate image frame. If the processing of the previous target image frame is completed, the processing time of the candidate image frame can remain unchanged, and the difference between the remaining time of the candidate image frame and the processing time of the previous target image frame can be used as the new remaining time of the candidate image frame.

[0036] In this embodiment, optionally, if a new candidate image frame is received transmitted through any image transmission channel, the processing time and remaining time of the candidate image frame are determined, including: determining the time required to process all pixels of the new candidate image frame based on the ratio of the resolution of the new candidate image frame to the number of pixels processed by the image processing device per unit time; determining the processing time of the new candidate image frame based on the time required to process all pixels of the new candidate image frame; determining the time occupied by the new candidate image frame based on the frame rate of the video to which the new candidate image frame belongs; and taking the difference between the time occupied by the new candidate image frame and the processing time as the remaining time of the new candidate image frame.

[0037] The image processing device can refer to any device capable of image processing, such as a computer. The unit time can be set according to actual needs; this embodiment is not limited in this regard. For example, the unit time can be set to 1 second or 1 μs. The frame rate refers to the number of images transmitted per second and can be used to characterize the image transmission speed. A higher frame rate indicates a faster image transmission speed.

[0038] In this embodiment, if the candidate image frame is a new candidate image frame, the resolution of the new candidate image frame and the number of pixels processed by the image processing device per unit time (the processing capability of the image processing device) are first determined. Then, based on the resolution of the new candidate image frame and the processing capability of the image processing device, the time required to process all pixels of the new candidate image frame is determined, and the processing time of the new candidate image frame is determined based on this time. For example, assuming the resolution of the new candidate image frame is 1920×1080 pixels, and the processing capability of the image processing device is 300Mpixel / s, the processing time can be determined based on the ratio of the resolution to the number of pixels processed by the image processing device per unit time. The processing time is determined by several factors. For example, the ratio of resolution to the number of pixels processed by the image processing device per unit time can be used as the processing time. Alternatively, the ratio can be rounded up or down to an integer consistent with the minimum timing unit of the timer. Then, the duration occupied by the new candidate image frame is determined based on the frame rate of the video to which it belongs. For instance, the reciprocal of the frame rate of the video to which the new candidate image frame belongs can be used to determine the duration occupied by the new candidate image frame. For example, assuming the frame rate of the video to which the new candidate image frame belongs is 60fps, the duration occupied by the new candidate image frame can be determined as 1 / 60 = 16667 μs. The difference between the duration occupied by the new candidate image frame and the processing time is then used as the remaining duration of the new candidate image frame. Based on the above example, the remaining duration of the new candidate image frame can be determined as 16667 - 6912 = 9755 μs.

[0039] This solution, through such a setting, allows for the rapid and accurate determination of processing time based on the resolution of the new candidate image frame, the number of pixels processed by the image processing device per unit time, and the rapid and accurate determination of the remaining duration of the new candidate image frame based on the frame rate of the video to which the new candidate image frame belongs, as well as the processing time.

[0040] In this embodiment, optionally, if the candidate image frame is not a new candidate image frame, the processing time and remaining time of the candidate image frame are determined, including: for each candidate image frame, the updated processing time and remaining time based on the processing time of the previous target image frame are used as the processing time and remaining time of the candidate image frame.

[0041] In this embodiment, if a candidate image frame is not a new candidate image frame, for each candidate image frame, the updated processing time and remaining time based on the processing time of the previous target image frame can be used as the processing time and remaining time of the candidate image frame. Specifically, if the processing of the previous target image frame is completed, the processing time of each candidate image frame can remain unchanged, and the remaining time of each candidate image frame can be updated based on the difference between the remaining time of the candidate image frame and the processing time of the previous target image frame, while the processing time of the previous target image frame becomes zero.

[0042] It should be noted that candidate image frames are not fixed and can be dynamically adjusted based on image processing progress. If a candidate image frame is determined to be the target image frame, then the candidate image frame becomes the target image frame. The target image frame can refer to the image frame currently being processed. If the updated processing time of the previous target image frame is not zero, then the previous target image frame is used as a candidate image frame. Specifically, if the updated processing time of the previous target image frame is not zero, it indicates that a new candidate image frame has been received at some point, but the processing of the previous target image frame has not yet finished. In this case, the previous target image frame needs to be used again as a candidate image frame so that the target image frame can be re-determined based on the candidate image frame later.

[0043] This scheme, through such settings, can quickly and accurately determine the processing time and remaining time of candidate image frames.

[0044] S120, determine the target image frame from the candidate image frames based on the processing time of the candidate image frames and the remaining time.

[0045] In this embodiment, after determining the processing time and remaining time of the candidate image frames, the target image frame can be determined from the candidate image frames based on the processing time and remaining time. Optionally, determining the target image frame from the candidate image frames based on the processing time and remaining time of the candidate image frames includes: selecting the target image frame from the candidate image frames whose processing time is not zero based on the remaining time of the candidate image frames.

[0046] In this embodiment, a target image frame can be selected from candidate image frames with a non-zero processing time based on the remaining duration of the candidate image frames. A non-zero processing time indicates that the candidate image frame has not started processing or has not been processed completely. If the processing time is zero, it indicates that the corresponding image frame has been processed completely, and that image frame is no longer considered a candidate image frame. For example, a candidate image frame with a relatively short remaining duration and / or processing time can be selected as the target image frame from among the candidate image frames with a non-zero processing time. The remaining duration can be considered as a priority factor. For instance, the candidate image frame with the smallest remaining duration can be selected as the target image frame from among the candidate image frames with a non-zero processing time, thereby ensuring image processing quality and avoiding discontinuous video display.

[0047] This solution, through this setting, can determine the target image frame from candidate image frames with non-zero processing time based on the remaining duration of the candidate image frames, thus ensuring image processing quality and avoiding discontinuous video display.

[0048] In this embodiment, optionally, selecting a target image frame from candidate image frames with non-zero processing time based on the remaining duration of the candidate image frames includes: determining the candidate image frame with the smallest remaining duration from the candidate image frames with non-zero processing time as the target image frame; or, determining the candidate image frame with the smallest non-zero processing time from the candidate image frames with equal remaining durations as the target image frame; or, randomly determining a candidate image frame as the target image frame from the candidate image frames with equal remaining durations and equal non-zero processing times.

[0049] In this embodiment, the candidate image frame with the smallest remaining processing time can be selected from candidate image frames with a non-zero processing time as the target image frame, thereby ensuring image processing quality. If the remaining processing time of all candidate image frames is equal, the candidate image frame with the smallest non-zero processing time is determined as the target image frame, thereby improving image processing efficiency. If the processing time and remaining processing time of all candidate image frames are equal, the image frame processing order is not restricted, and a candidate image frame can be randomly selected from candidate image frames with a non-zero processing time as the target image frame.

[0050] This solution, through its configuration, can better balance image processing quality and processing efficiency, and can better meet the needs of practical applications.

[0051] S130, process the target image frame, and update the processing time and remaining time of the candidate image frame and the target image frame according to the processing time of the target image frame; wherein, the processing time is the time from the start of processing the target image frame to the current time.

[0052] In this embodiment, after the target image frame is determined, it can be processed by an image processing device. Then, based on the processing time already taken for the target image frame, the processing times and remaining times for both the candidate image frames and the target image frame are updated. It should be noted that if the target image frame processing is complete, the processed time is equal to the time required to process all pixels of the target image frame; if a new candidate image frame is received before the target image frame processing is complete, the processed time is less than the time required to process all pixels of the target image frame.

[0053] In this embodiment, optionally, the processing time and remaining time of the candidate image frame and the target image frame are updated according to the processing time of the target image frame, including: taking the difference between the remaining time of the candidate image frame and the processing time as the updated remaining time of the candidate image frame, and keeping the processing time of the candidate image frame unchanged; taking the difference between the processing time of the target image frame and the processing time as the updated processing time of the target image frame, and keeping the remaining time of the target image frame unchanged.

[0054] In this embodiment, when updating the processing time and remaining time of candidate image frames, the difference between the remaining time of the candidate image frame and the processed time of the target image frame can be used as the updated remaining time of the candidate image frame, while the processing time of the candidate image frame remains unchanged. Similarly, when updating the processing time and remaining time of the target image frame, the difference between the processing time of the target image frame and the processed time can be used as the updated processing time of the target image frame, while the remaining time of the target image frame remains unchanged.

[0055] It should be noted that in the embodiments of this application, if there is idle time, that is, the time between the end of processing the previous target image frame and the processing of the current target image frame when no image frame is processed, the processing time and the remaining time are updated according to the idle time. That is, the processing time is reduced by the idle time to obtain the updated processing time, and the remaining time is reduced by the idle time to obtain the updated remaining time.

[0056] The technical solution of this invention involves determining the processing time and remaining time of a candidate image frame if a new candidate image frame is received via any image transmission channel, or if processing of the previous target image frame is completed. The candidate image frame is an unprocessed image frame transmitted via at least two image transmission channels. The processing time represents the time required to process unprocessed pixels in the candidate image frame, and the remaining time represents the difference between the current time and the processing time of the next candidate image frame transmitted via the image transmission channel corresponding to the received candidate image frame. Based on the processing time and remaining time of the candidate image frame, a target image frame is determined from the candidate image frames. The target image frame is processed, and the processing time and remaining time of both the candidate image frame and the target image frame are updated based on the processed time of the target image frame. The processed time is the time from the start of processing the target image frame to the current time. This technical solution enables video images to be processed according to the actual frame rate of each video image on a single data link, which better balances image processing quality and resource cost, ensures the continuity of video display, saves video processing bandwidth, and reduces resource consumption, thereby better meeting the needs of practical applications.

[0057] Figure 2 This is a schematic diagram of an image processing method provided in Embodiment 1 of the present invention. Ch1, Ch2, and Ch3 represent three image transmission channels. The image frames transmitted through Ch1-Ch3 all have a resolution of 1920×1080, and frame rates of 60fps, 50fps, and 30fps, respectively. That is, the durations occupied by candidate image frames are 1 / 60 = 16667μs, 1 / 50 = 20000μs, and 1 / 30 = 33333μs, respectively. Assuming that candidate image frames from channels Ch1 to Ch3 arrive simultaneously at time 0, and the image processing device processes 300 Mpixels / s per unit time, the processing time and remaining time of the candidate image frames at time 0 can be determined as follows: the processing time for channel Ch1 is 6912 μs, and the remaining time is 9755 μs; the processing time for channel Ch2 is 6912 μs, and the remaining time is 13088 μs; the processing time for channel Ch3 is 6912 μs, and the remaining time is 26421 μs.

[0058] At time 0, candidate image frames Ch1-Ch3 arrive simultaneously, triggering a priority sorting operation to determine the target image frame. At time 0, the processing time of the candidate image frame in channel Ch1 is not zero and the remaining time is the smallest, therefore, the candidate image frame in channel Ch1 can be prioritized as the target image frame. The target image frame in channel Ch1 is processed at time 6912, taking 6912μs, at which point priority sorting is triggered again. At time 6912, the processing time of the target image frame in channel Ch1 is updated to 0, and the remaining time remains unchanged at 9755; the processing time of the candidate image frame in channel Ch2 remains unchanged at 6912, and the remaining time is updated to 6176 (i.e., 13088-6912); the processing time of the candidate image frame in channel Ch3 remains unchanged at 6912, and the remaining time is updated to 19509 (i.e., 26421-6912). At time 6912, the processing time of the candidate image frames in channel Ch2 is not zero and the remaining time is the smallest. Therefore, the candidate image frames in channel Ch2 can be prioritized as target image frames. Ch2 processing is completed at time 13824, taking 6912 μs, at which point priority sorting is triggered again. At time 13824, the processing time of the candidate image frames in channel Ch1 remains 0, and the remaining time remains 9755; the processing time of the target image frames in channel Ch2 remains updated to 0, and the remaining time remains 6176; the processing time of the candidate image frames in channel Ch3 remains 6912, and the remaining time is updated to 12597 (i.e., 19509-6912). At time 13824, only the processing time of the candidate image frames in channel Ch3 is not zero, therefore, the candidate image frames in channel Ch3 can be prioritized as target image frames.

[0059] At time 16667, a new candidate image frame is received in channel Ch1. The processing time for the candidate image frame in channel Ch1 needs to be updated to 6912, the remaining time to 9755, and a priority sorting is triggered. At time 16667, the target image frame in channel Ch3 has not yet finished processing. This target image frame is again considered a candidate image frame, and its processing time needs to be updated to 6912 - (16667 - 13824) = 4069, while the remaining time remains unchanged at 12597. At time 16667, the processing time for the candidate image frame in channel Ch1 is not zero, and its remaining time is the smallest. Therefore, the candidate image frame in channel Ch1 can be prioritized as the target image frame.

[0060] At time 20000, channel Ch2 receives a new candidate image frame. The processing time for this candidate image frame needs to be updated to 6912, the remaining time to 13088, and a priority sorting is triggered. At time 20000, the target image frame in channel Ch1 is still not finished processing. It is re-selected as a candidate image frame, and its processing time needs to be updated again to 6912 - (20000 - 16667) = 3579, while the remaining time remains unchanged at 9755. At time 20000, the processing time for the candidate image frame in channel Ch3 is not zero, and its remaining time is the smallest. Therefore, it is prioritized as the target image frame. At time 24069, the target image frame in channel Ch3 is processed, taking 4069 μs. Priority sorting is triggered again. This process is repeated cyclically to determine the priority of each video stream and process them accordingly, ensuring that each video stream is processed according to its own frame rate.

[0061] Example 2

[0062] Figure 3 This is a flowchart of an image processing method provided in Embodiment 2 of the present invention. This embodiment is based on the above embodiment and optimized. Specifically, the optimization is as follows: the method further includes: if a new candidate image frame is received through any image transmission channel and the processing of the previous target image frame has not been completed, then the processed pixel row of the previous target image frame is recorded, and the previous target image frame is used as a candidate image frame. When the candidate image frame is determined as the target image frame again for processing, processing starts from the next row of processed pixel rows.

[0063] like Figure 3 As shown, the method in this embodiment specifically includes the following steps:

[0064] S210: If a new candidate image frame is received through any image transmission channel, or the processing of the previous target image frame is completed, the processing time and remaining time of the candidate image frame are determined.

[0065] Among them, the candidate image frame is an unprocessed image frame transmitted through at least two image transmission channels; the processing time is used to represent the time required to process the unprocessed pixels in the candidate image frame; the remaining time is used to represent the time from the current moment to the next candidate image frame transmitted through the image transmission channel corresponding to the received candidate image frame, and the difference between the remaining time and the processing time of the candidate image frame.

[0066] S220, determine the target image frame from the candidate image frames based on the processing time of the candidate image frames and the remaining time.

[0067] S230, process the target image frame, and update the processing time and remaining time of the candidate image frame and the target image frame according to the processing time of the target image frame; wherein, the processing time is the time from the start of processing the target image frame to the current time.

[0068] The specific implementation methods of S210-S230 can be found in the detailed description of S110-S130, and will not be repeated here.

[0069] S240: If a new candidate image frame is received through any image transmission channel and the processing of the previous target image frame has not been completed, the processed pixel row of the previous target image frame is recorded, and the previous target image frame is used as a candidate image frame. When the candidate image frame is determined as the target image frame again for processing, processing starts from the next row of processed pixel rows.

[0070] In this context, "processed pixel rows" refers to the pixel rows in the previous target image frame that have already been processed. In this embodiment, if a new candidate image frame is received via any image transmission channel, and the processing of the previous target image frame is not yet complete, the target image frame needs to be redefined, thus interrupting the processing of the previous target image frame. To improve image processing efficiency, the processed pixel rows of the previous target image frame can be recorded so that when the previous target image frame is subsequently determined as the target image frame again, processing can continue directly from the next row of processed pixel rows, avoiding redundant processing of processed pixel rows. It should be noted that when redefined as the target image frame, the previous target image frame must be considered as a candidate image frame.

[0071] The technical solution of this invention, if a new candidate image frame is received transmitted through any image transmission channel and the processing of the previous target image frame has not yet ended, records the processed pixel rows of the previous target image frame and uses the previous target image frame as a candidate image frame. When the candidate image frame is determined as the target image frame again for processing, processing begins from the next row of processed pixel rows. This technical solution enables video image processing on a single data link according to the actual frame rate of each video image, effectively balancing image processing quality and resource costs, ensuring video display continuity, saving video processing bandwidth, and reducing resource consumption. By recording the processed pixel rows of the previous target image frame, processing can continue from the next row when the previous target image frame is processed again, thus avoiding repeated processing of processed pixel rows, improving image processing efficiency, and better meeting practical application needs.

[0072] Figure 4This is a schematic diagram of an image processing system provided in Embodiment 2 of the present invention. This image processing system is a video stitching and control system, capable of processing multiple video streams with different frame rates through a single data link. For example... Figure 4 As shown, the system may include a video acquisition unit, a pulse sequencing unit, a video stream control unit, a video storage unit, a storage medium, and a video processing unit. Among these, the video splicing and control system contains multiple video acquisition units (…). Figure 4 Taking three units as an example, each video acquisition unit is an image transmission channel, which can be located in different external chips. These external chips can be FPGAs (Field Programmable Gate Arrays), DSPs (Digital Signal Processing), or SOCs (System on Chips), etc. Specifically, the video acquisition units acquire video images from the input sources and transmit the acquired image data to the video storage unit. Simultaneously, the video acquisition units are responsible for calculating the frame rate and resolution information of each video input source and sending this information, along with the frame pulse signal, to the pulse sequencing unit.

[0073] The pulse sorting unit processes the video pulses and frame rate information transmitted by the video acquisition unit, sorts and selects the image transmission channel with the highest priority, and identifies the image frame of that channel as the target image frame. Specifically, for each input source video, the pulse sorting unit provides two timers: a frame processing timer and a frame idle timer. The frame processing timer records the processing duration of the image frame, and the frame idle timer records the remaining duration of the image frame. Whenever a new video stream frame pulse signal arrives (i.e., a new candidate image frame is received), or when processing of the target image frame is completed, priority reordering can be triggered to redetermine the target image frame.

[0074] The video stream control unit can generate video read commands and send them to the video storage unit. Specifically, the video stream control unit operates on a line-by-line basis and internally records the line count value processed for each video stream. Simultaneously, the video stream control unit supports pausing the current video stream, saving the line count value being processed (i.e., the processed pixel lines), and then switching to a higher-priority video stream (i.e., the redefined target image frame), reading the processed pixel lines recorded in the redefined target image frame, and generating a read command.

[0075] The video storage unit stores video image data from the video acquisition unit into a storage medium, and reads and transmits video image data from the storage medium according to commands from the video stream control unit. The storage medium can be used to store video image data from various input sources. The video processing unit processes the video image data read from the storage medium and may include one or more units. The video processing unit is a critical channel limiting system performance; these units can be distributed across different hardware chips and connected via high-speed interfaces.

[0076] Example 3

[0077] Figure 5 This is a schematic diagram of an image processing apparatus provided in Embodiment 3 of the present invention. This apparatus can execute the image processing method provided in any embodiment of the present invention, and possesses the corresponding functional modules and beneficial effects for executing the method. For example... Figure 5 As shown, the device includes:

[0078] The duration determination module 310 is used to determine the processing duration and remaining duration of the candidate image frame if a new candidate image frame is received through any image transmission channel, or if the processing of the previous target image frame is completed.

[0079] Wherein, the candidate image frame is an unprocessed image frame transmitted through at least two image transmission channels, the processing time is used to represent the time required to process the unprocessed pixels in the candidate image frame, and the remaining time is used to represent the time from the current moment to the receipt of the next candidate image frame transmitted through the image transmission channel corresponding to the candidate image frame, and the difference between the remaining time and the processing time of the candidate image frame.

[0080] The target image frame determination module 320 is used to determine the target image frame from the candidate image frames based on the processing time and remaining time of the candidate image frames;

[0081] The duration update module 330 is used to process the target image frame and update the processing duration and remaining duration of the candidate image frame and the target image frame according to the processing duration of the target image frame; wherein, the processing duration is the duration from the start of processing the target image frame to the current time.

[0082] Optionally, the duration update module 330 includes:

[0083] The candidate image frame duration update unit is used to take the difference between the remaining duration of the candidate image frame and the processed duration as the updated remaining duration of the candidate image frame, and keep the processing duration of the candidate image frame unchanged.

[0084] The target image frame duration update unit is used to take the difference between the processing duration of the target image frame and the already processed duration as the updated processing duration of the target image frame, and keep the remaining duration of the target image frame unchanged.

[0085] Optionally, the target image frame determination module 320 includes:

[0086] The target image frame determination unit is used to select a target image frame from the candidate image frames whose processing time is not zero, based on the remaining duration of the candidate image frames.

[0087] Optionally, the target image frame determination unit is used for:

[0088] From the candidate image frames whose processing time is not zero, determine the candidate image frame with the smallest remaining time as the target image frame; or...

[0089] From the candidate image frames with equal remaining durations, determine the candidate image frame with the smallest non-zero processing duration as the target image frame; or,

[0090] From candidate image frames with equal remaining duration and equal processing duration, randomly select candidate image frames as target image frames.

[0091] Optionally, if the candidate image frame is a new candidate image frame, then the duration determination module 310 is used to:

[0092] The time required to process all pixels of the new candidate image frame is determined based on the ratio of the resolution of the new candidate image frame to the number of pixels processed by the image processing device per unit time. The processing time of the new candidate image frame is determined based on the time required to process all pixels of the new candidate image frame.

[0093] Based on the frame rate of the video to which the new candidate image frame belongs, determine the duration occupied by the new candidate image frame, and use the difference between the duration occupied by the new candidate image frame and the processing time as the remaining duration of the new candidate image frame.

[0094] Optionally, if the candidate image frame is not a new candidate image frame, the duration determination module 310 is used to:

[0095] For each candidate image frame, the updated processing time and remaining time will be used as the processing time and remaining time of the candidate image frame, based on the processing time of the previous target image frame.

[0096] Optionally, the device further includes:

[0097] The processed pixel row recording module is used to record the processed pixel rows of the previous target image frame if a new candidate image frame is received through any image transmission channel and the processing of the previous target image frame has not been completed, and to use the previous target image frame as a candidate image frame so that when the candidate image frame is determined as the target image frame again for processing, the processing starts from the next row of processed pixel rows.

[0098] The image processing apparatus provided in this embodiment of the invention can execute an image processing method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0099] Example 4

[0100] Figure 6 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0101] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0102] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0103] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as image processing methods.

[0104] In some embodiments, the image processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the image processing method by any other suitable means (e.g., by means of firmware).

[0105] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0107] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0108] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0109] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0110] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0111] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0112] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An image processing method, characterized in that, The method includes: Continuously monitor whether a new candidate image frame is received through any image transmission channel, or whether the processing of the previous target image frame has ended; If a new candidate image frame is received through any image transmission channel, or if the processing of the previous target image frame is completed, the processing time and remaining time of the candidate image frame are determined. Wherein, the candidate image frame is an unprocessed image frame transmitted through at least two image transmission channels, the processing time is used to represent the time required to process the unprocessed pixels in the candidate image frame, and the remaining time is used to represent the time from the current moment to the receipt of the next candidate image frame transmitted through the image transmission channel corresponding to the candidate image frame, and the difference between the remaining time and the processing time of the candidate image frame. From candidate image frames with non-zero processing time, determine the candidate image frame with the smallest remaining time as the target image frame; if there are multiple candidate image frames with the smallest remaining time, determine the candidate image frame with non-zero processing time and the smallest remaining time from candidate image frames with equal remaining time as the target image frame; if there are multiple candidate image frames with non-zero processing time and the smallest remaining time, randomly determine the candidate image frame as the target image frame from candidate image frames with equal remaining time and equal processing time. The target image frame is processed, and the processing time and remaining time of the candidate image frame and the target image frame are updated according to the processing time of the target image frame; wherein, the processing time is the time from the start of processing the target image frame to the current time.

2. The method according to claim 1, characterized in that, Based on the processing time of the target image frame, the processing time and remaining time of the candidate image frame and the target image frame are updated, including: The difference between the remaining duration of the candidate image frame and the processed duration is used as the updated remaining duration of the candidate image frame, while the processing duration of the candidate image frame remains unchanged. The difference between the processing time of the target image frame and the already processed time is used as the updated processing time of the target image frame, while the remaining time of the target image frame remains unchanged.

3. The method according to claim 1, characterized in that, If a new candidate image frame is received via any image transmission channel, the processing time and remaining time of the candidate image frame are determined, including: The time required to process all pixels of the new candidate image frame is determined based on the ratio of the resolution of the new candidate image frame to the number of pixels processed by the image processing device per unit time. The processing time of the new candidate image frame is determined based on the time required to process all pixels of the new candidate image frame. Based on the frame rate of the video to which the new candidate image frame belongs, determine the duration occupied by the new candidate image frame, and use the difference between the duration occupied by the new candidate image frame and the processing time as the remaining duration of the new candidate image frame.

4. The method according to claim 1, characterized in that, If the candidate image frame is not a new candidate image frame, then the processing time and remaining time of the candidate image frame are determined, including: For each candidate image frame, the updated processing time and remaining time will be used as the processing time and remaining time of the candidate image frame, based on the processing time of the previous target image frame. If the processing of the previous target image frame is completed, the updated processing time of the previous target image frame is zero, the processing time of the remaining candidate image frames remains unchanged, and the remaining time of each candidate image frame is updated according to the difference between the remaining time of the candidate image frame and the processing time of the previous target image frame.

5. The method according to claim 1, characterized in that, The method further includes: If a new candidate image frame is received through any image transmission channel, and the processing of the previous target image frame has not been completed, the processed pixel row of the previous target image frame is recorded, and the previous target image frame is used as a candidate image frame. When the candidate image frame is determined as the target image frame again for processing, processing starts from the next row of processed pixel rows. The previous target image frame is the image frame that was being processed before the new candidate image frame was received.

6. An image processing apparatus, characterized in that, The device includes: The duration determination module is used to continuously monitor whether a new candidate image frame is received through any image transmission channel, or whether the processing of the previous target image frame has ended; if a new candidate image frame is received through any image transmission channel, or the processing of the previous target image frame has ended, the processing duration and remaining duration of the candidate image frame are determined. Wherein, the candidate image frame is an unprocessed image frame transmitted through at least two image transmission channels, the processing time is used to represent the time required to process the unprocessed pixels in the candidate image frame, and the remaining time is used to represent the time from the current moment to the receipt of the next candidate image frame transmitted through the image transmission channel corresponding to the candidate image frame, and the difference between the remaining time and the processing time of the candidate image frame. The target image frame determination module is used to determine the candidate image frame with the smallest remaining time from the candidate image frames with non-zero processing time as the target image frame; if there are multiple candidate image frames with the smallest remaining time, the candidate image frame with the smallest processing time with non-zero processing time is determined from the candidate image frames with equal remaining time as the target image frame; if there are multiple candidate image frames with the smallest processing time with non-zero processing time, a candidate image frame is randomly determined from the candidate image frames with equal remaining time and equal processing time as the target image frame. The duration update module is used to process the target image frame and update the processing duration and remaining duration of the candidate image frame and the target image frame according to the processing duration of the target image frame; wherein, the processing duration is the duration from the start of processing the target image frame to the current time.

7. An image processing electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the image processing method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the image processing method according to any one of claims 1-5.

Images