A data stream transmission channel switching method, device, equipment and storage medium

Abstract

Embodiments of the present application disclose a data stream transmission channel switching method, device, equipment and storage medium. The method comprises: comparing the similarity of a current image frame in a current channel with a candidate image frame in a target channel which is pre-stored; determining a target image frame from the candidate image frame according to the similarity comparison result; and switching the current channel to the target channel from the target image frame, so as to output data stream through the target channel. The technical scheme compares the current image frame with the candidate image frame, determines the candidate image frame with the highest similarity to the current image frame as the target image frame, realizes smooth switching between data streams, solves the problem of split feeling of a displayed image caused by data stream switching, and improves the image display effect.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data stream transmission channel switching method, apparatus, device, and storage medium. Background Technology

[0002] The application of image acquisition devices is becoming increasingly widespread, with at least two image acquisition devices potentially installed in products such as mobile phones and surveillance systems. In products equipped with at least two image acquisition devices, the parameters of each device often differ, and the overall image display effect is improved through the linkage between the at least two image acquisition devices.

[0003] In existing solutions, when switching between images from at least two image acquisition devices, the displayed image appears disjointed, the switching process is not smooth enough, and the display effect is poor. Summary of the Invention

[0004] This application provides a data stream transmission channel switching method, apparatus, device, and storage medium to achieve smooth switching of images between image acquisition devices and reduce the sense of disjointedness.

[0005] In one embodiment, this application provides a data stream transmission channel switching method, including:

[0006] Compare the current image frame in the current channel with the candidate image frames in the pre-cached target channel;

[0007] Based on the similarity comparison results, the target image frame is determined from the candidate image frames;

[0008] Starting from the target image frame, the current channel is switched to the target channel so that data stream output is performed through the target channel.

[0009] In another embodiment, this application provides a data stream transmission channel switching device, comprising:

[0010] The similarity comparison module is used to compare the current image frame in the current channel with the candidate image frames in the pre-cached target channel.

[0011] The target image frame determination module is used to determine the target image frame from the candidate image frames based on the similarity comparison results;

[0012] The data stream output module is used to switch the current channel to the target channel starting from the target image frame, so as to output the data stream through the target channel.

[0013] In yet another embodiment, this application provides an electronic device, including:

[0014] One or more processors;

[0015] Storage device for storing one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the data stream transmission channel switching method described in any of the embodiments of this application.

[0017] In one embodiment, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data stream transmission channel switching method described in any of the embodiments of this application.

[0018] This application provides a data stream transmission channel switching method, apparatus, device, and storage medium, including: comparing the similarity of the current image frame in the current channel with candidate image frames in a pre-cached target channel; determining the target image frame from the candidate image frames based on the similarity comparison result; and switching the current channel to the target channel starting from the target image frame to output the data stream through the target channel. This technical solution, by comparing the current image frame and candidate image frames, determines the candidate image frame with the highest similarity to the current image frame as the target image frame, achieving smooth switching between data streams, solving the problem of disjointed images caused by image acquisition device switching, and improving image display quality. Attached Figure Description

[0019] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 A flowchart illustrating a data stream transmission channel switching method provided in one embodiment of this application;

[0021] Figure 2 A flowchart illustrating a data stream transmission channel switching method provided in another embodiment of this application;

[0022] Figure 3 A flowchart illustrating a data stream transmission channel switching method provided in another embodiment of this application;

[0023] Figure 4 This is a schematic diagram of candidate image frame selection provided in another embodiment of this application;

[0024] Figure 5 This is a flowchart illustrating the specific implementation of a data stream transmission channel switching method according to an embodiment of this application.

[0025] Figure 6 This is a schematic diagram of a candidate image frame provided in one embodiment of this application;

[0026] Figure 7 This is a schematic diagram of pixel GMS mapping relationship provided in one embodiment of this application;

[0027] Figure 8 This is a schematic diagram of the structure of a data stream transmission channel switching device provided in one embodiment of this application;

[0028] Figure 9 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0029] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. Furthermore, the embodiments and features described herein can be combined with each other unless otherwise specified. It should also be noted that, for ease of description, only the parts relevant to the present application are shown in the accompanying drawings, not the entire structure.

[0030] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. The process can be terminated when its operation is complete, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0031] It should be noted that the concepts of "first" and "second" mentioned in the embodiments of this application are only used to distinguish different devices, modules, units or other objects, and are not used to limit the order or interdependence of the functions performed by these devices, modules, units or other objects.

[0032] To better understand the embodiments of this application, the relevant technologies are described below.

[0033] Figure 1 This is a flowchart illustrating a data stream transmission channel switching method according to one embodiment of this application. This embodiment is applicable to situations involving data stream transmission channel switching. Specifically, the data stream transmission channel switching method can be executed by a data stream transmission channel switching device, which can be implemented through software and / or hardware and integrated into an electronic device. Further, the electronic device includes, but is not limited to, electronic devices such as surveillance cameras, laptops, smartphones, and servers.

[0034] like Figure 1 As shown, the method specifically includes the following steps:

[0035] S110, compare the similarity of the current image frame in the current channel with the candidate image frames in the pre-cached target channel.

[0036] In this system, image frames acquired by an image acquisition device can be transmitted through a data stream transmission channel and displayed on a display device. In specific application scenarios, at least two image acquisition devices may be involved, each transmitting image frames through its own connected data stream transmission channel. These at least two image acquisition devices have different magnifications, resulting in image frames with different magnification levels. When the display device displays the data stream transmitted from one channel, and the magnification of the current image displayed on the display device is insufficient, channel switching is required to transmit and display image frames acquired by other image acquisition devices through other channels. The current channel can be the channel transmitting the image currently displayed on the display device, and the target channel can be the data stream transmission channel after the channel switch.

[0037] In this embodiment, the current image frame can be the image frame transmitted in the current channel at the time point when channel switching is determined to be required. Candidate image frames can be selected from image frames acquired by the target image acquisition device connected to the target channel. The selection of candidate image frames can be based on the current time, for example, selecting the image frame acquired by the target image acquisition device at the current time, as well as image frames located before and / or after that image frame. Alternatively, selection can be based on the time point required for channel switching. The number of candidate image frames is not limited and can be determined according to the actual situation.

[0038] Specifically, different image acquisition devices may be affected by factors such as hardware, clock source, and acquisition cycle, resulting in different image frames acquired at the same time. If channel switching is performed directly when needed, it may lead to significant abrupt changes in the displayed image, creating a fragmented visual effect. In this embodiment, the current image frame is compared with candidate image frames to determine their similarity and differences, facilitating the subsequent determination of the appropriate time for channel switching. Similarity comparison methods include, but are not limited to, pixel comparison, centroid comparison, projection comparison, block comparison, and GMS (Gradient Magnitude Similarity).

[0039] In the embodiments of this application, different image acquisition devices may have spatial differences, and the images acquired by each may have parallax. Affine transformations can be performed spatially to eliminate or reduce the impact of parallax between the image in the current channel and the image in the target channel on the accuracy of similarity comparison. The similarity comparison is then performed after the image has undergone the affine transformation. Specifically, the current image frame can be affine-transformed and then compared with candidate image frames, or the current image frame can be compared with the affine-transformed candidate image frames.

[0040] S120, Based on the similarity comparison results, determine the target image frame from the candidate image frames.

[0041] The target image frame can be the candidate image frame with the highest similarity to the current image frame. If at least two candidate image frames have the same similarity comparison result with the current image frame and have the highest similarity, then any one of the at least two candidate image frames can be the target image frame. Alternatively, the candidate image frame whose acquisition time point is closest to the time point when channel switching is required can be used as the target image frame. The target image frame can also be a candidate image frame selected from the candidate image frames whose similarity to the current image frame reaches a preset similarity threshold. The target image frame can also be a candidate image frame selected from the candidate image frames whose similarity to the current image frame reaches a preset similarity threshold and has the highest similarity.

[0042] Specifically, based on the similarity comparison results, the candidate image frame with the highest similarity is selected as the target image frame. This facilitates switching based on the position of the target image frame when switching channels, so that the image frame displayed on the display device after channel switching is as similar as possible to the current image frame displayed on the display device before channel switching, thereby reducing the visual discontinuity caused by channel switching and achieving smooth channel switching.

[0043] S130, starting from the target image frame, switch the current channel to the target channel so as to output the data stream through the target channel.

[0044] Specifically, after determining the target image frame, the data stream transmission channel is switched. The last image frame displayed before the channel switch is the current image frame. Starting from the target image frame, the system switches to the target channel so that the display device displays the target image frame transmitted through the target channel, and subsequently displays image frames acquired by the image acquisition device connected to the target channel.

[0045] The beneficial effect of the above scheme is that the display device displays the current image frame before the channel switching and the display device displays the target image frame after the channel switching. Since the target image frame and the current image frame have the greatest similarity, the difference between the images seen by the user after the channel switching is minimal, and the visual image jump is minimal, thus achieving a smooth switching.

[0046] In this embodiment of the application, for example, the maximum magnification of the image transmitted to the display device via channel one is 1.5x. If the user needs to view an image with a higher magnification, and the image transmitted to the display device via channel two has a magnification greater than 10x, then channel switching is performed. If the user needs to view an image with a magnification less than 1.5x, channel switching can be performed again to switch back to channel one, so that the magnification of the image displayed on the display device is less than 1.5x.

[0047] This application provides a data stream transmission channel switching method, including: comparing the similarity of the current image frame in the current channel with candidate image frames in a pre-cached target channel; determining the target image frame from the candidate image frames based on the similarity comparison result; and switching the current channel to the target channel starting from the target image frame to output the data stream through the target channel. This technical solution, by comparing the current image frame and candidate image frames, determines the candidate image frame with the highest similarity to the current image frame as the target image frame, achieving smooth switching between data streams, solving the problem of disjointed images caused by image acquisition device switching, and improving image display quality.

[0048] Figure 2 This is a flowchart illustrating a data stream transmission channel switching method according to another embodiment of this application. This embodiment optimizes the above embodiments and provides a detailed description of the data stream transmission channel switching. It should be noted that technical details not described in detail in this embodiment can be found in any of the above embodiments.

[0049] Specifically, such as Figure 2 As shown, the method specifically includes the following steps:

[0050] S210: Select a first preset number of candidate image frames from the image frames acquired by the image acquisition device corresponding to the target channel, with the shortest interval between the acquisition time and the channel switching time point.

[0051] The image acquisition device can be a camera, such as a camera used in surveillance scenarios. The first preset number can be determined according to the actual situation, and this application embodiment does not limit this; for example, the first preset number can be 5. The channel switching time point can be the time point when it is detected that a channel switching is required. Specifically, when transmitting data stream through the current channel, the current magnification of the image frame displayed on the display device is detected in real time. The current magnification is compared with the maximum magnification of the current image acquisition device connected to the current channel. If the current magnification is less than the maximum magnification, no channel switching is required. If the current magnification is equal to the maximum magnification and there is still a need to magnify the currently displayed image frame, a channel switching is triggered, and the current time point is determined as the channel switching time point.

[0052] Specifically, image frames acquired near the channel switching time point are likely to be more similar to the current image frame. A first preset number of candidate image frames, whose acquisition time is closest to the channel switching time point, can be selected to quickly identify candidate image frames with a high degree of similarity to the current image frame.

[0053] For example, the image acquisition device can have a fixed acquisition frequency, such as 25 frames per second. Assuming the first preset number is 5, if the image acquisition device acquires one frame at the channel switching time point, the candidate image frames can be the image frames acquired by the image acquisition device at the channel switching time point, 40 milliseconds before the channel switching time point, 80 milliseconds before the channel switching time point, 40 milliseconds after the channel switching time point, and 80 milliseconds after the channel switching time point. If the image acquisition device does not acquire an image at the channel switching time point, the candidate image frames can be the four image frames acquired by the image acquisition device within 80 milliseconds before and 80 milliseconds after the channel switching time point, as well as the image frame whose acquisition time point is closest to the channel switching time point among the remaining images.

[0054] S220, the candidate image frames are cached.

[0055] The cache can be a data storage device, such as storing candidate image frames in the central processing unit (CPU) of electronic devices like mobile phones, computers, and surveillance equipment. For example, storing candidate image frames in the CPU is much faster than storing them in memory, which facilitates subsequent data processing.

[0056] Specifically, in practical applications, the solution can be as follows: the current channel transmits the data stream to the display for playback and stores the data stream; other channels only cache the two most recently acquired image frames from the image acquisition device connected to that channel, without storing or playing image data acquired by other image acquisition devices connected to that channel. Without channel switching, if an image acquisition device connected to another channel acquires a newest image frame, it caches that newest image frame, deletes the earliest acquired image frame, and keeps the two most recently acquired image frames in the cache. If a channel switching time point is reached, the image frames acquired by the image acquisition device connected to the target channel at that time point, along with the two consecutive image frames acquired after the channel switching time point, are cached, resulting in a total of five cached image frames as candidate image frames.

[0057] S230, sort the candidate image frames according to the order of acquisition time to form a queue; starting from the first acquired candidate image frame, traverse the candidate image frames in the queue and compare their similarity with the current image frame.

[0058] The queue can be a linear list, allowing only deletion at the front and insertion at the back. The queue can contain sorted candidate image frames; for example, if there are five candidate image frames, they are placed in the queue sequentially according to their acquisition time. Specifically, the first acquired candidate image frame is stored in the queue first, and the last acquired candidate image frame is stored last. To reduce data stream switching time, similarity comparison can begin with the first acquired candidate image frame in the queue, continuing until all candidate image frames have been traversed.

[0059] In this embodiment of the application, traversing the candidate image frames in the queue and comparing them with the current image frame includes:

[0060] If the process starts from the first collected candidate image frame, then the preceding image frame of the candidate image frame with the highest similarity obtained in the current traversal will be deleted.

[0061] If the process is to traverse from the last collected candidate image frame, then after the traversal is completed, the preceding image frame of the candidate image frame with the highest similarity will be deleted.

[0062] Wherein, the preceding image frame is a candidate image frame whose acquisition time is earlier than the acquisition time of the candidate image frame corresponding to the maximum similarity. Wherein, the clearing may be deleting the candidate image frame from the cache. Specifically, the candidate image frames are stored in a queue. During the process of comparing the candidate image frame with the current image frame, if the similarity between the earliest acquired candidate image frame and the current image frame is relatively low, the candidate image frame has no use value anymore and can be cleared. The above solution can clear unnecessary candidate image frames in time, thereby releasing the cache and further improving the processing efficiency of the processor.

[0063] For example, for candidate image frames (T-2), (T-1), T, (T+1) and (T+2) arranged according to the acquisition order, each is subjected to similarity comparison with the current image frame. If traversal is performed starting from the earliest acquired candidate image frame, the similarity between the (T-2) frame and the current image frame is compared first, which is recorded as S1, and then the similarity between the (T-1) frame and the current image frame is compared, which is recorded as S2. If S1 < S2, the candidate image frame of (T-2) is cleared to release the cache. If S1 > S2, the (T-2) frame and the (T-1) frame are retained. Then the similarity between the T frame and the current image frame is compared, which is recorded as S3. If S3 is the largest among S1, S2 and S3, the (T-2) frame and the (T-1) frame are cleared to release the cache. If S2 is the largest among S1, S2 and S3, the (T-2) frame is cleared to release the cache, and both the (T-1) frame and the T frame are retained, so that when the (T-1) frame is the target image frame, the T frame can continue to be displayed after switching channels from the (T-1) frame. If S1 is the largest among S1, S2 and S3, the (T-2) frame, the (T-1) frame and the T frame are all retained, so that when the (T-2) frame is the target image frame, the (T-1) frame and the T frame can continue to be displayed after switching channels from the (T-2) frame. And so on. When the similarity between an earlier acquired candidate image frame and the current image frame is higher than the similarity between a later acquired candidate image frame and the current image frame, after channel switching is started from the earlier acquired candidate image frame, the later acquired candidate image frames still need to be displayed continuously. Therefore, it is not appropriate to delete the later acquired candidate image frames, but both candidate image frames are retained to ensure normal and smooth image display.

[0064] If we iterate through the last acquired candidate image frames, first compare the similarity between frame (T+2) and the current image frame, which is S1; then compare the similarity between frame (T+1) and the current image frame, which is S2; then compare the similarity between frame T and the current image frame, which is S3; then compare the similarity between frame (T-1) and the current image frame, which is S4; then compare the similarity between frame (T-2) and the current image frame, which is S5. After all comparisons are completed, we get S1, S2, S3, S4, and S5. If S5 is the largest, then frames (T-2), (T-1), T, and (T+1) are cleared. If S3 is the largest, then frames (T-2) and (T-1) are cleared.

[0065] In this scheme, candidate image frames that meet the clearing conditions are cleared. The clearing process conforms to the working logic of the queue and reduces cache usage.

[0066] S240, Based on the similarity comparison results, determine the target image frame from the candidate image frames.

[0067] S250, starting from the target image frame, switch the current channel to the target channel so as to output the data stream through the target channel.

[0068] Specifically, after the data stream is output through the target channel, the image data acquired by the image acquisition device connected to the target channel is played and stored. At this time, the target channel is the current channel in this embodiment. For the image data acquired by the image acquisition device connected to other channels, only the two most recently acquired image frames are cached.

[0069] The technical solution in this application embodiment caches a first preset number of candidate image frames whose acquisition time is closest to the channel switching time point, and compares them with the current image frame in turn. Since the acquisition time of the candidate image frames is close to the channel switching time point, they are most likely to be similar to the current image frame. This allows for faster and more efficient selection of the target image frame with the highest similarity from the candidate image frames, and then the smooth switching of the data stream transmission channel is completed based on the target image frame to improve the image display effect during channel switching.

[0070] Figure 3 This is a flowchart illustrating a data stream transmission channel switching method according to another embodiment of this application. This embodiment is a further optimization of the above embodiments; details not described in detail in this embodiment are provided in the above embodiments. See also... Figure 3 The image acquisition method provided in this application embodiment may include:

[0071] S310, if there is historically stored location information of the target image frame, then according to the location information, a second preset number of image frames are selected from the image frames of the target channel as candidate image frames.

[0072] In this embodiment of the application, the location information can be the position of the target image frame within the candidate image frames obtained during the last channel switching process. For example, as shown... Figure 4 As shown, assume that five candidate image frames were identified during the previous channel switching process, namely frames (T-2), (T-1), T, (T+1), and (T+2). If the fourth candidate image frame has the highest similarity to the current image frame, that is, frame (T+1) has the highest similarity to the current image frame, then the fourth frame is taken as the target image frame. At this time, the position of the target image frame is recorded, which is the fourth frame among the five buffered candidate image frames, that is, the frame following the candidate image frame acquired at the channel switching time point.

[0073] Since the location information of the target image frame from the previous channel switch has been stored, it serves as a reference for determining the target image frame in the current channel switch. This means that the candidate image frame at that location is most likely to be the image frame with the highest similarity to the current image frame. Therefore, candidate image frames can be selected from the image frames of the target channel based on the location information. For example, the image frame pointed to by the location information, as well as the image frames before and after that image frame, can be selected as candidate image frames. In other words, the second preset number of image frames closest to the acquisition time of that image frame can be selected as candidate image frames. For instance, during the previous channel switch, the candidate image frames were frames (T-2), (T-1), T, (T+1), and (T+2). The location information of the target image frame during the previous channel switch was frame 4, which is frame (T+1). Assuming the second preset number is 3, then the image frames before and after (T+1), frame T and frame (T+2), as well as frame (T+1), can be selected as candidate image frames. The second preset number of candidate image frames is not specifically limited and can be other values, such as 5. In this case, the selected candidate image frames are (T-1) frames, T frames, (T+1) frames, (T+2) frames, and (T+3) frames. In this embodiment, since the position information of the target image frame in the previous channel switching is already available as a reference, that is, the position information of the image frame most likely to have the greatest similarity to the current image frame is provided as a reference, the target image frame can be located more quickly during the current channel switching process. This reduces the number of candidate image frames selected, i.e., the second preset number is less than the first preset number, thereby reducing the amount of processing required for one-to-one similarity comparison between the current image frame and the candidate image frames and improving processing efficiency.

[0074] S320, the candidate image frames are cached.

[0075] S330, sort the candidate image frames according to the order of acquisition time to form a queue; starting from the first acquired candidate image frame, traverse the candidate image frames in the queue and compare their similarity with the current image frame.

[0076] S340: Determine if there is a candidate image frame that has the highest similarity to the current image frame and reaches the similarity threshold. If so, proceed to S350; otherwise, proceed to S360.

[0077] The similarity threshold can be set according to the actual situation, such as the similarity threshold between visually different and non-visually different values.

[0078] Specifically, after comparing the similarity of the current image frame with that of the second preset number of candidate image frames, the candidate image frame with the highest similarity to the current image frame is obtained. However, although the similarity between this candidate image frame and the current image frame is the highest, the similarity may not be high enough to reach the similarity threshold. If the frame is switched directly, it will still cause a sense of disjointedness in the image. In this embodiment, after determining the candidate image frame with the highest similarity to the current image frame, it is further determined whether the maximum similarity is greater than the similarity threshold to further determine the degree of similarity between the candidate image frame and the current image frame.

[0079] S350, the candidate image frame is selected as the target image frame.

[0080] Specifically, if there is a candidate image frame that has the highest similarity to the current image frame and reaches the similarity threshold, then the similarity between the candidate image frame and the current image frame has reached the preset requirement, and therefore the candidate image frame can be used as the target image frame.

[0081] S360: From the image frames acquired by the image acquisition device corresponding to the target channel, select the first preset number of candidate image frames with the shortest interval between the acquisition time and the channel switching time point, compare them with the current image frame to determine the target image frame.

[0082] Specifically, since the similarity between all candidate image frames and the current image frame does not reach the similarity threshold, directly switching channels may result in a visual disjointed feeling, indicating that the second preset number of candidate image frames are not suitable as target image frames for channel switching. In this embodiment, the first preset number of candidate image frames with the shortest interval between the acquisition time and the channel switching time point are selected, and their similarity is re-compared with the current image frame.

[0083] Specifically, such as Figure 4As shown, assuming the candidate image frames in the current channel switching are frame T, frame (T+1), and frame (T+2), the similarity is compared with the current image frame to determine that the frame with the highest similarity is frame (T+1). If the similarity of this candidate image frame does not reach the similarity threshold, the first preset number of candidate image frames with the shortest interval between the acquisition time and the channel switching time point are selected, namely frames (T-2), (T-1), T, (T+1), and (T+2), and the similarity is compared with the current image frame one by one again. The candidate image frame with the highest similarity is taken as the target image frame.

[0084] The beneficial effect of the above scheme is that although the location of the target image frame with the highest similarity to the current image frame was marked in the previous channel switch, the acquisition period may have a certain deviation due to the change in the acquisition parameters of the two image acquisition devices, resulting in asynchronous image acquisition. This can lead to low similarity between image frames acquired at the same time, or the image frame in the target channel determined according to historical location information may not be very similar to the current image frame in the current channel. Therefore, the above scheme can solve the problem of inaccurate target image frame selection due to reliance on historical experience, and improves the accuracy of target image frame selection by performing dual judgment.

[0085] S370, Starting from the target image frame, switch the current channel to the target channel so as to output the data stream through the target channel.

[0086] S380, update the position information of the historically stored target image frames using the position information of the target image frames.

[0087] Specifically, if the second preset number of candidate image frames obtained based on the location information of the target image frame in historical storage have low similarity and are no longer of reference value, their location information can be deleted. Updating the location information of the target image frame can serve as a reference for the selection of candidate image frames during the next data stream transmission channel switch.

[0088] The technical solution in this application embodiment determines a second preset number of candidate image frames based on the location information of historically stored target image frames. After comparing the similarity between the current image frame and the candidate image frames, it is determined whether the candidate image frames reach a similarity threshold. If they do, the candidate image frame with the highest similarity is determined as the target image frame; otherwise, a first preset number of candidate image frames are selected and compared with the current image frame to determine the candidate image frame with the highest similarity as the target image frame, and the location information of the historically stored target image frames is updated. By using the existing location information of historically stored target image frames and determining a second preset number of candidate image frames, and by imposing requirements on the similarity of the target image frames, the switching quality is ensured while minimizing channel switching time.

[0089] Figure 5 is a specific implementation flow chart of the data flow transmission channel switching method provided by the embodiments of the present application. The embodiments of the present application provide a detailed description of a specific implementation of the data flow transmission channel switching method, and for details not described in detail in the embodiments of the present application, reference is made to the above embodiments. For example Figure 5 shown in the drawings, the method may comprise:

[0090] taking a monitoring system with dual cameras as an example, as the display screen is zoomed in and out, it may be necessary to switch cameras to obtain a better display effect. The magnification switching point between the two cameras is fixed, and the magnification switching point can be calculated by combining the hardware parameters of the two cameras.

[0091] in the embodiments of the present application, all hardware serving for data acquisition and transmission of the two channels should be consistent, the tolerance of the hardware should be minimized, clock synchronization should be maintained, data should be acquired synchronously, and internal image processing algorithms and procedures should be kept consistent, so as to ensure the fluency of channel switching.

[0092] before channel 2 is switched, it caches current images, with a maximum cache of 2 frames. If no switching operation is performed, the cached frames are replaced, and the 2 latest acquired image frames are cached. The 2 image frames cached by channel 2 are used for subsequent similarity comparison. The current magnification of image frames in channel 1 is detected in real time. If the current magnification has reached the maximum magnification corresponding to channel 1 and there is still a requirement for magnification amplification, a switching operation is triggered.

[0093] there may be differences in spatial positions between images acquired by channel 1 and channel 2, so affine transformation in space is required, and the spatial affine transformation will not be repeated herein. The similarity comparison process can be divided into two cases. The first case: there is no position information of the target image frame stored in history, the current image frame is frame-matched with candidate image frames cached in channel 2, and the candidate image frames are 5 frames before and after the corresponding frame. The corresponding frame is an image frame acquired at the same time as the current image frame in channel 2. The matching similarity between the (T-2)th frame and the current image frame is preferentially calculated to obtain a similarity value S1, then the matching similarity between the (T-1)th frame and the current image frame is calculated to obtain a similarity value S2. If S1 < S2, the (T-2)th frame in the cache is cleared and the cache is updated. Then the matching similarity between the T-th frame and the current image frame is calculated to obtain a similarity value S3, and the calculation is performed sequentially until all 5 frames are processed, and the candidate image frame with the maximum similarity is taken as the switching frame. The second case: there is position information of a target image frame stored in history, that is, there is a reference frame, the current image frame is frame-matched with candidate image frames cached in channel 2, and the candidate image frames are the reference frame, the previous frame of the reference frame and the next frame of the reference frame. The calculation method is consistent with that of the first case. If the similarity of all frames in the second case is less than the similarity threshold, all steps of the first case are then performed.

[0094] Here, the corresponding frame can be the image frame acquired by channel two at the current image frame acquisition time, such as... Figure 6 As shown, taking 5 candidate image frames as an example, the acquisition time of the corresponding frame is the same as the acquisition time of the current image frame. Frame (T-1) is the frame before the corresponding frame, frame (T-2) is the frame before frame (T-1), frame (T+1) is the frame after the corresponding frame, and frame (T+2) is the two frames after the corresponding frame. Updating the cache can be done by clearing the image frame with the lowest similarity and the first cached image frame from channel two.

[0095] For example, the GMS similarity comparison method can be used to compare the similarity between the current image frame and the candidate image frames. Motion detection can be used to obtain the motion region of the current image frame, and the motion region is preferentially selected as feature points. The number of feature points can be up to five; this embodiment does not limit the number of feature points, and obtaining the motion region through motion detection is a conventional technique, which will not be elaborated upon here. After determining the feature points, the feature points at the same location in the current image frame and the candidate image frames are compared. Specifically, the similarity between the macroblocks where the feature points are located can be compared. The size of the macroblock can be determined according to the actual situation, for example, 16*16. Figure 7 As shown, the pixel-based GMS mapping relationship is calculated for feature points at the same location. The calculation process is as follows:

[0096] Define gradient template:

[0097]

[0098] Among them, h x For the horizontal gradient template, h y This is a vertical gradient template. It defines the gradient value m of a pixel i within the feature points of the current image frame. c (i) is defined as follows:

[0099]

[0100] In the formula, c is a 3×3 matrix centered at pixel i, and the matrix content is the pixel value. If the boundary of c is not on the feature point, the boundary point is taken as the value of the adjacent pixel in c.

[0101] Define the gradient value of the matching pixel i corresponding to the candidate image frame as m. r (i) is defined as follows:

[0102]

[0103] In the formula, r is a 3×3 matrix centered at pixel i, and the matrix content is the pixel value. If the boundary of r is not on the feature point, the boundary point is taken as the value of the adjacent pixel in r.

[0104] The gradient similarity of image pixel i is defined as GMS(i), as follows:

[0105]

[0106] In the formula, c is a constant set to avoid the numerator and denominator being zero, and its value can be 170. Since the gradient template size is 3×3, the gradient of the image boundary pixels is not calculated, and the GMS value of the adjacent pixels is directly used as the gradient of the image boundary pixels. When the image pixel is not a boundary point, the GMS value of that pixel is calculated.

[0107] Define macroblock similarity (MBGMS).

[0108]

[0109] In the formula, M×N represents the size of the macroblock. The average macroblock similarity is used as the similarity between the current image frame and the candidate image frame.

[0110] After calculating the target image frame with the highest similarity, the monitoring device switches to display the image transmitted in channel two, starting from the target image frame. The position information of the target image frame during this channel switch is recorded to provide a reference for the next switch.

[0111] Optionally, start monitoring the channel 2 ratio, buffer the frames in channel 1, and if channel 2 reaches the ratio switching point, replace channel 1 and channel 2, repeat the above steps in this embodiment to complete the channel switching.

[0112] Figure 8 This is a schematic diagram of a data stream transmission channel switching device provided in one embodiment of this application. The data stream transmission channel switching device provided in this embodiment of the application includes:

[0113] The similarity comparison module 810 is used to compare the current image frame in the current channel with the candidate image frames in the pre-cached target channel.

[0114] The target image frame determination module 820 is used to determine the target image frame from the candidate image frames based on the similarity comparison results;

[0115] The data stream output module 830 is used to switch the current channel to the target channel starting from the target image frame, so as to output the data stream through the target channel.

[0116] Optionally, based on the above embodiments, the device further includes:

[0117] The first selection module is used to select a first preset number of candidate image frames from the image frames acquired by the image acquisition device corresponding to the target channel, which have the shortest interval between the acquisition time and the channel switching time point.

[0118] An image frame caching module is used to cache the candidate image frames.

[0119] Optionally, the device further includes:

[0120] The second selection module is used to select a second preset number of image frames as candidate image frames from the image frames of the target channel according to the location information if there is historically stored location information of the target image frames.

[0121] An image frame caching module is used to cache the candidate image frames.

[0122] Optionally, the similarity comparison module 810 includes:

[0123] The queue generation unit is used to sort candidate image frames according to the order of their acquisition time to form a queue;

[0124] The similarity comparison unit is used to traverse the candidate image frames in the queue and compare them with the current image frame.

[0125] Optionally, the similarity comparison unit may also include:

[0126] The candidate image frame clearing subunit is used to delete the preceding image frame of the candidate image frame with the highest similarity obtained during the current traversal if the traversal starts from the first acquired candidate image frame; and to delete the preceding image frame of the candidate image frame with the highest similarity after the traversal ends if the traversal starts from the last acquired candidate image frame. The preceding image frame is the candidate image frame whose acquisition time is earlier than the acquisition time of the candidate image frame with the highest similarity.

[0127] Optionally, the target image frame determination module 820 includes:

[0128] The first determining unit is used to determine the target image frame if there is a candidate image frame that has the highest similarity to the current image frame and reaches the similarity threshold.

[0129] The second determining unit is used to otherwise select a first preset number of candidate image frames with the shortest interval between the acquisition time and the channel switching time from the image frames acquired by the image acquisition device corresponding to the target channel, and compare them with the current image frame to determine the target image frame.

[0130] Optionally, the device further includes:

[0131] The location information update module is used to update the location information of historically stored target image frames using the location information of the target image frame.

[0132] The data stream transmission channel switching device provided in this application embodiment can be used to execute the data stream transmission channel switching method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0133] Figure 9 A schematic diagram of an electronic device 10 that can be used to implement embodiments of this application is shown. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 10 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, user equipment, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the application described and / or claimed herein.

[0134] like Figure 9 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0135] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks and wireless networks.

[0136] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data stream transmission channel switching methods.

[0137] In some embodiments, the data stream transmission channel switching method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data stream transmission channel switching method by any other suitable means (e.g., by means of firmware).

[0138] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0139] Computer programs used to implement the data stream transmission channel switching method of this application can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer programs can be executed entirely on the machine, partially on the machine, or as a standalone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0140] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0141] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device 10, which includes: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device 10. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0142] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0143] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0144] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0145] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for switching data stream transmission channels, characterized in that, The method includes: The current image frame in the current channel is compared with the candidate image frames in the pre-cached target channel based on similarity; the image acquisition device connected to the current channel and the image acquisition device connected to the target channel have different magnification. Based on the similarity comparison results, the target image frame is determined from the candidate image frames; Starting from the target image frame, the current channel is switched to the target channel so that data stream output is performed through the target channel.

2. The method according to claim 1, characterized in that, The method further includes: From the image frames acquired by the image acquisition device corresponding to the target channel, select the first preset number of candidate image frames with the shortest interval between the acquisition time and the channel switching time. The candidate image frames are cached.

3. The method according to claim 1, characterized in that, The method further includes: If there is location information of the target image frame stored in history, then a second preset number of image frames are selected from the image frames of the target channel as candidate image frames according to the location information. The candidate image frames are cached.

4. The method according to any one of claims 1-3, characterized in that, The current image frame in the current channel is compared with the candidate image frames in the pre-cached target channel, including: Candidate image frames are sorted according to the order of their acquisition time to form a queue; The candidate image frames in the queue are traversed, and their similarity is compared with the current image frame.

5. The method according to claim 4, characterized in that, The process involves iterating through the candidate image frames in the queue and comparing them with the current image frame, including: If the process starts from the first collected candidate image frame, then the preceding image frame of the candidate image frame with the highest similarity obtained in the current traversal will be deleted. If the process is to traverse from the last collected candidate image frame, then after the traversal is completed, the preceding image frame of the candidate image frame with the highest similarity will be deleted. Among them, the preceding image frame is the candidate image frame whose acquisition time is earlier than the acquisition time of the candidate image frame corresponding to the maximum similarity.

6. The method according to claim 3, characterized in that, Based on the similarity comparison results, the target image frame is determined from the candidate image frames, including: If there is a candidate image frame that has the highest similarity to the current image frame and reaches the similarity threshold, then the candidate image frame is taken as the target image frame. Otherwise, from the image frames acquired by the image acquisition device corresponding to the target channel, select the first preset number of candidate image frames with the shortest interval between the acquisition time and the channel switching time point, compare them with the current image frame to determine the target image frame.

7. The method according to claim 6, characterized in that, The method further includes: The location information of historically stored target image frames is updated using the location information of the target image frames.

8. A data stream transmission channel switching device, characterized in that, include: The similarity comparison module is used to compare the current image frame in the current channel with the candidate image frames in the pre-cached target channel. The image acquisition device connected to the current channel has a different magnification than the image acquisition device connected to the target channel; The target image frame determination module is used to determine the target image frame from the candidate image frames based on the similarity comparison results; The data stream output module is used to switch the current channel to the target channel starting from the target image frame, so as to output the data stream through the target channel.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data stream transmission channel switching method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data stream transmission channel switching method as described in any one of claims 1-7.

Images