Synchronization method, system, and electronic device

Abstract

Embodiments of the present application provide a synchronization method, system and electronic device. The method comprises: a server receiving image frames sent by a plurality of IPCs and time information corresponding to the image frames, the image frames being obtained by encoding original images, the plurality of IPCs comprising a first IPC and a second IPC; when it is determined that frame times of the first IPC and the second IPC are not synchronized according to the time information corresponding to the image frames sent by the first IPC and the time information corresponding to the image frames sent by the second IPC, sending an encoding adjustment instruction to the first IPC, the encoding adjustment instruction being used to trigger the first IPC to perform encoding adjustment, so that the time at which the first IPC reads the original image of the next frame to be encoded is adjusted. In this way, the frame times of the plurality of IPCs are synchronized by the server interacting with the IPC data. Compared with the prior art, the present application does not need to rely on hardware, has good robustness, and can reduce costs and facilitate deployment of the plurality of IPCs.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of communication, and in particular to a synchronization method, system and electronic device. BACKGROUND

[0002] An IPC (IP Camera) is generally connected with a server (such as an MP (Media Process) device, an MEP (Multi-access Edge Platform) device, etc.) through a network. Generally, multiple IPCs collect original images and encode the original images into image frames, and then send the image frames to the server; then the server selects an image frame from the image frames sent by each IPC, and performs video stitching and FVV (Free-Viewpoint Video) video production processing.

[0003] Since the encoding start times of different IPCs may not be consistent, this will cause differences in the frame times of different IPCs, and further cause time differences (i.e., frame time differences) in the pictures captured by the multiple image frames sent by the multiple IPCs received by the server in the same time period (such as the time period of 8:10:22:10-20 milliseconds on the same day), that is, the frame times of the multiple IPCs are not synchronized; thereby causing problems such as jerkiness and incoherence in the pictures at the connection of the image frames from different IPCs in the stitched image.

[0004] In order to solve the problem of different frame times of multiple IPCs, the prior art adopts a data line to connect two IPCs, and then controls the encoding video frame time of the IPC by a hardware control circuit, so that the video frame times of all the IPCs are consistent, and frame time synchronization is achieved. The disadvantage of using hardware to achieve frame time synchronization of multiple IPCs is that additional hardware is required, which is high in cost; and the connection by a data line is not conducive to the deployment of the IPCs; in addition, if the hardware line contact is poor, it will cause the frame times to be different, poor robustness, etc. SUMMARY

[0005] In order to solve the above technical problems, the present application provides a synchronization method, system and electronic device. In the method, without the aid of hardware, frame time synchronization between multiple IPCs can be achieved through data interaction between the server and the IPCs, which is good in robustness; and can reduce the cost and facilitate the deployment of multiple IPCs.

[0006] In a first aspect, an embodiment of the present application provides a synchronization method, which comprises the following steps: first, a server receives image frames sent by a plurality of network cameras (IPCs) and time information corresponding to the image frames, wherein the image frames are obtained by encoding original images, the plurality of IPCs include a first IPC and a second IPC, and the second IPC is a reference for determining whether the frame time of the first IPC and the second IPC is synchronized; then, when the server determines that the frame time of the first IPC and the second IPC is not synchronized according to the time information corresponding to the image frames sent by the first IPC and the time information corresponding to the image frames sent by the second IPC, the server sends an encoding adjustment instruction to the first IPC, and the encoding adjustment instruction is used to trigger the first IPC to perform encoding adjustment, so that the time for the first IPC to read the original image of the next frame to be encoded is adjusted. In this way, the server triggers the first IPC to perform one or more times of encoding adjustment by sending one or more times of encoding adjustment instructions to the first IPC, so that the time for the first IPC to read the original image of the next frame to be encoded is adjusted one or more times, and then the frame time of the first IPC and the second IPC is synchronized; thereby avoiding the time difference of the pictures captured by the plurality of image frames received by the server in the same time period, and realizing the frame time synchronization of the plurality of IPCs.

[0007] Compared with the prior art that synchronizes the frame time of the plurality of IPCs by means of hardware, the present application does not need to rely on hardware, but can realize the frame time synchronization between the plurality of IPCs through data interaction between the server and the IPCs, has good robustness, and can reduce the cost and facilitate the deployment of the plurality of IPCs.

[0008] Illustratively, the plurality of IPCs include at least one first IPC and one second IPC.

[0009] Illustratively, the original image can be an image collected by a light sensor of the IPC, that is, a RAW (unprocessed) image.

[0010] Illustratively, the server can be a single server or a server cluster (such as a cloud server), and the present application does not limit this. For example, the server can be an MP device, an MEP device, etc.

[0011] Illustratively, the image frame can also be referred to as a code stream.

[0012] Illustratively, the image frame can be a video frame.

[0013] According to a first aspect, the server sends an encoding adjustment indication to the first IPC, comprising: the server sends an encoding control message to the first IPC; wherein the encoding control message comprises an offset time, the offset time being determined by the server according to a time difference between time information corresponding to an image frame sent by the first IPC and time information corresponding to an image frame sent by the second IPC; the encoding control message is used to trigger the first IPC to adjust the time of acquiring a next frame of original image to be encoded according to the offset time. In this way, when the first IPC receives the encoding control message, the time of acquiring the next frame of original image to be encoded can be moved by the offset time in the encoding control message; and then the first IPC can realize frame time synchronization with the second IPC through one-time encoding adjustment.

[0014] Illustratively, the offset time can be a time difference between the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC.

[0015] Illustratively, the offset time can be a time difference between the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC, and a sum or difference value with a preset period. The preset period can be a preset encoding period such as 30 ms, which can be set according to requirements, and the present application does not limit this.

[0016] According to the first aspect, or any one of the implementation modes of the first aspect, the server sends an encoding adjustment indication to the first IPC, comprising: the server sends an encoding restart message to the first IPC, the encoding restart message being used to trigger the first IPC to perform encoding restart. In this way, when the first IPC receives the encoding restart message, it can perform encoding restart; after the encoding restart is successful, it can read the next frame of original image to be encoded for encoding; and then the time of reading the next frame of original image to be encoded by the first IPC is adjusted. In this case, the first IPC can realize frame time synchronization with the second IPC through one-time or multiple-time encoding restart.

[0017] According to the first aspect, or any one of the implementation modes of the first aspect, the server sends an encoding adjustment indication to the first IPC, comprising: the server sends an encoding flush message to the first IPC, the encoding flush message being used to trigger the first IPC to perform frame flush. In this way, when the first IPC receives the encoding flush message, it can perform frame flush, i.e., start the next encoding, and then read the next frame of original image to be encoded and encode the next frame of original image to be encoded, and then the time of reading the next frame of original image to be encoded by the first IPC is adjusted. In this case, the first IPC can realize frame time synchronization with the second IPC through one-time or multiple-time frame flush.

[0018] According to a first aspect, or any possible implementation mode of the above first aspect, the determining that the frame times of the first IPC and the second IPC are not synchronized comprises: determining, by the server, that a time difference between the synchronization time information corresponding to the image frame sent by the first IPC and the synchronization time information corresponding to the image frame sent by the second IPC is greater than a threshold value.

[0019] For example, when the time difference between the synchronization time information corresponding to the image frame sent by the first IPC and the synchronization time information corresponding to the image frame sent by the second IPC is less than or equal to the threshold value, it can be determined that the frame times of the first IPC and the second IPC are synchronized.

[0020] According to the first aspect, or any possible implementation mode of the above first aspect, the time information corresponding to the image frame sent by each IPC in the plurality of IPCs is the time at which each IPC reads the original image of the next frame to be encoded. In this way, the time information corresponding to each image frame sent by each IPC is closer to the time at which the scene of each image frame is actually captured, so that the time difference between the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC determined by the server is more accurate, and the server sends the first IPC a smaller number of encoding adjustment instructions, so as to synchronize the frame times of the first IPC and the second IPC, and reduce the length of time for synchronizing the frame times of the first IPC and the second IPC.

[0021] According to a second aspect, the embodiments of the present application provide a synchronization method, which comprises the following steps: first, a first IPC reads an original image to be encoded; then, the first IPC encodes the original image into an image frame, and determines time information corresponding to the image frame; subsequently, the first IPC sends the image frame and the time information corresponding to the image frame. After that, the first IPC can receive an encoding adjustment instruction, which is generated and sent by a server according to the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by a second IPC when the server determines that the frame times of the first IPC and the second IPC are not synchronized, and the second IPC is a reference for determining whether the frame times of the first IPC and the second IPC are synchronized; then, the first IPC performs encoding adjustment according to the encoding adjustment instruction, so that the time at which the first IPC reads the original image of the next frame to be encoded is adjusted. In this way, the first IPC performs one or more times of encoding adjustment according to one or more encoding adjustment instructions sent by the server, so that the time at which the original image of the next frame to be encoded is read one or more times is adjusted, so as to synchronize the frame times with the second IPC; thereby avoiding the time difference between the scenes captured by the plurality of image frames received by the server in the same time period, and synchronizing the frame times of the plurality of IPCs.

[0022] Compared with the prior art of frame time synchronization of multiple IPCs by means of hardware, the application can realize frame time synchronization between multiple IPCs by data interaction between the server and the IPCs without the aid of hardware, has good robustness, and can reduce cost and facilitate deployment of multiple IPCs.

[0023] For example, the second IPC can perform the following steps: first, the second IPC reads the original image to be encoded; then, the second IPC encodes the original image into an image frame, and the second IPC determines time information corresponding to the image frame; and then, the second IPC sends the image frame and the time information corresponding to the image frame.

[0024] For example, the original image can be an image collected by a light sensor of the IPC, that is, a RAW (unprocessed) image.

[0025] For example, the image frame can also be referred to as a code stream.

[0026] For example, the image frame can be a video frame.

[0027] According to the second aspect, the first IPC determines the time information corresponding to the image frame, including: the first IPC determines the time information corresponding to the image frame according to the time of reading the original image to be encoded. In this way, the time information corresponding to each image frame sent by each IPC is closer to the time when the picture actually occurs, so that the time difference between the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC determined by the server is more accurate, and then the first IPC can realize frame time synchronization with the second IPC by performing a smaller number of encoding adjustments, thereby reducing the time length of synchronizing the frame time of the first IPC and the second IPC.

[0028] It should be noted that the time information can be determined according to any time before sending the image frame, for example, the time information corresponding to the image frame is determined according to the time of reading the original image. For example, the time information corresponding to the image frame is determined according to the time of encoding the image frame. For example, the time information corresponding to the image frame is determined according to the time of performing various encoding steps (such as encoding prediction, quantization, entropy encoding, etc.) in the encoding process; and the like, which are not limited in the application.

[0029] According to the second aspect, or any one of the implementation modes of the second aspect, the first IPC sends the image frame and the time information corresponding to the image frame, including: the first IPC encapsulates the time information corresponding to the image frame into an extension field of the image frame, and sends the encapsulated image frame.

[0030] For example, the first IPC can first encapsulate the image frame according to a video coding protocol, and encapsulate the time information into a first extension field of the image frame (i.e., an extension field of the video coding protocol) when encapsulating the image frame according to the video coding protocol. Then, the first IPC can second encapsulate the first encapsulated image frame according to a network transmission protocol, and send the second encapsulated image frame.

[0031] For example, the first IPC can first encapsulate the image frame according to a video coding protocol, and encapsulate the time information into a first extension field of the image frame (i.e., an extension field of the video coding protocol) when encapsulating the image frame according to the video coding protocol. Then, the first IPC can second encapsulate the first encapsulated image frame according to a network transmission protocol, and send the second encapsulated image frame.

[0032] According to a second aspect, or any possible implementation mode of the second aspect, the encoding adjustment indication is an encoding control message, the encoding control message comprises an offset time, the offset time is determined by the server according to a time difference between the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC; the first IPC performs the encoding adjustment according to the encoding adjustment indication, comprising: the first IPC adjusts the time of reading the next frame of original image to be encoded according to the offset time. In this way, the first IPC can realize the frame time synchronization with the second IPC through one encoding adjustment.

[0033] According to a second aspect, or any possible implementation mode of the second aspect, the encoding adjustment indication is an encoding control message, the encoding control message comprises an offset time, the offset time is determined by the server according to a time difference between the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC; the first IPC performs the encoding adjustment according to the encoding adjustment indication, comprising: the first IPC adjusts the time of reading the next frame of original image to be encoded according to the offset time. In this way, the first IPC can realize the frame time synchronization with the second IPC through one encoding adjustment.

[0034] According to a second aspect, or any possible implementation mode of the second aspect, the encoding adjustment indication is an encoding flush message, and the first IPC performs the encoding adjustment according to the encoding adjustment indication, including: the first IPC performs frame flushing according to the encoding flush message. Wherein, the frame flushing is to start the next encoding, at this time, the first IPC can read the next frame of original image to be encoded, and then encodes the next frame of original image to be encoded. Further, in this way, the time for the first IPC to read the next frame of original image to be encoded can also be adjusted, that is, the time for the first IPC to read the next frame of original image to be encoded is adjusted to the current time. In this case, the first IPC can realize the frame time synchronization with the second IPC through one or more times of frame flushing.

[0035] The second aspect and any possible implementation mode of the second aspect correspond to the first aspect and any possible implementation mode of the first aspect respectively. The technical effects of the second aspect and any possible implementation mode of the second aspect can refer to the technical effects of the first aspect and any possible implementation mode of the first aspect, which will not be repeated here.

[0036] In a third aspect, the present application provides a synchronization system, including: a plurality of network cameras IPCs and a server, the plurality of IPCs including a first IPC and a second IPC, the second IPC being a reference for whether the frame time of the first IPC and the second IPC is synchronized;

[0037] The first IPC is configured to read a first original image to be encoded, encode the first original image to a first image frame, determine time information corresponding to the first image frame, send the first image frame and the time information corresponding to the first image frame to the server, and receive an encoding adjustment indication; and perform encoding adjustment according to the encoding adjustment indication, so as to adjust the time for reading a next frame of first original image to be encoded;

[0038] The second IPC is configured to read a second original image to be encoded, encode the second original image to a second image frame, determine time information corresponding to the second image frame, and send the second image frame and the time information corresponding to the second image frame to the server;

[0039] The server is configured to receive the first image frame and the time information corresponding to the first image frame sent by the first IPC, and receive the second image frame and the time information corresponding to the second image frame sent by the second IPC; when it is determined that the frame time of the first IPC and the second IPC is not synchronized according to the time information corresponding to the first image frame sent by the first IPC and the time information corresponding to the second image frame sent by the second IPC, send an encoding adjustment indication to the first IPC.

[0040] The third aspect and any implementation manner of the server of the third aspect correspond to the first aspect and any implementation manner of the first aspect respectively. The technical effects of the third aspect and any implementation manner of the server of the third aspect correspond to the technical effects of the first aspect and any implementation manner of the first aspect, which are not described herein again.

[0041] The third aspect and any implementation manner of the first IPC of the third aspect correspond to the second aspect and any implementation manner of the second aspect respectively. The technical effects of the third aspect and any implementation manner of the first IPC of the third aspect correspond to the technical effects of the second aspect and any implementation manner of the second aspect, which are not described herein again.

[0042] In the fourth aspect, the embodiments of the present application provide a network camera IPC, which can be used to implement the second aspect and any implementation manner of the second aspect.

[0043] The technical effects of the fourth aspect and any implementation manner of the first IPC of the fourth aspect correspond to the technical effects of the second aspect and any implementation manner of the second aspect, which are not described herein again.

[0044] In the fifth aspect, the embodiments of the present application provide an electronic device, which includes a memory and a processor, the memory is coupled to the processor, and the memory stores program instructions, when the program instructions are executed by the processor, the electronic device executes the synchronization method in the first aspect or any possible implementation manner of the first aspect.

[0045] The fifth aspect and any implementation manner of the fifth aspect correspond to the first aspect and any implementation manner of the first aspect respectively. The technical effects of the fifth aspect and any implementation manner of the fifth aspect correspond to the technical effects of the first aspect and any implementation manner of the first aspect, which are not described herein again.

[0046] In the sixth aspect, the embodiments of the present application provide an electronic device, which includes a memory and a processor, the memory is coupled to the processor, and the memory stores program instructions, when the program instructions are executed by the processor, the electronic device executes the synchronization method in the second aspect or any possible implementation manner of the second aspect.

[0047] The sixth aspect and any implementation manner of the sixth aspect correspond to the second aspect and any implementation manner of the second aspect respectively. The technical effects of the sixth aspect and any implementation manner of the sixth aspect correspond to the technical effects of the second aspect and any implementation manner of the second aspect, which are not described herein again.

[0048] In a seventh aspect, an embodiment of the present application provides a chip, comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of an electronic device and send a signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the synchronization method in the first aspect or any possible implementation manner of the first aspect.

[0049] The seventh aspect and any possible implementation manner of the seventh aspect correspond to the first aspect and any possible implementation manner of the first aspect respectively. For details, refer to the technical effects of the first aspect and any possible implementation manner of the first aspect, which will not be repeated here.

[0050] In an eighth aspect, an embodiment of the present application provides a chip, comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of an electronic device and send a signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the synchronization method in the second aspect or any possible implementation manner of the second aspect.

[0051] The eighth aspect and any possible implementation manner of the eighth aspect correspond to the second aspect and any possible implementation manner of the second aspect respectively. For details, refer to the technical effects of the second aspect and any possible implementation manner of the second aspect, which will not be repeated here.

[0052] In a ninth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the synchronization method in the first aspect or any possible implementation manner of the first aspect.

[0053] The ninth aspect and any possible implementation manner of the ninth aspect correspond to the first aspect and any possible implementation manner of the first aspect respectively. For details, refer to the technical effects of the first aspect and any possible implementation manner of the first aspect, which will not be repeated here.

[0054] In a tenth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the synchronization method in the second aspect or any possible implementation manner of the second aspect.

[0055] The tenth aspect and any kind of implementation manner of the tenth aspect correspond to the second aspect and any kind of implementation manner of the second aspect respectively. The technical effects corresponding to the tenth aspect and any kind of implementation manner of the tenth aspect can refer to the technical effects corresponding to the second aspect and any kind of implementation manner of the second aspect, which will not be described herein again.

[0056] In a eleventh aspect, an embodiment of the present application provides a computer program product, which comprises a software program. When the software program is executed by a computer or a processor, the computer or the processor executes the synchronization method in the first aspect or any possible implementation manner of the first aspect.

[0057] The eleventh aspect and any kind of implementation manner of the eleventh aspect correspond to the first aspect and any kind of implementation manner of the first aspect respectively. The technical effects corresponding to the eleventh aspect and any kind of implementation manner of the eleventh aspect can refer to the technical effects corresponding to the first aspect and any kind of implementation manner of the first aspect, which will not be described herein again.

[0058] In a twelfth aspect, an embodiment of the present application provides a computer program product, which comprises a software program. When the software program is executed by a computer or a processor, the computer or the processor executes the synchronization method in the second aspect or any possible implementation manner of the second aspect.

[0059] The twelfth aspect and any kind of implementation manner of the twelfth aspect correspond to the second aspect and any kind of implementation manner of the second aspect respectively. The technical effects corresponding to the twelfth aspect and any kind of implementation manner of the twelfth aspect can refer to the technical effects corresponding to the second aspect and any kind of implementation manner of the second aspect, which will not be described herein again. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1a An application scenario schematic diagram is exemplarily shown;

[0061] Figure 1b An application scenario schematic diagram is exemplarily shown;

[0062] Figure 1c An application scenario schematic diagram is exemplarily shown;

[0063] Figure 1d An IPC deployment manner schematic diagram is exemplarily shown;

[0064] Figure 1e An IPC structure schematic diagram is exemplarily shown;

[0065] Figure 1f A synchronization component structure schematic diagram is exemplarily shown;

[0066] Figure 2 synchronization process diagram shown for example;

[0067] Figure 3a synchronization process diagram shown for example;

[0068] Figure 3b frame structure diagram of encapsulated image frame shown for example;

[0069] Figure 3c frame structure diagram of encapsulated image frame shown for example;

[0070] Figure 3d frame structure diagram of encapsulated image frame shown for example;

[0071] Figure 3e message structure diagram of encoding control message shown for example;

[0072] Figure 4 synchronization process diagram shown for example;

[0073] Figure 5 synchronization process diagram shown for example;

[0074] Figure 6 synchronization system diagram shown for example;

[0075] Figure 7 structure diagram of apparatus shown for example. DETAILED DESCRIPTION

[0076] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0077] The term “and / or” in the present application is only used to describe the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone.

[0078] The terms “first” and “second” and the like in the specification and claims of the embodiments of the present application are used to distinguish different objects, and are not used to describe the specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, and are not used to describe the specific order of the target objects.

[0079] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0080] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0081] Figure 1a This is a schematic diagram illustrating an application scenario.

[0082] Reference Figure 1a For example, n (n is an integer greater than 1) IPCs are connected to the same MP device via a router. After the n IPCs are started, each IPC can acquire the raw image of the target object (i.e., RAW image), encode the raw image into an image frame, and send the image frame to the router; the router then forwards the image frame to the MP device. After receiving the image frames sent by the n IPCs, the MP device can process the image frames sent by the n IPCs, such as video stitching, FVV video production, etc., which is not limited in this application. Subsequently, the MP device can send the processed image frames to terminal devices such as mobile phones, tablets, etc.

[0083] Figure 1b This is a schematic diagram illustrating an application scenario.

[0084] Reference Figure 1b For example, n IPCs are connected to the same MEP device via a base station and a core network. After the n IPCs are started, each IPC can acquire the original image of the target object, encode the original image into an image frame, and send the image frame to the base station. After receiving the image frames sent by the n IPCs, the base station can send the image frames sent by the n IPCs to the core network, and the UPF (User Plane Function) network element of the core network will send the image frames sent by the n IPCs to the MEP device. After receiving the image frames sent by the n IPCs, the MEP device can process the image frames sent by the n IPCs, such as video stitching, FVV video production, etc., which is not limited in this application. Subsequently, the MEP device can send the processed image frames to terminal devices such as mobile phones, tablets, etc.

[0085] Figure 1c This is a schematic diagram illustrating an application scenario.

[0086] Referring to Figure 1c , for example, n IPCs are connected to the same MP device through a communication satellite. After the n IPCs are started, each IPC can capture an original image of a target object, then encode the original image into an image frame, and send the image frame to the communication satellite. After the communication satellite receives the image frames sent by the n IPCs, the communication satellite can send the image frames sent by the n IPCs to the MP device. After the MP device receives the image frames sent by the n IPCs, the MP device can process the images sent by the n IPCs, such as video stitching, FVV video production, etc., which are not limited by the present application. Subsequently, the MP device can send the image frames obtained by processing to a terminal device such as a mobile phone, a tablet computer, etc.

[0087] It should be noted that the MP device and the MEP device can be a server, which can refer to a single server, or a server cluster (for example, a cloud server, etc.), which are not limited by the present application.

[0088] It should be noted that the IPC can continuously send multiple image frames to the MP device / MEP device. In one possible way, the continuously sent multiple image frames can constitute a video, and at this time, the image frame can also be referred to as a video frame.

[0089] Figure 1d A schematic diagram of the deployment mode of the IPC shown for example. In Figure 1d , the deployment mode of the multiple IPCs relative to the target object is shown.

[0090] Referring to Figure 1d (1), for example, n (n = 6) IPCs (including IPC1, IPC2, IPC3, IPC4, IPC5, and IPC6) can be deployed around the target object, and each IPC is deployed at a different angle relative to the target object. It should be understood that Figure 1d (1) is only one example of the deployment mode of the multiple IPCs, and the number of IPCs deployed around the target object by the present application can be more than Figure 1d (1) shown, which are not limited by the present application.

[0091] For example, the application scenario using the deployment mode shown in Figure 1d (1) can include a variety of, for example, artistic dance performances, sports competitions, film and television variety shows, net red performances, scenic spot photo taking, live streaming, exhibition drainage, action teaching, etc. In this way, after the MP device / MEP device performs FVV video production on the image frames sent by the n IPCs, the MP device / MEP device can provide users with a variety of different viewing angles of the picture.

[0092] Referring to Figure 1d(2) For example, n (n=5) IPCs (including IPC1, IPC2, IPC3, IPC4 and IPC5) are arranged in a fan shape with the user position as the center and the target object as the direction. It should be understood that, Figure 1d (2) is only one example of the arrangement of multiple IPCs, and the number of IPCs arranged in a fan shape with the user position as the center and the target object as the direction can be more than or less than n in the present application. Figure 1d (2) More or less can be shown, which is not limited in the present application.

[0093] For example, the arrangement of (2) can be applied in various scenarios, such as a virtual reality (VR) scenario, an augmented reality (AR) scenario, etc. In this way, the MP device / MEP device can perform video stitching on the image frames sent by two IPCs adjacent in position in the n IPCs, and form a panoramic image for VR scenario and AR scenario display. Figure 1d (2) More or less can be shown, which is not limited in the present application.

[0094] Figure 1e The structure of the IPC is shown for example.

[0095] Referring to Figure 1e For example, the IPC can include a photosensor, an image buffer, an encoding component and a communication component. It should be understood that, Figure 1e (2) is only one example of the arrangement of multiple IPCs, and the number of IPCs arranged in a fan shape with the user position as the center and the target object as the direction can be more than or less than n in the present application. Figure 1e More components can be shown, which is not limited in the present application.

[0096] Referring to Figure 1e For example, the photosensor is a kind of light sensor that receives light signals through external exposure and converts them into electrical signals. The photosensor and the image buffer can be connected by a wire harness to convert the light signals into image data (hereinafter referred to as raw images) through photoelectric conversion and output them to the image buffer in real time. The encoding component can read the raw images to be encoded from the image buffer according to a preset period, encode the raw images, and output image frames. The encoded image frames are packaged in the communication component and sent to the MP device or the MEP device via the network.

[0097] It should be noted that the interval at which the IPC continuously sends image frames to the MP device / MEP device can be equal to the preset period.

[0098] For example, before video stitching or FVV video production, the MP device / MEP device needs to synchronize the frame time of multiple image frames sent by multiple IPCs (i.e. synchronize the pictures taken by multiple image frames sent by multiple IPCs received in the same time period (e.g. 8:10:22:10~20 ms on the same day)) to ensure that there is no time difference between the pictures taken by multiple image frames in the same group for video stitching or FVV video production.

[0099] At present, one of the reasons for causing the frame time of multiple IPCs to be unsynchronized (i.e. there is a time difference between the pictures taken by multiple image frames sent by multiple IPCs received by the server in the same time period (e.g. 8:10:22:10~20 ms on the same day)) is that the different IPCs have inconsistent encoding start time (i.e. the time of reading the original image to be encoded from the image cache for the first time). Based on this, the application proposes a synchronization method, which sends the time information corresponding to the image frame sent by the IPC at the same time as the image frame is sent; then, the MP device / MEP device determines whether the frame time of multiple IPCs is synchronized according to the time information corresponding to the image frame sent by each IPC; when it is determined that the frame time of multiple IPCs is unsynchronized, an encoding adjustment instruction can be sent to some IPCs to trigger some IPCs to perform encoding adjustment, thereby controlling the frame time of some IPCs, so that the frame time of multiple IPCs is synchronized, thereby realizing the synchronization of the frame time of multiple IPCs.

[0100] Figure 1f The structure of the synchronization component is shown for example. Figure 1f The synchronization component in the server (e.g. MP device, MEP device) is shown, which is used to ensure that the server side implements the synchronization method of the application.

[0101] Reference is made to Figure 1f For example, the synchronization component in the MP device can include a time comparison module, a decision module and a control module. The time comparison module is used to compare the time information corresponding to two image frames sent by two IPCs to determine the time difference; the decision module is used to decide whether to send an encoding adjustment instruction according to the time difference; and the control module is used to send the encoding adjustment instruction.

[0102] It should be understood that Figure 1f Only one example of the application, the synchronization component can include more or less modules than Figure 1f The application does not limit this.

[0103] The synchronization method of the application is described below.

[0104] Figure 2 The synchronization process is shown for example. In Figure 2In an embodiment, the number of the IPCs is n, and the n IPCs are connected to the same server, which can be a single server or a server cluster. The server can be an MP device or an MEP device.

[0105] In S201, the IPC reads an original image to be encoded.

[0106] For example, the n IPCs can include one second IPC and (n-1) first IPCs, where the second IPC is a reference for whether the frame time of the first IPC is synchronized with that of the second IPC.

[0107] In one possible way, the first IPC and the second IPC in the n IPCs can be determined by mutual negotiation among the n IPCs.

[0108] In one possible way, the server can determine the first IPC and the second IPC in the n IPCs when starting to synchronize the frame time of the multiple IPCs.

[0109] For example, when starting to synchronize the frame time of the multiple IPCs, the server can determine the first IPC and the second IPC in the n IPCs according to a preset rule. For example, the preset rule can be set according to requirements, such as random selection, selection according to a preset order, etc., which are not limited in the present application.

[0110] For example, after the n IPCs are started, the n IPCs can perform system time synchronization. For example, after each IPC is started, it can interact with a time server and then synchronize the system time with the time server, so that the system time of the n IPCs can be synchronized.

[0111] For example, S201 includes S201a and S201b:

[0112] In S201a, the first IPC reads an original image to be encoded.

[0113] In S201b, the second IPC reads an original image to be encoded.

[0114] For example, after the first IPC is started, the photosensor in the first IPC can receive a light signal and convert the light signal into an electrical signal. Then, the original image obtained through photoelectric conversion is output to the image buffer in real time. In this way, the image buffer can store multiple frames of original images.

[0115] For example, after the first IPC is started, the encoding component is also started; after the encoding component is started, the encoding component can read the original image to be encoded from the image cache according to a preset period. In this way, the (n-1) first IPCs can all read the original image to be encoded from the corresponding image cache. It should be noted that the encoding component can read one frame of original image to be encoded from the image cache in one preset period.

[0116] For example, after the second IPC is started, the encoding component in the second IPC can also read the original image to be encoded from the image cache according to a preset period.

[0117] In S202, the IPC encodes the original image into an image frame and determines time information corresponding to the image frame.

[0118] For example, S202 includes S202a and S202b:

[0119] In S202a, the first IPC encodes the original image into an image frame and determines time information corresponding to the image frame.

[0120] In S202b, the second IPC encodes the original image into an image frame and determines time information corresponding to the image frame.

[0121] For example, after the encoding component of the first IPC reads the original image, on the one hand, the encoding component can encode the original image to output an image frame (that is, a code stream). For example, the first IPC can encode the original image according to a demand, such as using an H.264, H.265, H.266, or the like, to encode the original image, and the present application does not limit this.

[0122] For example, after the encoding component of the first IPC reads the original image, on the other hand, the encoding component or the communication component can determine time information corresponding to the image frame. The time information corresponding to the image frame can be used for the server to synchronize the frame time of the plurality of IPCs. The time information can be used to identify time, and the time information can be a timestamp, and the present application does not limit this.

[0123] In one possible manner, the encoding component can determine the time information corresponding to the image frame according to the time at which the encoding component reads the original image to be encoded from the image cache. For example, the time at which the encoding component reads the original image to be encoded from the image cache can be taken as the time information corresponding to the image frame; for another example, a difference between the time at which the encoding component reads the original image to be encoded from the image cache and a preset time (such as 10 milliseconds on January 1, 1970, 0:00:00) can be taken as the time information corresponding to the image frame; and the present application does not limit this. For example, the unit of the time information corresponding to the image frame can be ms (milliseconds).

[0124] In a possible implementation, the time information corresponding to the image frame can be determined by the encoding component according to a time at which the image frame is encoded by the encoding component. For example, the time at which the image frame is encoded by the encoding component can be taken as the time information corresponding to the image frame; or for another example, a difference between the time at which the image frame is encoded by the encoding component and a preset time can be taken as the time information corresponding to the image frame. The present application does not limit this.

[0125] In a possible implementation, the time information corresponding to the image frame can be determined by the encoding component according to a time at which the encoding component performs various encoding steps (such as encoding prediction, quantization, entropy encoding, etc.) in the encoding process. For example, the time at which the encoding component starts to perform encoding prediction can be taken as the time information corresponding to the image frame; or for another example, the time at which the encoding component starts to perform quantization can be taken as the time information corresponding to the image frame; and so on, which the present application does not limit.

[0126] In a possible implementation, the time information corresponding to the image frame can be determined by the communication component according to a time at which the image frame is received. For example, the time at which the image frame is received by the communication component can be taken as the time information corresponding to the image frame; or for another example, a difference between the time at which the image frame is received by the communication component and a preset time can be taken as the time information corresponding to the image frame.

[0127] It should be understood that any time before the image frame is sent to the server can be taken as the time information corresponding to the image frame, which the present application does not limit.

[0128] In this way, the (n-1) first IPCs can all obtain the image frame and the time information corresponding to the image frame.

[0129] For example, after the encoding component of the second IPC reads the original image, the image frame and the time information corresponding to the image frame can be obtained by referring to the above manner.

[0130] S203: The IPC sends the image frame and the time information corresponding to the image frame to the server.

[0131] For example, S203 includes S203a and S203b:

[0132] S203a: The first IPC sends the image frame and the time information corresponding to the image frame to the server.

[0133] S203b: The second IPC sends the image frame and the time information corresponding to the image frame to the server.

[0134] For example, each of the (n-1) first IPCs can send the obtained image frame and the time information corresponding to the image frame to the same server.

[0135] For example, the second IPC can also send the obtained image frame and the time information corresponding to the image frame to the server.

[0136] S204, when the server determines that the frame time of the first IPC and the second IPC is not synchronized according to the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC, the server sends an encoding adjustment instruction to the first IPC.

[0137] For example, the server can determine whether the frame time of each first IPC and the second IPC is synchronized according to the time information corresponding to the image frame sent by each first IPC and the time information corresponding to the image frame sent by the second IPC.

[0138] For example, for a first IPC, the server can determine the time difference between the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC. If the time difference is greater than a threshold value, it can be determined that the frame time of the first IPC and the second IPC is not synchronized. If the time difference is less than or equal to the threshold value, it can be determined that the frame time of the first IPC and the second IPC is synchronized. The threshold value can be set according to requirements, and the present application does not limit this.

[0139] In this way, according to the above manner, it can be determined whether the frame time of each first IPC in the (n-1) first IPCs and the second IPC is synchronized; that is, it can be determined whether the frame time of the n IPCs is synchronized.

[0140] It should be understood that the server can determine whether the frame time of the (n-1) first IPCs and the second IPC is synchronized in series, or in parallel; the present application does not limit this.

[0141] For example, for a first IPC, if the server determines that the frame time of the first IPC and the second IPC is not synchronized, synchronization adjustment needs to be performed on the first IPC, at this time, an encoding adjustment instruction can be generated; and then the encoding adjustment instruction is sent to the first IPC. If the server determines that the frame time of the first IPC and the second IPC is synchronized, the synchronization adjustment of the first IPC can be ended, at this time, the encoding adjustment instruction does not need to be generated.

[0142] S205, the first IPC performs encoding adjustment according to the encoding adjustment instruction, so that the time of reading the original image of the next frame by the first IPC is adjusted.

[0143] For example, any one of the (n-1) first IPCs receives the encoding adjustment indication sent by the server, and adjusts the encoding according to the encoding adjustment indication, so that the time for reading the next frame of the original image to be encoded is adjusted. In this way, when the time for reading the next frame of the original image to be encoded is reached, the above S201-S205 can be executed again, and the cycle continues until the frame times of the (n-1) first IPCs and the second IPC are synchronized.

[0144] It should be noted that in one possible way, only part of the (n-1) first IPCs are out of synchronization with the second IPC, and at this time the server sends the encoding adjustment indication only to these first IPCs. Figure 2 In one possible way, the frame times of the (n-1) first IPCs are all out of synchronization with the second IPC, and at this time the server can send the encoding adjustment indication to these (n-1) first IPCs. Figure 2 In one possible way, the frame times of the (n-1) first IPCs are all out of synchronization with the second IPC, and at this time the server can send the encoding adjustment indication to these (n-1) first IPCs.

[0145] It should be noted that for one first IPC, it can be synchronized with the second IPC after receiving the encoding adjustment indication sent by the server once and adjusting the encoding according to the encoding adjustment indication; that is, the first IPC can be synchronized with the second IPC after adjusting the encoding once. It can also be synchronized with the second IPC after receiving the encoding adjustment indication sent by the server multiple times and adjusting the encoding according to the multiple encoding adjustment indications; that is, the first IPC can be synchronized with the second IPC after adjusting the encoding multiple times.

[0146] In this way, the first IPC adjusts the encoding according to the encoding adjustment indication sent by the server once or multiple times, and adjusts the time for reading the next frame of the original image to be encoded once or multiple times, so as to be synchronized with the second IPC; thereby avoiding the time difference between the pictures taken by the multiple image frames received by the server in the same time period, and realizing the synchronization of the frame times of the multiple IPCs.

[0147] Compared with the prior art of synchronizing the frame times of multiple IPCs by means of hardware, the present application does not need to rely on hardware, but can realize the synchronization of the frame times of multiple IPCs through data interaction between the server and the IPCs, and has good robustness; and can reduce the cost and facilitate the deployment of multiple IPCs.

[0148] Figure 3a The synchronization process is shown in the example. In the embodiment of Figure 3a The encoding adjustment indication is an encoding control message.

[0149] S301, the IPC reads an original image to be encoded.

[0150] For example, S301 includes S301a and S301b:

[0151] S301a, the first IPC reads an original image to be encoded.

[0152] S301b, the second IPC reads an original image to be encoded.

[0153] S302, the IPC encodes the original image into an image frame and determines time information corresponding to the image frame.

[0154] For example, S302 includes S302a and S302b:

[0155] S302a, the first IPC encodes the original image into an image frame and determines time information corresponding to the image frame.

[0156] S302b, the second IPC encodes the original image into an image frame and determines time information corresponding to the image frame.

[0157] For example, S301-S302 can refer to the description of S201-S202 above, and will not be described here.

[0158] S303, the IPC sends the image frame and the time information corresponding to the image frame to the server.

[0159] For example, S303 includes S303a and S303b:

[0160] S303a, the first IPC sends the image frame and the time information corresponding to the image frame to the server.

[0161] S303b, the second IPC sends the image frame and the time information corresponding to the image frame to the server.

[0162] For example, the first IPC can encapsulate the time information corresponding to the image frame together with the image frame, and then send the encapsulated image frame and the time information corresponding to the image frame to the server. The following takes one first IPC as an example to describe the process of the first IPC sending the image frame and the time information corresponding to the image frame to the server.

[0163] Figure 3b The frame structure of the encapsulated image frame is shown for example.

[0164] For example, the time information corresponding to the image frame can be encapsulated into the extension field of the image frame, and then the encapsulated image frame is sent to the server.

[0165] Exemplarily, a header and an extension field can be encapsulated for the image frame; and the time information corresponding to the image frame is encapsulated in the extension field; as shown in Figure 3b

[0166] Figure 3c An exemplary frame structure diagram of the encapsulated image frame is shown in Figure 3c In the embodiment, the time information corresponding to the image frame is encapsulated in the extension field of the video encoding protocol.

[0167] Exemplarily, after the encoding component of the first IPC encodes the original image to obtain the image frame, the image frame can be first encapsulated according to the video encoding protocol (such as H.264, H.265, H.266, etc.); and at the same time of first encapsulating the image frame according to the video encoding protocol, the time information corresponding to the image frame is encapsulated into the first extension field of the image frame. Exemplarily, in the process of first encapsulating the image frame according to the video encoding protocol, a header (header1) of the video encoding protocol and the first extension field (that is, the extension field of the video encoding protocol) can be encapsulated for the image frame; and the time information corresponding to the image frame is encapsulated in the first extension field; as shown in Figure 3c

[0168] In this case, the time information corresponding to the image frame can be determined according to any time before the image frame is encoded, such as the time when the encoding component reads the original image to be encoded from the image buffer, such as the time when various encoding steps are executed in the encoding process of the encoding component, such as the time when the image frame is encoded, etc., which is not limited by the present application.

[0169] Then, the encoding component can send the first encapsulated image frame to the communication component, and the communication component encapsulates the first encapsulated image frame according to the network transmission protocol to obtain the second encapsulated image frame. Exemplarily, in the process of second encapsulating the first encapsulated image frame according to the network transmission protocol, a header (header2) of the network transmission protocol and an extension field (hereinafter referred to as the second extension field) of the network transmission protocol can be encapsulated for the first encapsulated image frame; as shown in Figure 3c (2). Subsequently, the second encapsulated image frame is sent to the server.

[0170] Figure 3d An exemplary frame structure diagram of the encapsulated image frame is shown in Figure 3d In the embodiment, the time information corresponding to the image frame is encapsulated in the extension field of the network transmission protocol.

[0171] ​​For example, after the encoding component of the first IPC encodes the original image to obtain the image frame, the encoding component can perform first encapsulation on the image frame according to the video encoding protocol to obtain the first encapsulated image frame. For example, in the process of performing first encapsulation on the image frame according to the video encoding protocol, the video encoding protocol header (hereinafter referred to as header 1) and the first extension field (i.e., the extension field of the video encoding protocol) can be encapsulated for the image frame; as shown in FIG. 1. Figure 3d

[0172] Then, the encoding component can send the first encapsulated image frame to the communication component, and the communication component can perform second encapsulation on the first encapsulated image frame according to the network transmission protocol, and encapsulate the time information corresponding to the image frame into the second extension field of the image frame while performing the second encapsulation. For example, in the process of performing second encapsulation on the first encapsulated image frame according to the network transmission protocol, the network transmission protocol header (hereinafter referred to as header 2) and the second extension field (i.e., the extension field of the network transmission protocol) can be encapsulated for the image frame; and the time information corresponding to the image frame can be encapsulated in the second extension field; as shown in FIG. 2. Subsequently, the second encapsulated image frame is sent to the server. Figure 3d

[0173] In this case, the time information corresponding to the image frame can be determined at any time before the second encapsulation on the first encapsulated image frame, such as the time when the encoding component reads the original image from the image buffer, the time when various encoding steps are performed in the encoding process, the time when the image frame is encoded, the time when the image frame is first encapsulated, and the like, which are not limited in the present application.

[0174] For example, the second IPC can also send the image frame and the time information corresponding to the image frame to the server by encapsulating the time information corresponding to the image frame into the extension field of the image frame in the above manner, which will not be described herein again.

[0175] S304, the server determines whether the first IPC and the second IPC are synchronized according to the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC.

[0176] For example, S304 can refer to the description of S204 above, which will not be described herein again.

[0177] S305, when the server determines that the first IPC and the second IPC are not synchronized, the server sends an encoding control message to the first IPC.

[0178] ​​For example, for a first IPC, when the server determines that the first IPC is not synchronized with a second IPC, a control message can be generated according to a time difference between time information corresponding to an image frame sent by the first IPC and time information corresponding to an image frame sent by the second IPC. The control message can include a header and a time offset field, as shown in Figure 3e The time offset field is used to carry an offset time, which can be determined according to the time difference between the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC.

[0179] In one possible manner, the offset time can be a time difference obtained by subtracting the time information corresponding to the image frame sent by the first IPC from the time information corresponding to the image frame sent by the second IPC. In this case, the offset time can be positive or negative.

[0180] In one possible manner, if the time information corresponding to the image frame sent by the second IPC is less than the time information corresponding to the image frame sent by the second IPC, the time difference obtained by subtracting the time information corresponding to the image frame sent by the first IPC from the time information corresponding to the image frame sent by the second IPC is negative. In this case, the sum of the time difference and a preset period can be calculated, and the sum is used as the offset time. If the time information corresponding to the image frame sent by the second IPC is greater than the time information corresponding to the image frame sent by the second IPC, the time difference obtained by subtracting the time information corresponding to the image frame sent by the first IPC from the time information corresponding to the image frame sent by the second IPC is positive. In this case, the time difference is used as the offset time. That is, in this manner, the obtained offset time is positive.

[0181] In one possible manner, if the time information corresponding to the image frame sent by the second IPC is greater than the time information corresponding to the image frame sent by the second IPC, the time difference obtained by subtracting the time information corresponding to the image frame sent by the first IPC from the time information corresponding to the image frame sent by the second IPC is positive. In this case, the difference between the time difference and a preset period can be calculated, and the difference is used as the offset time. If the time information corresponding to the image frame sent by the second IPC is less than the time information corresponding to the image frame sent by the second IPC, the time difference obtained by subtracting the time information corresponding to the image frame sent by the first IPC from the time information corresponding to the image frame sent by the second IPC is negative. In this case, the time difference is used as the offset time. That is, in this manner, the obtained offset time is negative.

[0182] It should be understood that the present application does not limit the manner of determining the offset time.

[0183] S306, the first IPC adjusts the time for reading the next frame of the original image to be encoded according to the offset time.

[0184] For example, after receiving the encoding control message, the first IPC can parse the message to obtain the offset time carried by it. Then, the first IPC determines the time to read the next frame of the original image to be encoded according to a preset period; and then shifts the time for reading the next frame by this offset time. In this way, there will be no time difference between the next image frame sent by the first IPC and the next image frame captured by the second IPC, thus achieving frame time synchronization between the first and second IPCs.

[0185] In this way, after the first IPC receives an encoding control message from the server and adjusts the time for reading the next frame of the original image to be encoded according to the offset time in the encoding control message, it can achieve frame time synchronization with the second IPC. That is to say, in Figure 3a In one embodiment, for each first IPC, frame time synchronization with the second IPC can be achieved through a single encoding adjustment.

[0186] Furthermore, if the time information of the image frame is determined based on the time of reading the original image to be encoded, then the time information corresponding to each image frame sent by each IPC is closer to the actual time when the scene of each image frame was captured. In this way, the time difference between the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC determined by the server is more accurate. As a result, the server sends fewer encoding adjustment instructions to the first IPC, and the first IPC performs fewer encoding adjustments, thereby achieving frame time synchronization between the first IPC and the second IPC and reducing the time required to synchronize the frame time of the first IPC and the second IPC.

[0187] It should be noted that, for Figure 3a In a specific embodiment, for an IPC that does not have the function of processing encoded control messages, a service / application for processing encoded control messages can be added to the IPC; in this way, the IPC can complete the task. Figure 3a S306. Therefore, in accordance with Figure 3a The steps in this embodiment work in conjunction with the server to achieve frame time synchronization.

[0188] Figure 4 This is a schematic diagram illustrating an exemplary synchronization process. Figure 4 In this embodiment, the encoding adjustment instruction is an encoding restart message.

[0189] S401, IPC reads the raw image to be encoded.

[0190] S401 includes S401a and S401b.

[0191] S401a, the first IPC reads the original image to be encoded.

[0192] S401b, the second IPC reads the original image to be encoded.

[0193] S402, the IPC encodes the original image into image frames, and determines the time information corresponding to the image frames.

[0194] S402 includes S402a and S402b.

[0195] S402a, the first IPC encodes the original image into image frames, and determines the time information corresponding to the image frames.

[0196] S402b, the second IPC encodes the original image into image frames, and determines the time information corresponding to the image frames.

[0197] S401-S402 can refer to the description of S201-S202 above, and will not be repeated here.

[0198] S403, the IPC sends the image frames and the time information corresponding to the image frames to the server.

[0199] S403 includes S403a and S403b.

[0200] S403a, the first IPC sends the image frames and the time information corresponding to the image frames to the server.

[0201] S403b, the second IPC sends the image frames and the time information corresponding to the image frames to the server.

[0202] S403 can refer to the description of S303 above, and will not be repeated here.

[0203] S404, the server determines whether the first IPC and the second IPC are synchronized according to the time information corresponding to the image frames sent by the first IPC and the time information corresponding to the image frames sent by the second IPC.

[0204] S304 can refer to the description of S204 above, and will not be repeated here.

[0205] S405, the server sends a restart message to the first IPC when it is determined that the first IPC and the second IPC are not synchronized.

[0206] The restart message is used to trigger the first IPC to restart encoding, that is, to restart the encoding component.

[0207] S406, the first IPC performs encoding restart according to the encoding restart message.

[0208] For example, after receiving the encoding restart message, the first IPC can restart the encoding component; after the encoding component restarts successfully, it can start to obtain the next frame of original image to be encoded from the image cache, and then perform S402-S403 described above. In this way, the time for the first IPC to read the next frame of original image to be encoded from the image cache is adjusted to the time when the encoding restart is successful.

[0209] It should be noted that in one case, after the first IPC performs this encoding restart, the time for the first IPC to read the next frame of original image to be encoded from the image cache is the same as the time for the second IPC to read the next frame of original image to be encoded from the image cache; after the first IPC and the second IPC perform S401a-S403a and S401b-S403b again, the server can receive the next image frame and the time information corresponding to the image frame sent by the first IPC, and the next image frame and the time information corresponding to the image frame sent by the second IPC, and can determine that the frame time of the first IPC and the second IPC is synchronized, and does not need to generate an encoding restart message again. That is, the first IPC can achieve frame time synchronization with the second IPC after one encoding restart.

[0210] In one case, after the first IPC performs this encoding restart, the time for the first IPC to read the next frame of original image to be encoded from the image cache is still different from the time for the second IPC to read the next frame of original image to be encoded from the image cache; after the first IPC and the second IPC perform S401a-S403a and S401b-S403b again, the server can receive the next image frame and the time information corresponding to the image frame sent by the first IPC, and the next image frame and the time information corresponding to the image frame sent by the second IPC; and can determine that the frame time of the first IPC and the second IPC is not synchronized, at which time the server can send an encoding restart message to the first IPC again, and then the first IPC performs encoding restart again according to the encoding restart message. This cycle continues until the frame time of the first IPC and the second IPC is synchronized. That is, the first IPC needs to perform multiple encoding restarts to achieve frame time synchronization with the second IPC.

[0211] That is, in the embodiment of Figure 4 , the first IPC needs to perform one or more encoding restarts to achieve frame time synchronization with the second IPC.

[0212] Figure 5 The synchronization process is shown in the example.Figure 5 In an embodiment of the method, the encoding adjustment indication is an encoding flush message.

[0213] S501, at the beginning, the IPC reads an original image to be encoded.

[0214] S501 includes S501a and S501b, for example.

[0215] S501a, at the beginning, the first IPC reads an original image to be encoded.

[0216] S501b, at the beginning, the second IPC reads an original image to be encoded.

[0217] S502, the IPC encodes the original image into an image frame, and determines time information corresponding to the image frame.

[0218] S502 includes S502a and S502b, for example.

[0219] S502a, the first IPC encodes the original image into an image frame, and determines time information corresponding to the image frame.

[0220] S502b, the second IPC encodes the original image into an image frame, and determines time information corresponding to the image frame.

[0221] S501-S502 can refer to the description of S201-S202 above, and will not be described here again, for example.

[0222] S503, the IPC sends the image frame and the time information corresponding to the image frame to the server.

[0223] S503 includes S503a and S503b, for example.

[0224] S503a, the first IPC sends the image frame and the time information corresponding to the image frame to the server.

[0225] S503b, the second IPC sends the image frame and the time information corresponding to the image frame to the server.

[0226] S503 can refer to the description of S303 above, and will not be described here again, for example.

[0227] S504, the server determines whether the first IPC and the second IPC are synchronized according to the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC.

[0228] S304 can refer to the description of S204 above, and will not be described here again, for example.

[0229] S505, when the server determines that the first IPC is not synchronized with the second IPC, the server sends a coding refresh message to the first IPC.

[0230] For example, the coding refresh message is used to trigger the first IPC to perform frame refresh, and is used to trigger the coding component in the first IPC to perform frame refresh.

[0231] S506, the first IPC performs frame refresh according to the coding refresh message.

[0232] For example, after the first IPC receives the coding refresh message, the first IPC can trigger the coding component to perform frame refresh, i.e., trigger the coding component to immediately encode an image frame. At this time, the coding component can immediately obtain the next frame of original image to be encoded from the image cache, and then perform S502-S503. In this way, the time for the first IPC to read the next frame of original image to be encoded from the image cache is adjusted to the current time.

[0233] It should be noted that in one case, after the first IPC performs this coding refresh, the time for the first IPC to read the next frame of original image to be encoded from the image cache is the same as the time for the second IPC to read the next frame of original image to be encoded from the image cache. In this way, after the first IPC performs S501a-S503a again and the second IPC performs S501b-S503b again, the server can receive the next image frame and the time information corresponding to the image frame sent by the first IPC, and the next image frame and the time information corresponding to the image frame sent by the second IPC, and can determine that the frame time of the first IPC is synchronized with the frame time of the second IPC, and no longer needs to generate a coding refresh message. That is, the first IPC can achieve synchronization of the frame time with the second IPC after one frame refresh.

[0234] In one case, after the first IPC performs this coding refresh, the time for the first IPC to read the next frame of original image to be encoded from the image cache is still different from the time for the second IPC to read the next frame of original image to be encoded from the image cache. In this way, after the first IPC performs S501a-S503a again and the second IPC performs S501b-S503b again, the server can receive the next image frame and the time information corresponding to the image frame sent by the first IPC, and the next image frame and the time information corresponding to the image frame sent by the second IPC. The server can determine that the frame time of the first IPC is not synchronized with the frame time of the second IPC, and can send a coding refresh message to the first IPC again. Then, the first IPC performs coding refresh again according to the coding refresh message. This cycle continues until the frame time of the first IPC is synchronized with the frame time of the second IPC. That is, the first IPC can achieve synchronization of the frame time with the second IPC after multiple frame refreshes.

[0235] That is, in the embodiment of the application, the first IPC needs to realize frame time synchronization with the second IPC through one or more frame refreshes. Figure 5

[0236] It should be noted that the existing IPC has a coding restart function and a coding refresh function, so it can perform S406 in the embodiment of the application and S506 in the embodiment of the application. In this way, multiple IPCs cooperate with the server according to the steps in the embodiment of the application or cooperate with the server according to the steps in the embodiment of the application, so as to realize frame time synchronization. Figure 4 Figure 5 Figure 4 Figure 5 It should be noted that the server can periodically synchronize the frame times of multiple IPCs according to a set time length, and the set time length can be set according to requirements and can be greater than a preset period, which is not limited in the application.

[0237]

[0238] Figure 6 A schematic diagram of a synchronization system is shown for illustration. In the embodiment of the application, Figure 6 Figure 6 The synchronization system in the embodiment of the application can include multiple IPCs and a server, and the multiple IPCs can include at least one first IPC and one second IPC, and the second IPC is a reference for whether the frame times of the first IPC and the second IPC are synchronized; wherein,

[0239] The first IPC is configured to read a first original image to be encoded; encode the first original image into a first image frame, and determine time information corresponding to the first image frame; send the first image frame and the time information corresponding to the first image frame to the server; and receive a coding adjustment instruction; and perform coding adjustment according to the coding adjustment instruction.

[0240] The second IPC is configured to read a second original image to be encoded; encode the second original image into a second image frame, and determine time information corresponding to the second image frame; and send the second image frame and the time information corresponding to the second image frame to the server.

[0241] The server is configured to receive the first image frame and the time information corresponding to the first image frame sent by the first IPC, and receive the second image frame and the time information corresponding to the second image frame sent by the second IPC; and when it is determined that the frame times of the first IPC and the second IPC are not synchronized according to the time information corresponding to the first image frame sent by the first IPC and the time information corresponding to the second image frame sent by the second IPC, send a coding adjustment instruction to the first IPC.

[0242] ​​​​​​The server is configured to send a coding control message to the first IPC, wherein the coding control message comprises an offset time determined according to a time difference between time information corresponding to the first image frame sent by the first IPC and time information corresponding to the second image frame sent by the second IPC, and the coding control message is used to trigger the first IPC to adjust a time for acquiring a next frame of the first original image to be coded according to the offset time.

[0243] The server is configured to send a coding restart message to the first IPC, and the coding restart message is used to trigger the first IPC to perform coding restart.

[0244] The server is configured to send a coding refresh message to the first IPC, and the coding refresh message is used to trigger the first IPC to perform frame refresh.

[0245] The server is configured to send a coding restart message to the first IPC, and the coding restart message is used to trigger the first IPC to perform coding restart.

[0246] The first IPC is configured to determine the time information corresponding to the first image frame according to a time for reading the first original image to be coded.

[0247] The first IPC is configured to encapsulate the time information corresponding to the first image frame into an extension field of the first image frame and send the encapsulated first image frame.

[0248] The coding adjustment indication is a coding control message, the coding control message comprises an offset time determined according to a time difference between time information corresponding to the first image frame sent by the first IPC and time information corresponding to the second image frame sent by the second IPC, and the first IPC is configured to adjust a time for reading a next frame of the first original image to be coded according to the offset time.

[0249] The coding adjustment indication is a coding restart message, and the first IPC is configured to perform coding restart according to the coding restart message.

[0250] The coding adjustment indication is a coding refresh message, and the first IPC is configured to perform frame refresh according to the coding refresh message.

[0251] The second IPC is configured to determine the time information corresponding to the second image frame according to a time for reading the second original image to be coded.

[0252] The second IPC is configured to encapsulate the time information corresponding to the second image frame into an extension field of the second image frame and send the encapsulated second image frame.

[0253] It should be understood that the technical effects of the steps performed by the server and the IPC can refer to the technical effects described in any of the embodiments of Figure 2 、 Figure 3a 、 Figure 4 and Figure 5 , which will not be repeated here.

[0254] In one example, Figure 7 a schematic block diagram of an apparatus 700 is shown. The apparatus 700 can include a processor 701 and a transceiver / transceiver pin 702, and optionally further include a memory 703.

[0255] The various components of the apparatus 700 are coupled together by a bus 704, which can include a data bus, a power bus, a control bus, and a state signal bus. For the sake of clarity, the various buses are illustrated in Figure as the bus 704.

[0256] Optionally, the memory 703 can be used to store instructions in the foregoing method embodiments. The processor 701 can be used to execute the instructions in the memory 703, and control the receiving pin to receive signals and the transmitting pin to transmit signals.

[0257] The apparatus 700 can be an electronic device or a chip of an electronic device in the above method embodiments.

[0258] All related content of the steps involved in the above method embodiments can be referred to the function description of the corresponding function module, which will not be repeated here.

[0259] The embodiment also provides a computer readable storage medium, which stores computer instructions. When the computer instructions run on an electronic device, the electronic device executes the related method steps described above to implement the synchronization method in the above embodiment.

[0260] The embodiment also provides a computer program product, which makes a computer execute the related steps described above to implement the synchronization method in the above embodiment when the computer program product runs on the computer.

[0261] In addition, the embodiment of the present application also provides an apparatus, which can be a chip, a component or a module. The apparatus can include a processor and a memory connected to each other. The memory is used to store computer execution instructions. When the apparatus runs, the processor can execute the computer execution instructions stored in the memory to make the chip execute the synchronization method in the above method embodiments.

[0262] The electronic device, the computer readable storage medium, the computer program product or the chip provided in the embodiment are used for executing the corresponding method provided above, and thus the beneficial effects achieved by the electronic device, the computer readable storage medium, the computer program product or the chip can refer to the beneficial effects of the corresponding method provided above, which will not be described here again.

[0263] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0264] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0265] The units described as separate components can or can not be physically separate, and the components shown as units can be one physical unit or multiple physical units, that is, can be located in one place, or can be distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0266] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0267] Any content of each embodiment of the present application, and any content of the same embodiment, can be freely combined. Any combination of the above is within the scope of the present application.

[0268] If the integrated unit is implemented in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application essentially or say the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, includes several instructions to make a device (which can be a single-chip microcomputer, a chip, etc.) or a processor execute all or part of the steps of the various embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0269] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are only illustrative, not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope of protection of the claims, and all of them belong to the protection of the present application.

[0270] The steps of the method or algorithm described in combination with the disclosure of the embodiments of the present application can be implemented in hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a compact disc (CD-ROM), or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0271] Those skilled in the art can understand that the functions described in the embodiments of the present application in the one or more examples above can be implemented in hardware, software, firmware or any combination thereof. When implemented in software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium. The computer readable medium includes computer readable storage medium and communication medium, wherein the communication medium includes any medium that facilitates transfer of a computer program from one place to another. The storage medium can be any available medium accessible by a general purpose or special purpose computer.

[0272] The embodiments of the present application are described above in conjunction with the drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative rather than limiting, and those of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims.

Claims

1. A method of synchronizing, comprising: The method comprises: The server receives image frames sent by a plurality of network cameras IPCs and time information corresponding to the image frames, wherein the plurality of IPCs comprise a first IPC and a second IPC, the second IPC is a reference for whether frame time of the first IPC and the second IPC is synchronized, the image frames comprise a first image frame and a second image frame, the first image frame is obtained by encoding a first original image by the first IPC, and the second image frame is obtained by encoding a second original image by the second IPC; When the server determines that the frame time of the first IPC and the second IPC is not synchronized according to the time information corresponding to the first image frame sent by the first IPC and the time information corresponding to the second image frame sent by the second IPC, the server sends an encoding adjustment instruction to the first IPC, and the encoding adjustment instruction is used to trigger the first IPC to perform encoding adjustment, so that the time for the first IPC to read a first original image of a next frame to be encoded is adjusted.

2. The method of claim 1, wherein, The server sends an encoding adjustment instruction to the first IPC, comprising: The server sends an encoding control message to the first IPC; The encoding control message comprises an offset time, and the offset time is determined by the server according to a time difference between the time information corresponding to the first image frame sent by the first IPC and the time information corresponding to the second image frame sent by the second IPC; and the encoding control message is used to trigger the first IPC to adjust the time for acquiring a first original image of a next frame to be encoded according to the offset time.

3. The method of claim 1, wherein, The server sends an encoding adjustment instruction to the first IPC, comprising: The server sends an encoding restart message to the first IPC, and the encoding restart message is used to trigger the first IPC to perform encoding restart.

4. The method of claim 1, wherein, The server sends an encoding adjustment instruction to the first IPC, comprising: The server sends an encoding flush message to the first IPC, and the encoding flush message is used to trigger the first IPC to perform frame flush.

5. The method according to any one of claims 1 to 4, characterized in that, The determination that the frame time of the first IPC and the second IPC is not synchronized comprises: The server determines that a time difference between the time information corresponding to the first image frame sent by the first IPC and the time information corresponding to the second image frame sent by the second IPC is greater than a threshold value.

6. The method of any one of claims 1 to 4, wherein The time information corresponding to the image frame sent by each IPC in the plurality of IPCs is the time for the each IPC to read an original image to be encoded.

7. A method of synchronizing, comprising: The method comprises: A first network camera IPC reads a first original image to be encoded; The first IPC encodes the first original image into a first image frame, and the first IPC determines time information corresponding to the first image frame; The first IPC sends the first image frame and the time information corresponding to the first image frame; The first IPC receives an encoding adjustment indication, the encoding adjustment indication is generated and sent by the server when the first IPC and the second IPC are determined to be out of synchronization in frame time according to time information corresponding to a first image frame sent by the first IPC and time information corresponding to a second image frame sent by the second IPC, the second IPC is a reference for whether the first IPC and the second IPC are synchronized in frame time, and the second image frame is obtained by encoding a second original image to be encoded read by the second IPC; The first IPC performs encoding adjustment according to the encoding adjustment indication, so that the time of reading the next frame of the first original image to be encoded is adjusted.

8. The method of claim 7, wherein, The first IPC determines the time information corresponding to the first image frame, comprising: The first IPC determines the time information corresponding to the first image frame according to the time of reading the first original image to be encoded.

9. The method according to claim 7 or 8, characterized in that, The first IPC sends the first image frame and the time information corresponding to the first image frame, comprising: The first IPC encapsulates the time information corresponding to the first image frame into the extension field of the first image frame, and sends the encapsulated first image frame.

10. The method according to claim 7 or 8, characterized in that, The encoding adjustment indication is an encoding control message, the encoding control message includes an offset time, and the offset time is determined by the server according to the time difference between the time information corresponding to the first image frame sent by the first IPC and the time information corresponding to the second image frame sent by the second IPC; The first IPC performs encoding adjustment according to the encoding adjustment indication, comprising: The first IPC adjusts the time of reading the next frame of the first original image to be encoded according to the offset time.

11. The method of claim 7 or 8, wherein, The encoding adjustment indication is an encoding restart message, and the first IPC performs encoding adjustment according to the encoding adjustment indication, comprising: The first IPC performs encoding restart according to the encoding restart message.

12. The method of claim 7 or 8, wherein, The encoding adjustment indication is an encoding refresh message, and the first IPC performs encoding adjustment according to the encoding adjustment indication, comprising: The first IPC performs frame refresh according to the encoding refresh message.

13. A synchronization system, characterized by The synchronization system comprises: a plurality of network cameras IPC and a server, the plurality of IPCs comprise a first IPC and a second IPC, and the second IPC is a reference for whether the first IPC and the second IPC are synchronized in frame time; The first IPC is configured to read a first original image to be encoded, encode the first original image into a first image frame, determine time information corresponding to the first image frame, send the first image frame and the time information corresponding to the first image frame to the server, and receive an encoding adjustment indication, and perform encoding adjustment according to the encoding adjustment indication to adjust the time of reading the next frame of the first original image to be encoded. The second IPC is configured to read a second original image to be encoded, encode the second original image into a second image frame, determine time information corresponding to the second image frame, and send the second image frame and the time information corresponding to the second image frame to the server. The server is configured to receive the first image frame and the time information corresponding to the first image frame sent by the first IPC, and receive the second image frame and the time information corresponding to the second image frame sent by the second IPC, and send an encoding adjustment instruction to the first IPC when it is determined that the frame time of the first IPC is not synchronized with the frame time of the second IPC according to the time information corresponding to the first image frame sent by the first IPC and the time information corresponding to the second image frame sent by the second IPC.

14. An Internet Protocol Camera (IPC), comprising: A synchronization method according to any one of claims 7 to 12.

15. An electronic device, comprising: Comprising: a memory and a processor, the memory being coupled to the processor; the memory stores program instructions which, when executed by the processor, cause the electronic device to perform the synchronization method according to any one of claims 1 to 6.

16. An electronic device, comprising: Comprising: a memory and a processor, the memory being coupled to the processor; the memory stores program instructions which, when executed by the processor, cause the electronic device to perform the synchronization method according to any one of claims 7 to 12.

17. A chip, characterized by Comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from the memory of the electronic device and send the signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the synchronization method according to any one of claims 1 to 6.

18. A chip, characterized by Comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from the memory of the electronic device and send the signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the synchronization method according to any one of claims 7 to 12.

19. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program which, when running on a computer or a processor, causes the computer or the processor to perform the synchronization method according to any one of claims 1 to 12.

20. A computer program product, characterised in that, The computer program product comprises a software program which, when executed by a computer or a processor, causes the steps of the method according to any one of claims 1 to 12 to be performed.

Images