An MIPI-based adaptive real-time image transmission method

Abstract

The present invention provides a method for adaptive real-time image transmission based on MIPI, including: receiving data from multiple image sources using an image transmission device and obtaining the image line length; determining the transmission period based on the proportional relationship of the image line lengths; within the transmission period, setting the same number of transmission slices according to the number of rows of data to be sent included in the image source corresponding to the minimum image line length in the transmission period, and dividing the data to be sent of each image source within the transmission period into different transmission slices, and sequentially sending them by the image transmission device at intervals of the minimum image line length. The transmission method of the present invention determines the transmission period based on the relationship of the image line lengths of the image sources, divides the data of each image source within the transmission period into different transmission slices according to the line length proportional relationship, realizes adaptive division of the transmission slice length, and can achieve "interleaved" transmission during the blank waiting time of the transmission slice, shortening the additional waiting time, thereby improving the real-time image frame rate that the image transmission device can support.

Description

Technical Field

[0001] This application relates to the field of data transmission, and particularly to an adaptive image real-time transmission method based on MIPI. Background Art

[0002] With the rapid development of computer vision technology, it has been widely applied in fields such as public security and autonomous driving. At the same time, it also puts forward requirements such as multi-angle and good real-time performance for image transmission. With the increase in the number of accessed image sources, the number of MIPI interfaces of the video access ports of processing or display chips is limited. A dedicated chip is required to receive multiple video streams and splice them into one stream by hardware and send it out.

[0003] Common image transmission devices use segmented transmission. Multiple image sources are divided into one or more categories according to information such as frame rate. Multiple categories are transmitted in a cyclic manner by slices. If the frame rate relationship is not an integer multiple, the greater the difference, the more categories there are, and at the same time, the more hardware logic resources are consumed. The real-time transmission of images from multiple image sources faces many challenges, such as the synchronization problem between video images, redundancy and complementarity between video images, and real-time performance. When facing image sources with different resolutions, most of the existing image real-time methods use the method based on the time consumed for transmitting the image with the maximum resolution, which additionally occupies a large amount of bandwidth and system resources, and the real-time performance is easily affected.

[0004] At present, many MIPI-based image transmission devices have the function of combining and transmitting multi-channel image data. Usually, it is sent by sub-transmission chips, and at most one row of image data is transmitted from each image source within each transmission chip. Therefore, the length of the transmission chip needs to be greater than the sum of the single-row transmission times of all image sources. Different from the application scenarios with low real-time requirements, in the real-time transmission scenario, since the images are not pre-stored in several frames, image data available for transmission cannot be obtained every time an image is needed. For example, when transmitting multi-image source data with a large difference between the transmission line length and the number of lines, the image data will be sliced by rows. When the transmission line length of an image source with a shorter transmission line length is less than the length of the transmission chip, after sending multiple transmission chips, this image source has temporarily stored several times the number of image data to be sent, consuming a large amount of hardware resources for storage, increasing the design area and design cost. In the early stage of this frame transmission, the image source with a shorter line length continuously stores image data and waits for the transmission of the longer line length, and cannot be sent immediately; in the later stage of this frame transmission, the image source with a longer line length has finished sending due to fewer lines, but needs to wait for the image source with a shorter line length to continue sending the cached image lines. Not only does the image source with a short line length need to cache multiple lines of data, resulting in waste of hardware resources, but also the image source with a long line length needs to wait for several blank times of lines additionally, thereby reducing the supported frame rate. At the same time, if the frame rates of multiple image sources are not equal, a multi-classification form needs to be adopted to ensure that the number of line transmissions matches the frame rate. The greater the difference multiple or when it is not an integer multiple, the more classifications are required, and the more hardware logic resources are consumed. Therefore, most image transmission devices limit the maximum number of slice types. For multi-image sources without an integer multiple relationship, there may be a shortage of classification resources.

[0005] It should be noted that the above introduction of the technical background is only for the convenience of clearly and completely explaining the technical solutions of the present application and facilitating the understanding of those skilled in the art. It cannot be considered that the above technical solutions are well-known to those skilled in the art just because these solutions are described in the background art part of the present application. Summary of the Invention

[0006] To solve the above problems, the following provides an MIPI-based adaptive image real-time transmission method, including the following steps:

[0007] Use an image transmission device to simultaneously receive data from multiple image sources and obtain the image line lengths corresponding to the multiple image sources;

[0008] Determine a transmission cycle based on the proportional relationship of the multiple image line lengths, and each image source has at least one row of data to be sent corresponding to the transmission cycle;

[0009] During the transmission cycle, the same number of transmission slices are set according to the number of rows of the data to be transmitted included in the image source corresponding to the minimum image line length in the transmission cycle, and the data to be transmitted of each image source in the transmission cycle are divided into different transmission slices, and are sequentially sent by the image transmission device at intervals of the minimum image line length.

[0010] The transmission method of the present invention determines the transmission cycle based on the relationship of the image line lengths of the image sources, divides the data of each image source in the transmission cycle into different transmission slices according to the line length ratio relationship, realizes the adaptive division of the transmission slice length, and can achieve "insertion" transmission during the blank waiting time of the transmission slice, shortening the extra waiting time, thereby improving the real-time image frame rate that the image transmission device can support.

[0011] Taking the least common multiple of the image line lengths of the multiple image sources as a transmission cycle, each image source has at least one row of data to be transmitted in the transmission cycle. Taking the least common multiple as the transmission cycle is the smallest repeated cycle, and the optimal solution of global transmission can be achieved after the optimal division within this cycle.

[0012] The image transmission device is used to receive the data of multiple image sources simultaneously and cache them, and the data cached by each image source is less than 2 rows. Caching one row of data for different image sources realizes the reasonable division of data with different line lengths; restricting the cached data to be less than 2 rows reduces the cache depth, saves hardware resources, and avoids the waste of hardware resources caused by the need to temporarily store multiple rows of image data for image sources with shorter line lengths in the prior art.

[0013] Dividing the data to be transmitted of each image source in the transmission cycle into different transmission slices includes: dividing the received data of the image source with the minimum image line length into the next transmission slice for sending; based on the corresponding ratio relationship i between the image line length of each of the remaining image sources and the minimum image line length, dividing the received data of each of the remaining image sources into any one of the subsequent i transmission slices for sending. Based on the data with the shortest line length as the benchmark, each time a row is received and full, it is sent, and according to the ratio relationship i, the remaining data can be sent within i rows. With this rule, the data to be transmitted of different image sources can be reasonably divided according to the ratio relationship i to minimize the duration of the transmission slice as much as possible.

[0014] The ratio relationship i is the floor value of the ratio of the image line length of each image source to the minimum image line length. Rounding the line length ratio can simplify the division process, and using the floor value can avoid excessive accumulation of cached data and save hardware resources.

[0015] When there are four image sources and the ratio i of the image line lengths of the remaining three image sources to the minimum image line length is greater than or equal to 3: Divide the received data of the image source with the minimum image line length and send it in the next transmission slice; Divide the received data of the remaining three image sources non-overlappingly and send it in any one of at least three subsequent transmission slices. Specific optimization is carried out for the common four-image-source transmission, providing an optimal division method under four image sources, minimizing the transmission slice length as much as possible, so as to allow supporting higher frame rate image sources.

[0016] When there are four image sources and the ratio i of the image line lengths of the remaining three image sources to the minimum image line length is such that 2 are greater than or equal to 3 and 1 is 2: Divide the received data of the image source with the minimum image line length and send it in the next transmission slice; Divide the received data of the corresponding image source with the ratio i of 2 and send it in any one of the next 2 transmission slices; Divide the received data of the corresponding 2 image sources with the ratio i greater than or equal to 3 non-overlappingly and send it in any one of at least three subsequent transmission slices. Specific optimization is carried out for the common four-image-source transmission, providing an optimal division method under four image sources, minimizing the transmission slice length as much as possible, so as to allow supporting higher frame rate image sources.

[0017] When there are four image sources and the ratio i of the image line lengths of the remaining three image sources to the minimum image line length is such that 1 is greater than or equal to 3 and 2 are 2: Divide the received data of the image source with the minimum image line length and send it in the next transmission slice; Divide the received data of the corresponding 2 image sources with the ratio i of 2 non-overlappingly and send it in any one of the next 2 transmission slices; Divide the received data of the corresponding image source with the ratio i greater than or equal to 3 and send it in any one of at least three subsequent transmission slices. Specific optimization is carried out for the common four-image-source transmission, providing an optimal division method under four image sources, minimizing the transmission slice length as much as possible, so as to allow supporting higher frame rate image sources.

[0018] When there are four image sources and the ratio i of the image line lengths of the remaining three image sources to the minimum image line length is all 2: Divide the received data of the image source with the minimum image line length and send it in the next transmission slice; Divide the received data of any 2 corresponding image sources with the ratio i of 2 non-overlappingly and send it in any one of the next 2 transmission slices; Divide the received data of the remaining 1 corresponding image source with the ratio i of 2 and send it in any one of the next 2 transmission slices. Specific optimization is carried out for the common four-image-source transmission, providing an optimal division method under four image sources, minimizing the transmission slice length as much as possible, so as to allow supporting higher frame rate image sources.

[0019] When there are four image sources, and there is a ratio relationship i of 1 between the image line lengths of the remaining three image sources and the minimum image line length: the received data of the image source with the minimum image line length and the received data of the corresponding image source with the ratio relationship i of 1 are simultaneously divided and sent in the next transmission slice. Specific optimizations are carried out for the common four-image-source transmission, providing an optimal partitioning method under four image sources, minimizing the length of the transmission slice as much as possible, thereby allowing support for higher frame rate image sources.

[0020] Compared with the prior art, the beneficial effects of the present invention mainly include the following: 1) This method can adaptively allocate the optimal slice transmission method according to the image line length relationship of multiple image sources; 2) There is no limit to the frame rate relationship of multiple image sources; 3) Compared with the single-slice transmission method according to the frame rate, the length of the transmission slice is reduced, thereby allowing support for higher frame rate image sources; 4) Compared with the single-slice transmission method according to the frame rate, the requirement for the cache depth is reduced, thereby saving hardware resources; 4) Compared with the multi-slice transmission method, the transmission slice is dynamically configured, and it is not restricted by the transmission slice classification resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the specific embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0022] Figure 1 It is a schematic diagram of the real-time image transmission method provided by the present invention with single-slice segmentation according to the frame rate.

[0023] Figure 2 It is a schematic diagram of the real-time image transmission method provided by the present invention with multi-slice segmentation according to the line length. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] Regarding the foregoing and other technical contents, features and effects of the present invention, they will be clearly presented in the following detailed description of a preferred embodiment in conjunction with the reference drawings. The directional terms mentioned in the following embodiments, such as: up, down, left, right, front or back, etc., are only for reference to the directions of the attached drawings. Therefore, the directional terms used are for illustration and not for limiting the present invention.

[0025] The following will elaborate on each embodiment of the present application in conjunction with the drawings. However, those of ordinary skill in the art can understand that in each embodiment of the present application, many technical details are proposed for the convenience of readers to understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions required to be protected by the present application can still be realized.

[0026] The steps in the following embodiments do not correspond one by one to the content of the invention.

[0027] In the prior art, an MIPI-based image transmission device has the function of combining and transmitting multi-channel image data. It usually uses split transmission chips for transmission, and each image source in each transmission chip transmits at most one row of image data.

[0028] Specifically, the image transmission device has multiple data input interfaces for receiving data transmitted from different image sources. First, multiple image sources are divided into one or more types of transmission chips according to information such as frame rate for transmission. For example, if the image sources include two types of image data with frame rates of 60 and 30, they usually need to be divided into 2 types of transmission chips. One type of transmission chip is only used to transmit images with a frame rate of 60, and the other type of transmission chip is used to transmit the mixture of images with frame rates of 60 and 30, so that all of them can be evenly transmitted. The greater the difference in frame rate relationships, or if they are not integer multiples, the more types of transmission chips are ultimately required.

[0029] Furthermore, for image sources with the same frame rate, currently, the method of single-chip segmentation is usually adopted. Refer to Figure 1 As shown, it is a real-time image transmission method with single-chip segmentation according to frame rate in the prior art. When performing single-chip segmentation according to frame rate, that is, only divided into one type of transmission chip, and each image source in each transmission chip transmits at most one row of image data. It can be understood that the time occupied by each row of the image in the image source is called the image line length, the number of rows included in each frame of the image is called the image line number, and the number of image frames included in the unit time is called the image frame rate. The product of the image line length, the image line number, and the image frame rate is the unit time. In the MIPI protocol, it is stipulated that data is sent through the method of short packets + long packets, that is, first send a short packet to declare the end of the previous row and the start of the next row, and then send a long packet to carry image data and error correction information. Then the total time for sending short packets and long packets is the shortest image line transmission time stipulated by the MIPI protocol. Only when the image line length is greater than this time can it be completely sent through the MIPI protocol.

[0030] In Figure 1 , there are 4 image sources with the same frame rate: image source a, image source b, image source c, and image source d. The relationship of the image line lengths of the 4 image sources is a:b:c:d = 3:1:3:2. Figure 1 In , image a1 and image a2 respectively represent the image line lengths of the first row and the second row of image source a, image b1 - image b6 respectively represent the image line lengths of the 1st - 6th rows of image source b, and so on.

[0031] In Figure 1Among them, before time T0, there is a transmission slice (assuming that the data of the 0th row of all 4 image sources have been received simultaneously at this moment), then the data of the 0th row of the 4 image sources can be sent as a transmission slice. It can be understood that the MIPI protocol stipulates that it should be sent by the method of short packets + long packets, that is, the short packets and long packets of the 0th row of image source a, the 0th row of image source b, the 0th row of image source c, and the 0th row of image source d are sent in sequence.

[0032] When the time reaches T0, at this time the image transmission device has completed sending the first transmission slice and has also received the complete data of image b1 (the 1st row of image source b) (the 1st rows of the other image sources are not fully received and cannot be sent). Between T0 and T1, only the data of image b1 is completely received. Therefore, the second transmission slice only includes the data of image b1; and so on. At T1, only the data of image b2 and image d1 are completely received (image b1 has been sent), then between T1 and T2, the third transmission slice only includes the data of image b2 and image d1. Subsequently, it continues in the above method. From Figure 1 It can be seen that for the transmission slices obtained by single-slice segmentation according to the frame rate, there are blank waiting times in many transmission slices.

[0033] It can be understood that because sometimes it is necessary to send one row of data of each of the four image sources in a transmission slice, it is first required that the length of the transmission slice be greater than or equal to the sum of the row transmission durations of the four image sources on MIPI. Then the minimum value of the transmission slice length (i.e., the minimum transmission slice length) is the sum of the row transmission durations of the four image sources on MIPI. It should be understood that the transmission slice length is related to the image row length of the supported image sources. The minimum transmission slice length is the minimum value of the image row lengths of the supported image sources. Therefore, reducing the minimum transmission slice length can allow image sources with smaller image row lengths to be supported.

[0034] The above transmission method limits the image sources with a large number of image rows and a short row length from sending image data, and there are two impacts:

[0035] 1. The requirements for the hardware storage device may become higher: When multiple image sources send row by row in sequence, when an image source with a long row length finishes sending one row of data, an image source with a short row length may have stored multiple rows of data, and as time accumulates, the storage quantity will increase, thus requiring a larger hardware storage depth;

[0036] 2. The frame rate of the supported image sources may become lower: Since the row transmission duration of the transmission slice is fixed and each image sends one row of data according to the priority, after an image source with a small number of rows finishes sending the data, it still needs to wait for the image source with a large number of rows to finish sending before it can enter the next transmission slice to transmit new frame image data. The extra waiting blank rows reduce the frame rate of the supported image sources.

[0037] Therefore, if we can use as few transmission slice classification resources as possible, shorten the transmission slice length at the same time, dynamically divide the transmission slice content, and send it "inserting into the gaps", it can not only adapt to any frame rate relationship, but also reduce the extra waiting time and achieve the maximum transmission rate of real-time image transmission.

[0038] Embodiment 1

[0039] The present invention provides a method for real-time adaptive image transmission based on MIPI, including the following steps:

[0040] Step 1: Receive image data from multiple image sources and obtain the image line length of the image sources.

[0041] The transmission method provided by the present invention is based on MIPI. It can be understood that an image transmission device based on MIPI has the function of combining and sending multi-channel image data. It usually sends data in transmission slices, and at most one line of image data of each image source can be transmitted within each transmission slice.

[0042] Specifically, the image data of multiple image sources are sent row by row in sequence, and the image transmission device has multiple data input interfaces, which are respectively used to receive the data transmitted by different image sources. In this embodiment, an example of the existence of 2 image sources is used for introduction. There are 2 image sources, namely image source a and image source b; the line lengths of the 2 image sources are obtained, and their image line lengths are Ta and Tb respectively. In this embodiment, the frame rate relationship of the 2 image sources is not limited, and classification is not required according to the frame rate first.

[0043] Step 2: Determine the transmission period based on the line length relationship of the image sources.

[0044] This method divides the transmission slices according to the line length ratio relationship of the image sources. First, a transmission period can be found, and then the transmission slice length and content distribution within this transmission period can be determined. Subsequently, cyclic transmission can be performed according to this transmission period, and finally the smooth realization of the entire transmission can be ensured.

[0045] Based on this idea, the least common multiple of the line lengths of image source a and image source b is selected as a transmission period. Specifically, take the least common multiple X of the image line lengths Ta and Tb, that is, X = A * Ta = B * Tb; where A and B correspond to the number of lines required for the 2 image sources to take the least common multiple for the line length. That is, within one transmission period, A lines of image source a and B lines of image source b are sent respectively, and cyclic transmission is performed in subsequent transmission periods.

[0046] Step 3: Based on the relationship of the image line lengths of the image sources, perform multi-slice division with the minimum image line length as the benchmark to determine the transmission slice length and content.

[0047] Taking the minimum image line length Tb of the two image sources as the benchmark, the corresponding proportional relationship i between the image line length of the image source and the minimum image line length is determined. In this embodiment, the line length ratio of the two image sources is Ta:Tb, which is further simplified to floor(Ta / Tb):1, where floor() is the floor function.

[0048] It can be understood that the purpose of this technical solution is to divide the multi-line data of different image sources into different transmission slices in one transmission cycle to minimize the length of the transmission slice as much as possible. As described above, the minimum length of the transmission slice is the minimum value of the image line length that can support the image source. Therefore, in order to support the above two image sources, the set transmission slice length should be less than or equal to the minimum line length Tb in the image source. In order to send all the B-line images b in one transmission cycle, at least B transmission slices are required, and each transmission slice contains one line of data of the image source b. Subsequently, as long as the data of the image source a is reasonably divided into each transmission slice.

[0049] The division method of the remaining one image source data can be carried out according to its proportional relationship of the image line length. It should be stated first that in order to ensure the transmission efficiency and reduce the cache space required to store the image data, this application stipulates that one line of image cache data can be reserved for each image source. Then, for the image source a, assuming that the value of floor(Ta / Tb) is i (i is greater than or equal to 1), then every time i transmission slices are sent, the data of the previous line of the image source a that has been received must be divided into the above i transmission slices and sent out (if it is not full of one line, it is regarded as empty and not sent).

[0050] Therefore, when there are two image sources, there must be a transmission slice that contains one line of data of both the image source a and the image source b at the same time. At this time, the transmission slice length can be taken as the sum of the line transmission durations of the image source a and the image source b. The following discusses the proportional relationship i of the line length in different cases:

[0051] 1) When i is greater than or equal to 1, the transmission slice length is the sum of the line transmission durations of the image source a and the image source b. The types and contents of the transmission slices are confirmed according to the actual situation. One possible division method is: for example, when Ta:Tb is exactly 1:1, the sending order in one transmission cycle is ab, ab, and ab, and this cycle is sent; one possible division method is: for example, when Ta:Tb is 3:2, the sending order in one cycle is b, ab, and ab, and this cycle is sent. There is no need to give more examples here.

[0052] Regarding the calculation method of the line transmission duration τ, first, the line transmission durations of the two image sources correspond to τ a and τ bIt should be understood that the MIPI transmission of images is divided into short packet transmission duration (Tsp) and long packet transmission duration (Tlp), that is, the line transmission duration of the image source is the corresponding Tsp + Tlp. Tsp is only related to the MIPI configuration, and Tlp is also related to the quantity of the transmitted image data. Then the line transmission durations of image sources a and b are respectively: τ a = Tsp + Tlpa, τ b = Tsp + Tlpb.

[0053] After the host computer calculates the ratio i of the image line length to the minimum image line length of each image source and corresponds it to the above types, the transmission slice length and the content transmitted in the transmission slice are obtained.

[0054] It can be understood that taking the image line length relationship Ta: Tb = 3: 1 as an example. The selected transmission period, that is, 1 * Ta = 3 * Tb, that is, within one transmission period, 1 line of image source a and 3 lines of image source b are sent respectively, and subsequent transmission periods can be cyclically transmitted accordingly. The ratio relationship of the image line lengths Ta: Tb is simplified to: floor(Ta / Tb): 1 = 3: 1. Corresponding to the above case two, that is: the transmission slice length = τ a + τ b ; the contents in the transmission slice are b, b, ab respectively, and are cyclically sent with this as the period.

[0055] Embodiment 2

[0056] Step 1, Receive the image data from multiple image sources and obtain the image line lengths of the image sources.

[0057] In this embodiment, three image sources are taken as an example for introduction. First, there are three image sources, namely image source a, image source b, and image source c; the line lengths of the three image sources are obtained, and their image line lengths correspond to Ta, Tb, and Tc respectively. In this embodiment, the frame rate relationship of the three image sources is not limited, and classification according to the frame rate is not required either.

[0058] Step 2, Determine the transmission period based on the line length relationship of the image sources.

[0059] This method is segmented according to the line length relationship. First, a transmission period can be found, and then the transmission slice length and content distribution within this transmission period can be determined. Then subsequent cyclic transmission can be carried out according to this transmission period, and ultimately the smooth realization of the entire transmission can be ensured.

[0060] Based on this idea, the least common multiple of the line lengths of image sources a - c is selected as a transmission period. Specifically, take the least common multiple X of the line lengths Ta, Tb, and Tc of the images, that is, X = A * Ta = B * Tb = C * Tc; where A, B, and C are the number of lines required for the three image sources to take the least common multiple for the line length. That is, within one transmission period, A lines of image source a, B lines of image source b, and C lines of image source c are sent respectively, and then the cycle transmission can be carried out in subsequent transmission periods.

[0061] Step 3: Based on the minimum image line length as a reference, perform multi - slice segmentation according to the relationship of the image line lengths of the image sources, and determine the transmission slice length and content.

[0062] Taking the minimum image line length among the three image sources (in this embodiment, Tc is the minimum value and Ta is the maximum value) as a reference, the proportional relationship Ta:Tb:Tc of the image line lengths of the three image sources can be simplified to: floor(Ta / Tc):floor(Tb / Tc):1; where floor() is the floor function.

[0063] It can be understood that the purpose of this technical solution is to divide the multi - line data of different image sources into different transmission slices within one transmission period to minimize the transmission slice length as much as possible. Then, as described above, the minimum length of the transmission slice is the minimum value of the image line lengths of the supported image sources. Therefore, to support the above three image sources, the set transmission slice length should be less than or equal to the minimum line length Tc of the image sources. To send all C lines of image c within one transmission period, at least C transmission slices are required, and each transmission slice contains one line of data of image source c. Subsequently, as long as the data of the remaining image sources is reasonably divided into each transmission slice.

[0064] The division method of the data of the remaining two image sources can be carried out according to their proportional relationship of the image line lengths. It should be stated first that to ensure the transmission efficiency and reduce the cache space required to store the image data, this application stipulates that one line of image cache data can be reserved for each image source. Then, for image source a, assuming that the value of floor(Ta / Tc) is 3, then every time 3 transmission slices are sent, the data of the previous line of image source a that has been received must be divided into the above 3 transmission slices and sent out (if it is not full of one line, it is regarded as empty and not sent). The same is true for image source b. That is, when the proportional relationship between the image line length and the minimum image line length is i (i is greater than or equal to 1), then every time i transmission slices are sent, the data of the previous line of the image source that has been received must be divided into the above i transmission slices and sent out (if it is not full of one line, it is regarded as empty and not sent).

[0065] Overall, to ensure that the set transmission slice length is as small as possible, the image sources a and b should be divided into different transmission slices as much as possible. The following discusses the proportional relationships of the image line lengths in different cases:

[0066] 1) If the proportional relationships i of the image line lengths of the two image sources to the minimum image line length are both greater than or equal to 2. That is, as long as two lines of data are sent in two transmission slices, it is inevitable that the two lines of data are not in the same transmission slice. Therefore, each transmission slice only needs to include at most two lines of data, and at this time, the transmission slice length can be taken as τ c + MAX(τ a , τ b ); The types and contents of the transmission slices are confirmed according to the actual situation. A possible division method is: for example, when Ta:Tb:Tc is exactly 2:2:1, the sending order within one transmission cycle is ca, cb, ca, and cb, and this cycle is repeated; A possible division method is: for example, when Ta:Tb:Tc is 3:2:1, the sending order within one cycle is ca, cb, c, cb, ca, and cb, and this cycle is repeated. There is no need to give more examples here.

[0067] 2) If there is a 1 in the proportional relationships i of the image line lengths of the two image sources to the minimum image line length. For the image source with i being 1, that is, each time image c is sent, the data of this image source also needs to be sent correspondingly, that is, one line of data of this image source should be included in each transmission slice (since the proportional relationship i is calculated using the floor function, not every time this image source has a complete line of data, but it is still considered to exist). Multiple image sources with i being 1 can be regarded as the same image source. For example, when the proportional relationships i of image source b and image source c are 1, image source b and image source c are regarded as one image source, and graphic source a is regarded as another image source, and then Example 1 can be referred to. It can be understood that the data of image source b and image source c should exist in each transmission slice, so the transmission slice length = τ a + τ b + τ c .

[0068] Example 3

[0069] Step 1: Receive the image data from multiple image sources and obtain the image line lengths of the image sources.

[0070] In this embodiment, an example with four image sources is introduced. First, there are four image sources, namely image source a, image source b, image source c, and image source d; Obtain the line lengths of the 4 image sources, and their image line lengths are Ta, Tb, Tc, and Td respectively. In this embodiment, the frame rate relationships of the four image sources are not limited, and it is not necessary to classify them according to the frame rate first.

[0071] Step 2: Determine the transmission period based on the line length relationship of the image sources.

[0072] This method performs segmentation based on the line length relationship. First, a transmission period can be found, and then the transmission slice length and content distribution within this transmission period can be determined. Subsequently, cyclic transmission can be carried out according to this transmission period, ultimately ensuring the smooth realization of the entire transmission.

[0073] Based on this idea, the least common multiple of the line lengths of image sources a - d is selected as a transmission period. Specifically, take the least common multiple X of the image line lengths Ta, Tb, Tc, and Td, i.e., X = A * Ta = B * Tb = C * Tc = D * Td; where A, B, C, and D are the number of lines required for the four image sources to take the least common multiple for the line length. That is, within one transmission period, A lines of image source a, B lines of image source b, C lines of image source c, and D lines of image source d are sent respectively, and cyclic transmission can be carried out in this way in subsequent transmission periods.

[0074] Step 3: Based on the relationship of the image line lengths of the image sources with the smallest image line length as the benchmark, perform multi - slice segmentation to determine the transmission slice length and content.

[0075] Taking the smallest image line length among the four image sources (in this embodiment, Td is the minimum value and Ta is the maximum value) as the benchmark, the ratio relationship Ta:Tb:Tc:Td of the image line lengths of the four image sources can be simplified to: floor(Ta / Td):floor(Tb / Td):floor(Tc / Td):1; where floor() is the floor function.

[0076] It can be understood that the purpose of this technical solution is to divide the multi - line data of different image sources into different transmission slices within one transmission period to minimize the transmission slice length as much as possible. As described above, the minimum length of the transmission slice is the minimum value of the image line lengths of the image sources that can be supported. Therefore, to support the above four image sources, the set transmission slice length should be less than or equal to the smallest line length Td in the image sources. To send all D lines of image d within one transmission period, at least D transmission slices are required, and each transmission slice contains one line of data of image source d. Subsequently, as long as the data of the remaining image sources is reasonably divided into each transmission slice.

[0077] The partitioning method for the remaining three image source data can be based on the proportional relationship of their image line lengths. It should be stated first that, to ensure transmission efficiency and reduce the cache space required for storing image data, this application stipulates that one line of image cache data can be reserved for each image source. Then, for image source a, assuming the value of floor(Ta / Td) is 3, then for every 3 transmission slices sent, the data of the previous line on image source a that has been received must be divided into the above 3 transmission slices and sent out (if not full of one line, it is regarded as empty and not sent out). The same applies to image source b and image source c. That is, when the proportional relationship between the image line length of the image source and the minimum image line length is i (i is greater than or equal to 1), then for every i transmission slices sent, the data of the previous line on image source a and image source c that has been received must be divided into the above i transmission slices and sent out (if not full of one line, it is regarded as empty and not sent out).

[0078] Overall, to ensure that the set transmission slice length is as small as possible, image sources a, b, and c should be divided into different transmission slices as much as possible. The following discusses the proportional relationship of transmission durations in different cases:

[0079] 1) If the proportional relationship i between the image line lengths of the 3 image sources and the minimum image line length is greater than or equal to 3 for all. That is, as long as 3 lines of data are sent out in 3 transmission slices, it is inevitable that the 3 lines of data are not in the same transmission slice with each other. Thus, each transmission slice only needs to include at most 2 lines of data, and at this time, the transmission slice length can be taken as τ d +MAX(τ a ,τ b ,τ c ). The types and contents of the transmission slices are confirmed according to the actual situation. One possible partitioning method is: for example, when Ta:Tb:Tc:Td is exactly 3:3:3:1, the sending order within one transmission cycle is da, db, and dc, and this is sent in a cycle. One possible partitioning method is: for example, when Ta:Tb:Tc:Td is exactly 6:3:3:1, the sending order within one transmission cycle is da, db, dc, db, dc, and d, and this is sent in a cycle. There is no need to give more examples here.

[0080] 2) If there is 1 with a proportional relationship i of 2 (for example, it is image source c) among the proportional relationships i between the image line lengths of the 3 image sources and the minimum image line length, and the remaining proportional relationships i are greater than or equal to 3 (such as image sources a and b). That is, the 2 image sources with a proportional relationship i greater than or equal to 3 will surely not be in the same transmission slice with each other, but for the one with a proportional relationship of 2 (image source c), it has to be sent out once in 2 transmission slices and must be divided into the same transmission slice with one of image sources a and b. Thus, each transmission slice only needs to include at most 3 lines of data (that is, one line of image source d + image source c + any one of the other image sources), and at this time, the transmission slice length can be taken as τ d +τc + MAX(τ a , τ b ). The type and content of the transmission slice are confirmed according to the actual situation. One possible division method is: for example, when Ta:Tb:Tc:Td is 3:3:2:1, the sending order within one transmission cycle is dc, da, dcb, da, dcb, and d, and this cycle is repeated. There are no more examples here.

[0081] 3) If there are 2 ratios i of the image line lengths of 3 image sources to the minimum image line length that are 2 (for example, image source b and image source c), and the remaining ratio i is greater than or equal to 3 (image source a). That is, the 2 image sources with a ratio i of 2 must not be in the same transmission slice with each other, and image source a should be divided into the same transmission slice with one of image source b and image source c. Therefore, each transmission slice only needs to include at most 3 lines of data (i.e., one line of image source d + image source a + any one of the other image sources), then at this time, the transmission slice length = τ d + τ a + MAX(τ b , τ c ). The type and content of the transmission slice are confirmed according to the actual situation. One possible division method is: for example, when Ta:Tb:Tc:Td is 3:2:2:1, the sending order within one transmission cycle is dba, dc, db, dca, db, and dc, and this cycle is repeated. There are no more examples here.

[0082] 4) If the ratio i of the image line lengths of 3 image sources to the minimum image line length is all 2. Then 2 of them can be separated into different transmission slices, and the data of the remaining one image source must be divided into the same transmission slice with any one of the previous two image sources. Therefore, each transmission slice only needs to include at most 3 lines of data (i.e., one line of image source d + any other 2 image sources), then at this time, the transmission slice length = τ d + MAX(τ a + τ b , τ a + τ c , τ b + τ c ). The type and content of the transmission slice are confirmed according to the actual situation. One possible division method is: when Ta:Tb:Tc:Td is 2:2:2:1, the sending order within one transmission cycle is dba and dc, and this cycle is repeated. There are no more examples here.

[0083] 5) If there is a 1 in the ratio relationship i of the image line lengths of the three image sources to the minimum image line length. For the image source with i equal to 1, that is, each time the image d is sent, the data of this image source also needs to be sent correspondingly, that is, one line of data of this image source should be included in each transmission slice (since the floor function is used to calculate i, not every time this image source has a complete line of data when sending, but it is still considered to exist).

[0084] 5.1) There is 1 image source with a ratio relationship i equal to 1 (such as image source b), and the other ratio relationships i are greater than or equal to 2. Each transmission slice only needs to include at most 3 lines of data (that is, one line of image source d + image source b + any one of the other image sources), then at this time, the transmission slice length can be taken as = τ b +τ d +MAX(τ a ,τ c ). The types and contents of the transmission slices are confirmed according to the actual situation. One possible division method is: when Ta:Tb:Tc:Td is 2:1:2:1, the sending order within one transmission cycle is dba and dbc, and this cycle is repeated for sending. There is no need to give more examples here.

[0085] 5.2) There are 2 image sources with a ratio relationship i equal to 1 (such as image source b and image source c). Each transmission slice only needs to include at most 4 lines of data, then at this time, the transmission slice length can be taken as = τ a +τ b +τ c +τ d . The types and contents of the transmission slices are confirmed according to the actual situation. One possible division method is: when Ta:Tb:Tc:Td is 2:1:1:1, the sending order within one transmission cycle is dbca and dbc, and this cycle is repeated for sending. There is no need to give more examples here.

[0086] Regarding the calculation method of the line transmission duration, first, the line transmission durations of the four image sources correspond to τ a 、τ b 、τ c and τ d . It should be understood that the MIPI transmission of images is divided into short packet transmission duration (Tsp) and long packet transmission duration (Tlp), that is, the line transmission duration τ of the image source is the corresponding Tsp + Tlp. Tsp is only related to the MIPI configuration, and Tlp is also related to the quantity of the transmitted image data. Then the line transmission durations of image source a to image source d are respectively: τ a =Tsp + Tlpa, τ b =Tsp + Tlpb, τ c =Tsp + Tlpc, τ d =Tsp + Tlpd.

[0087] After the host computer calculates the ratio relationship i between the image line lengths of each image source and the minimum image line length, and corresponds it to the above types, the transmission film length and the content transmitted in the transmission film are obtained.

[0088] For the sake of easy understanding, a practical situation is taken for introduction below. Specifically:

[0089] Step 1: Receive the image data from multiple image sources and obtain the image line lengths of the image sources.

[0090] Specifically, in this embodiment, as Figure 2 shown, assume that the relationship of the image line lengths of four image sources is Ta:Tb:Tc:Td = 3:1:3:2.

[0091] Step 2: Determine the transmission period based on the line length relationship of the image sources.

[0092] Select the least common multiple of the line lengths of image sources a - d as a transmission period, that is, 2*Ta = 6*Tb = 2*Tc = 3*Td. That is, within one transmission period, 2 lines of image source a, 6 lines of image source b, 2 lines of image source c, and 3 lines of image source d are sent respectively, and subsequent transmission periods can be cyclically transmitted in this way.

[0093] Step 3: Based on the relationship of the image line lengths of the image sources, perform multi - slice segmentation with the minimum image line length as the benchmark to determine the transmission slice length and content.

[0094] Taking the minimum image line length Tb among the four image sources as the benchmark, the ratio relationship Ta:Tb:Tc:Td of the image line lengths of the four image sources can be simplified to: floor(Ta / Tb):1:floor(Tc / Tb):floor(Td / Tb)= 3:1:3:2. Corresponding to Case 2 in this embodiment, that is:

[0095] For the ratio relationship i between the image line lengths of 3 image sources and the minimum image line length, there is 1 '2', and the rest of the ratio relationships i are greater than or equal to 3. Thus, each transmission slice only needs to include at most 3 lines of data, and at this time the transmission slice length = τ b +τ d +MAX(τ a ,τ c );A possible division method is bc, bd, ab, bc, bd, abd. Then at this time, the transmission slice length can be taken as = τ b +τ d +MAX(τ a ,τ c ).

[0096] In this embodiment, the transmission slice length is τ b +τ d +MAX(τa , τ c ), when segmenting by frame rate in the prior art, it is τ a + τ b + τ c + τ d , the length of each transmission slice is relatively reduced by (Tsp * 3 + Tlpb + Tlpd + MAX(Tlpa, Tlpc)) / (Tsp * 4 + Tlpa + Tlpb + Tlpc + Tlpd), which can be used to support higher - definition image data.

[0097] Meanwhile, since the length of the transmission slice is based on the shortest image line length, when mixing and transmitting image sources with any frame rate ratio and any line length ratio, the stored image line data is about 1 line, greatly reducing the required temporary storage space. The setting of the transmission slice is dynamically implemented by the upper computer. Even for an image transmission device with very few sharding and classification resources, it can achieve adaptive maximum transmission rate transmission.

[0098] Embodiment 4

[0099] Step 1: Receive image data from N image sources and obtain the image line lengths of the image sources.

[0100] It can be understood that the transmission method can be applicable to the case of having N image sources (N ≥ 2).

[0101] In this embodiment, an example of having N image sources is introduced. The N image sources are respectively image source 1 - image source N; obtain the image line lengths corresponding to the N image sources, and their image line lengths are correspondingly T1 to TN. In this embodiment, the frame rate relationship of the N image sources is not limited, and it is not necessary to classify according to the frame rate first.

[0102] Step 2: Determine the transmission period based on the line length relationship of the image sources.

[0103] This method performs segmentation according to the line length relationship. First, a transmission period can be found, and then the length and content distribution of the transmission slices within this transmission period can be determined. Subsequently, cyclic transmission can be carried out according to this transmission period, and ultimately the smooth implementation of the entire transmission can be ensured.

[0104] Based on this idea, select the least common multiple of the line lengths of image sources 1 - N as a transmission period. Specifically, take the least common multiple X of the image line lengths T1 to TN, that is, X = A1 * T1 = A2 * Tb = … = AN * TN; where A1 to AN are the number of lines required for the N image sources to take the least common multiple for the line length. That is, within one transmission period, send A1 lines of image source 1, A2 lines of image source 2, …, and AN lines of image source N, and subsequent transmission periods can be cyclically transmitted in this way.

[0105] Step 3: Based on the minimum image line length as a reference, perform multi-slice segmentation according to the relationship of the image line lengths of the image sources, and determine the transmission slice length and content.

[0106] Taking the minimum image line length among N image sources (in this embodiment, TN is the minimum value and T1 is the maximum value) as a reference, the proportional relationship of the image line lengths of N image sources can be simplified to: floor(T1 / TN):floor(T2 / TN):…:floor(TN-1 / TN):1 = i1:i2:i3:…:in-1:1; where floor() is the floor function.

[0107] It can be understood that the purpose of this technical solution is to divide the multi-line data of different image sources into different transmission slices in one transmission cycle to minimize the transmission slice length as much as possible. As described above, the minimum length of the transmission slice is the minimum value of the image line lengths of the supported image sources. Therefore, to support the above N image sources, the set transmission slice length should be less than or equal to the minimum line length TN in the image sources. To send all AN lines of image N in one transmission cycle, at least AN transmission slices are required, and each transmission slice contains one line of data of image source N. Subsequently, as long as the data of the remaining image sources is reasonably divided into each transmission slice. Specifically, first, the received data of the image source with the minimum image line length is divided into the next transmission slice for sending; the division method of the data of the remaining N-1 image sources can be based on their corresponding proportional relationship i, and the received data of each of the remaining image sources is divided into any one of the subsequent i transmission slices for sending. Generally speaking, to ensure that the set transmission slice length is as small as possible, image sources 1-N should be divided into different transmission slices as much as possible.

[0108] Some common English nouns or letters used in the present invention for the convenience of clear description are only for exemplary reference rather than limiting interpretation or specific usage, and the protection scope of the present invention should not be limited by their possible Chinese translations or specific letters.

[0109] It should also be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

Claims

1. A MIPI-based adaptive image real-time transmission method, characterized in that: The steps include: Using an image transmission device to simultaneously receive data from multiple image sources, and obtaining image line lengths corresponding to the multiple image sources; Determine a transmission period based on a ratio of the lengths of a plurality of the image lines, wherein each of the image sources has at least one line of data to be sent corresponding to the transmission period; In the transmission cycle, the same number of transmission slices are set according to the number of rows of data to be sent contained in the image source corresponding to the minimum image row length in the transmission cycle, and the data to be sent of each image source in the transmission cycle is divided into different transmission slices, which are sequentially sent by the image transmission device at a time interval of the minimum image row length; Dividing the data to be sent of each of the image sources in the transmission period into different transmission slices comprises: Divide the received data of the image source of the minimum image line length into the next transmission slice for transmission; Based on the corresponding proportional relationship i between the image line length of each of the remaining image sources and the minimum image line length, the received data of each of the remaining image sources is divided into any one of the subsequent i transmission slices for transmission.

2. The method for real-time adaptive image transmission based on MIPI according to claim 1, characterized in that: The lowest common multiple of the image row lengths of the plurality of image sources is taken as a transmission period, and each of the image sources corresponds to at least one row of data to be sent in the transmission period.

3. The method for real-time adaptive image transmission based on MIPI according to claim 1, characterized in that: An image transmission device is used to simultaneously receive and cache data from a plurality of image sources, and the data cached by each image source is less than 2 lines.

4. The method for real-time adaptive image transmission based on MIPI according to claim 1, characterized in that: The proportional relationship i is a floor value of the ratio of the image line length of each image source to the minimum image line length.

5. The method for real-time adaptive image transmission based on MIPI according to claim 4, characterized in that: When there are four image sources, and the ratio i between the image line lengths of the remaining three image sources and the minimum image line length is greater than or equal to 3: Divide the received data of the image source of the minimum image line length into the next transmission slice for transmission; The received data of the remaining three image sources are divided into any one of at least three subsequent transmission slices for transmission without overlap.

6. The method for real-time adaptive image transmission based on MIPI according to claim 4, characterized in that: When there are four image sources, and the ratio i between the image line lengths of the other three image sources and the minimum image line length is 2 greater than or equal to 3 and 1 is 2: Divide the received data of the image source of the minimum image line length into the next transmission slice for transmission; Divide the received data of the corresponding image source whose ratio i is 2 into any one of the following two transmission slices for transmission; The received data corresponding to the two image sources with the ratio i greater than or equal to 3 and not overlapping are divided into any one of at least three subsequent transmission slices for transmission.

7. The method for real-time adaptive image transmission based on MIPI according to claim 4, characterized in that: When there are four image sources, and the ratio i between the image line lengths of the other three image sources and the minimum image line length is 1 greater than or equal to 3 and 2 is 2: Divide the received data of the image source of the minimum image line length into the next transmission slice for transmission; Divide the non-overlapping received data of the two image sources corresponding to the ratio i of 2 into any one of the two subsequent transmission slices for transmission; The received data of the corresponding image source whose ratio i is greater than or equal to 3 is divided into any one of at least 3 subsequent transmission slices for transmission.

8. The method for real-time adaptive image transmission based on MIPI according to claim 4, characterized in that: When there are four image sources, and the ratio i between the image line lengths of the remaining three image sources and the minimum image line length is 2: Divide the received data of the image source of the minimum image line length into the next transmission slice for transmission; Divide the non-overlapping received data of any two corresponding image sources whose ratio i is 2 into any one of the two subsequent transmission slices for transmission; The received data of the remaining image source corresponding to the image source with a ratio i of 2 is divided into any one of the following two transmission slices and sent.

9. The method for real-time adaptive image transmission based on MIPI according to claim 4, characterized in that: When there are four image sources and the ratio i between the image line lengths of the remaining three image sources and the minimum image line length is 1: The received data of the image source with the minimum image row length and the received data of the corresponding image source with the ratio i being 1 are simultaneously divided into the next transmission slice for transmission.

Images