Adaptive image real-time transmission method based on MIPI

Abstract

The invention provides a self-adaptive image real-time transmission method based on MIPI (Mobile Industry Processor Interface). The self-adaptive image real-time transmission method comprises the following steps: receiving data of a plurality of image sources by using an image transmission device and acquiring an image line length; determining a transmission period based on the proportional relation of the image line length; and in the transmission period, according to the line number of to-be-sent data contained in the image source corresponding to the minimum image line length in the transmission period, setting the same number of transmission pieces, dividing the to-be-sent data of each image source in the transmission period into different transmission pieces, and sequentially sending the transmission pieces by the image transmission device by taking the minimum image line length as a time interval. According to the transmission method, the transmission period is determined based on the image line length relation of the image sources, the data of each image source in the transmission period is divided into different transmission pieces according to the line length proportion relation, the length of the transmission pieces is divided in a self-adaptive mode, insertion sending can be achieved in the blank waiting time of the transmission pieces, the extra waiting time is shortened, and the transmission efficiency is improved. Therefore, the real-time image frame rate supported by the image transmission device is improved.

Description

Technical Field

[0001] The present application relates to the field of data transmission, and in particular to a MIPI-based adaptive image real-time transmission method. Background Art

[0002] With the rapid development of computer vision technology, it has been widely used in public safety, autonomous driving and other fields, and has also put forward requirements for image transmission from multiple angles and good real-time performance. As the number of access image sources increases, the number of MIPI interfaces for video access ports of processing or display chips is limited, and dedicated chips are needed to receive multiple video streams and splice them into one by hardware.

[0003] Common image transmission devices use sliced ​​transmission, which divides multiple image sources into one or more categories according to information such as frame rate, and transmits multiple categories cyclically by slice. If the frame rate relationship is not an integer multiple, the greater the difference, the more categories there are, and the more hardware logic resources are consumed. The real-time transmission of images from multiple image sources faces many challenges, such as synchronization issues between video images, redundancy and complementarity between video images, and real-time performance. When faced with image sources of different resolutions, most existing real-time image methods use the method based on the time consumed to transmit the image with the highest resolution, which takes up a lot of additional bandwidth and system resources, and the real-time performance is easily affected.

[0004] At present, many MIPI-based image transmission devices have the function of merging and sending multi-channel image data, which is usually sent in transmission slices. Each image source in each transmission slice transmits at most one line of image data, so the transmission slice length needs to be greater than the sum of the single-line transmission time of all image sources. Different from the application scenarios with low real-time requirements, in the real-time transmission scenario, since the image is no longer pre-stored for several frames, the image data available for transmission is not available every time the image is needed. For example, when transmitting multi-image source data with a large difference between the line length and the number of lines, the image data will be sliced ​​by line. When the image source with a shorter line length transmits a line length less than the length of the transmission slice, after sending multiple transmission slices, the image source has temporarily stored several times the number of image data to be sent, consuming a large amount of hardware resources for storage, increasing the design area and design cost. In the early stage of this frame transmission, the image source with a shorter line length constantly temporarily stores image data and waits for the longer line length to be sent, and cannot be sent immediately; in the later stage of this frame transmission, the image source with a longer line length has been sent due to the small number of lines, but it is necessary to wait for the image source with a shorter line length to continue to send the cached image line. Not only does it require image sources with short line lengths to cache multiple lines of data, resulting in a waste of hardware resources, but it also requires image sources with long line lengths to wait for several lines of blank time, thereby reducing the supported frame rate. At the same time, if the frame rates of multiple image sources are not equal, they need to be divided into multiple categories to ensure that the number of line transmissions matches the frame rate. The larger the difference or the non-integer multiple, the more categories there are, and the more hardware logic resources are consumed. Therefore, most image transmission devices limit the maximum number of slice types. For multiple image sources that do not have an integer multiple relationship, the classification resources may be insufficient.

[0005] It should be noted that the above introduction to the technical background is only for the convenience of providing a clear and complete description of the technical solutions of the present application and for the convenience of understanding by those skilled in the art. It cannot be considered that the above technical solutions are well known to those skilled in the art simply because they are described in the background technology section of the present application. Summary of the invention

[0006] In order to solve the above problems, a MIPI-based adaptive image real-time transmission method is provided below, comprising the following steps: Using an image transmission device to simultaneously receive data from multiple image sources, and obtaining image line lengths corresponding to the multiple image sources; Determine a transmission period based on a ratio of the lengths of a plurality of the image lines, wherein each of the image sources has at least one line of data to be sent corresponding to the transmission period; In the transmission cycle, the same number of transmission slices are set according to the number of rows of data to be sent contained in the image source corresponding to the minimum image row length in the transmission cycle, and the data to be sent of each image source in the transmission cycle is divided into different transmission slices, which are sent out sequentially by the image transmission device at time intervals of the minimum image row length.

[0007] The transmission method of the present invention determines the transmission period based on the image line length relationship of the image source, divides the data of each image source in the transmission period into different transmission slices according to the line length ratio relationship, realizes adaptive division of the transmission slice length, and can realize "interspersed" transmission in the blank waiting time of the transmission slice, shortens the additional waiting time, and thus improves the real-time image frame rate that the image transmission device can support.

[0008] The least common multiple of the image row lengths of the plurality of image sources is used as a transmission cycle, and each image source corresponds to at least one row of data to be sent in the transmission cycle. The transmission cycle using the least common multiple as the minimum repetition cycle is a minimum repetition cycle, and the optimal solution for global transmission can be achieved after achieving optimal division within the cycle.

[0009] The image transmission device is used to simultaneously receive and cache the data of multiple image sources, and the data cached by each image source is less than 2 lines. The data of different image sources is cached in one line to achieve the reasonable division of data of different line lengths; limiting the cached data to less than 2 lines is to reduce the cache depth, save hardware resources, and avoid the waste of hardware resources caused by the image source with shorter line length needing to temporarily store multiple lines of image data in the prior art.

[0010] Dividing the data to be sent of each image source in the transmission cycle into different transmission slices includes: dividing the received data of the image source with the smallest image row length into the next transmission slice for transmission; dividing the received data of each remaining image source into any one of the subsequent i transmission slices for transmission based on the corresponding ratio i between the image row length of each remaining image source and the smallest image row length. Taking the data with the shortest row length as the reference, each full row is sent, and according to the ratio i, the remaining data can be sent within i rows. By using this rule, the data to be sent of different image sources can be reasonably divided according to the ratio i, so as to minimize the transmission slice time.

[0011] The ratio i is the ratio of the image line length of each image source to the minimum image line length. Rounding the line length ratio can simplify the division process, and using the rounded value can avoid excessive accumulation of cache data and save hardware resources.

[0012] When there are four image sources, and the ratio i between the image row lengths of the remaining three image sources and the minimum image row length is greater than or equal to 3: the received data of the image source with the minimum image row length is divided into the next transmission slice for transmission; the received data of the remaining three image sources that do not overlap are divided into any one of at least three subsequent transmission slices for transmission. Specific optimization is performed for the common four-image source transmission, and an optimal division method under four image sources is provided to reduce the transmission slice length as much as possible, thereby allowing support for image sources with higher frame rates.

[0013] When there are four image sources, and the ratio i between the image line lengths of the other three image sources and the minimum image line length is 2 greater than or equal to 3, and 1 is 2: the received data of the image source with the minimum image line length is divided into the next transmission slice for transmission; the received data of the corresponding image source with the ratio i of 2 is divided into any one of the next two transmission slices for transmission; the received data of the corresponding two image sources with the ratio i greater than or equal to 3 that do not overlap is divided into any one of the next at least three transmission slices for transmission. Specific optimization is performed for the common four-image source transmission, and an optimal division method under four image sources is provided to reduce the transmission slice length as much as possible, thereby allowing support for image sources with higher frame rates.

[0014] When there are four image sources, and the ratio i between the image line lengths of the other three image sources and the minimum image line length is 1 greater than or equal to 3, and 2 are 2: the received data of the image source with the minimum image line length is divided into the next transmission slice for transmission; the received data of the two image sources corresponding to the ratio i of 2 that do not overlap are divided into any one of the next two transmission slices for transmission; the received data of the image sources corresponding to the ratio i of 3 greater than or equal to 3 are divided into any one of the next at least three transmission slices for transmission. Specific optimization is performed for the common four-image source transmission, and an optimal division method under four image sources is provided to reduce the transmission slice length as much as possible, thereby allowing support for image sources with higher frame rates.

[0015] When there are four image sources, and the ratio i of the image line lengths of the remaining three image sources to the minimum image line length is 2: the received data of the image source with the minimum image line length is divided into the next transmission slice for transmission; the received data of any two corresponding image sources with the ratio i of 2 that do not overlap are divided into any one of the next two transmission slices for transmission; the received data of the remaining corresponding image source with the ratio i of 2 is divided into any one of the next two transmission slices for transmission. Specific optimization is performed for the common four-image source transmission, and an optimal division method under four image sources is provided to reduce the transmission slice length as much as possible, thereby allowing support for image sources with higher frame rates.

[0016] When there are four image sources and the ratio i between the image line lengths of the other three image sources and the minimum image line length is 1: the received data of the image source with the minimum image line length and the received data of the corresponding image source with the ratio i being 1 are simultaneously divided into the next transmission slice for transmission. Specific optimization is performed for the common four-image source transmission, and an optimal division method under four image sources is provided to reduce the transmission slice length as much as possible, thereby allowing support for image sources with higher frame rates.

[0017] Compared with the prior art, the beneficial effects of the present invention mainly include the following: 1) This method can adaptively allocate the optimal slice transmission method according to the image row length relationship of multiple image sources; 2) There is no restriction on the frame rate relationship of multiple image sources; 3) Compared with the single-chip transmission method according to the frame rate, the transmission slice length is reduced in this method, thereby allowing support for higher frame rate image sources; 4) Compared with the single-chip transmission method according to the frame rate, the requirement for cache depth is reduced in this method, thereby saving hardware resources; 4) Compared with the multi-slice transmission method, this method dynamically configures the transmission slices and is not limited by the transmission slice classification resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the specific embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 The present invention provides a schematic diagram of a method for real-time transmission of images divided into single pieces according to frame rates.

[0020] Figure 2 The present invention provides a schematic diagram of a method for real-time transmission of an image divided into multiple slices according to line length. DETAILED DESCRIPTION

[0021] The above and other technical contents, features and effects of the present invention will be clearly presented in the following detailed description of a preferred embodiment with reference to the drawings. The directional terms mentioned in the following embodiments, such as up, down, left, right, front or back, etc., are only referenced to the directions of the attached drawings. Therefore, the directional terms used are used to illustrate and not to limit the present invention.

[0022] The following will describe the various embodiments of the present application in detail with reference to the accompanying drawings. However, it will be appreciated by those skilled in the art that in the various embodiments of the present application, many technical details are provided in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solution claimed in the present application can be implemented.

[0023] The steps in the following embodiments do not correspond one to one with the summary of the invention.

[0024] In the prior art, the MIPI-based image transmission device has the function of merging and sending multi-channel image data, which is usually sent in transmission slices, and each image source in each transmission slice transmits at most one line of image data.

[0025] Specifically, the image transmission device has multiple data input interfaces for receiving data transmitted from different image sources. First, multiple image sources are divided into one or more types of transmission slices for transmission according to information such as frame rate. For example, the image source contains two types of image data with frame rates of 60 and 30, which are usually divided into two types of transmission slices. One type of transmission slice is only used to transmit images with a frame rate of 60, and the other type of transmission slice is used to transmit mixed images with frame rates of 60 and 30, so that they can all be sent evenly. As the difference in the frame rate relationship becomes larger, or it is not an integer multiple, more types of transmission slices are ultimately required.

[0026] Furthermore, for image sources with the same frame rate, a single-chip segmentation method is currently commonly used. Figure 1 As shown, it is a real-time image transmission method of single-chip segmentation according to frame rate in the prior art. When single-chip segmentation is performed according to frame rate, it is divided into only one type of transmission slice, and each image source in each transmission slice transmits at most one line of image data. It can be understood that the time occupied by each row of the image in the image source is called the image row length, the number of rows contained in each frame of the image is called the number of image rows, the number of image frames contained per unit time is called the image frame rate, and the product of the image row length, the number of image rows and the image frame rate is the unit time. It is stipulated in the MIPI protocol that data is sent by the method of short packet + long packet, that is, first send a short packet to declare the end of the previous row and the beginning of the next row, and then send a long packet to carry image data and error correction information. Then the total time for sending short packets and long packets is the shortest transmission time of the image row specified in the MIPI protocol. Only when the image row length is greater than this time can it be completely sent through the MIPI protocol.

[0027] exist Figure 1 In the image source, there are four image sources with the same frame rate: image source a, image source b, image source c, and image source d. The image row length relationship of the four image sources is a:b:c:d=3:1:3:2. Figure 1 In the figure, image a1 and image a2 represent the image line lengths of the first and second lines of image source a, respectively; image b1-image b6 represent the image line lengths of the 1st-6th lines of image source b, respectively; and the rest are deduced in the same way.

[0028] exist Figure 1In the example, the time before T0 is a transmission slice (assuming that the 0th row data of the four image sources have been received at the same time at this moment), then the 0th row data of the four image sources can be sent as a transmission slice. It can be understood that the MIPI protocol stipulates that it should be sent through the method of short packet + long packet, that is, the short packets and long packets of the 0th row of image source a, the 0th row of image source b, the 0th row of image source c and the 0th row of image source d are sent in sequence.

[0029] When the time reaches T0, the image transmission device has sent the first transmission slice and has received the complete data of image b1 (the first row of image source b) (the first rows of the remaining image sources are fully received and cannot be sent). Between T0 and T1, only the data of image b1 is completely received, so the second transmission slice only includes the data of image b1; and so on, at T1, only the data of image b2 and image d1 are completely received (image b1 has been sent), so between T1 and T2, the third transmission slice only includes the data of image b2 and image d1. The above method is used in the follow-up. Figure 1 It can be seen that for the transmission slices obtained by segmenting the transmission slices according to the frame rate, there are blank waiting times in many transmission slices.

[0030] It can be understood that, because sometimes it is necessary to send one row of data of each of the four image sources in one transmission slice, the transmission slice length is first required to be greater than or equal to the sum of the row transmission durations of the four image sources on MIPI, and the minimum value of the transmission slice length (i.e., the minimum transmission slice length) is the sum of the row transmission durations of the four image sources on MIPI. It should be understood that the transmission slice length is related to the image row length of the image source that can be supported, and the minimum transmission slice length is the minimum value of the image row length of the image source that can be supported. Therefore, reducing the minimum transmission slice length can allow image sources with smaller image row lengths to be supported.

[0031] The above transmission method limits the image source with a large number of image lines and a short line length to send image data, which has two effects: 1. The requirements for hardware storage devices may become higher: multiple image sources are sent line by line in sequence. When the image source with a longer line length finishes sending a line of data, the image source with a shorter line length may have stored multiple lines of data. As time accumulates, the storage quantity will increase, thus requiring the hardware storage depth to become larger; 2. The frame rate of the supported image source may become lower: Since the line sending time of the transmission slice is fixed, and each image sends a line of data according to the priority, after the image source with a smaller number of lines has sent the data, it still needs to wait for the image source with a larger number of lines to finish sending before entering the next transmission slice to transmit a new frame of image data. The extra waiting empty line reduces the frame rate of the supported image source.

[0032] Therefore, if we can use as few transmission slice classification resources as possible and shorten the length of the transmission slice at the same time, dynamically divide the content of the transmission slice and send it "in between gaps", we can not only adapt to any frame rate relationship, but also reduce the additional waiting time and achieve the maximum transmission rate for real-time image transmission.

[0033] Embodiment 1 The present invention provides a MIPI-based adaptive image real-time transmission method, comprising the following steps: Step 1: Receive image data from multiple image sources and obtain the image line length of the image source.

[0034] The transmission method provided by the present invention is based on MIPI. It can be understood that the image transmission device based on MIPI has the function of merging and sending multi-channel image data. It is usually sent in divided transmission slices. Each image source in each transmission slice can transmit at most one line of image data.

[0035] Specifically, the image data of multiple image sources are sent line by line in sequence, and the image transmission device has multiple data input interfaces, which are respectively used to receive data transmitted from different image sources. In this embodiment, two image sources are used as an example for introduction. There are two image sources, namely image source a and image source b; the line lengths of the two image sources are obtained, and their image line lengths correspond to Ta and Tb. In this embodiment, the frame rate relationship between the two image sources is not limited, and it is not necessary to classify them according to the frame first.

[0036] Step 2: Determine the transmission period based on the row length relationship of the image source.

[0037] This method divides the transmission slices according to the line length ratio of the image source. First, a transmission cycle can be found, and then the transmission slice length and content distribution within the transmission cycle can be determined. Then, the transmission can be cyclically transmitted according to the transmission cycle, and finally the smooth implementation of the entire transmission can be guaranteed.

[0038] Based on this idea, the least common multiple of the row lengths of image source a and image source b is selected as a transmission cycle. Specifically, the least common multiple X of the image row lengths Ta and Tb is taken, that is, X=A*Ta=B*Tb; where A and B correspond to the number of rows required for the least common multiple of the row lengths of the two image sources. That is, in one transmission cycle, A rows of image source a and B rows of image source b are sent respectively, and the subsequent transmission cycles can be transmitted in this way in a cycle.

[0039] Step 3: Based on the minimum image row length, perform multi-slice segmentation based on the relationship between the image row lengths of the image source to determine the transmission slice length and content.

[0040] Taking the minimum image line length Tb of the two image sources as a reference, determine the corresponding proportional relationship i between the image line length of the image source and the minimum image line length. In this embodiment, the line length ratio of the two image sources is Ta:Tb, which is further simplified to floor(Ta / Tb):1, where floor() is a floor rounding function.

[0041] It can be understood that the purpose of this technical solution is to divide multiple lines of data from different image sources into different transmission slices in one transmission cycle to reduce the length of the transmission slice as much as possible. As mentioned above, the minimum length of the transmission slice is the minimum value of the image line length of the image source that can be supported. Therefore, in order to support the above two image sources, the set transmission slice length should be less than or equal to the minimum line length Tb in the image source. In order to send out all B lines of image b in one transmission cycle, at least B transmission slices are required, each of which contains a line of data from image source b. Subsequently, it is only necessary to reasonably divide the data from image source a into each transmission slice.

[0042] The remaining image source data can be divided according to the proportional relationship of its image row length. It should be stated first that in order to ensure transmission efficiency and reduce the cache space required to store image data, this application stipulates that one row of image cache data can be reserved for each image source. Then, for image source a, assuming that the value of floor(Ta / Tb) is i (i is greater than or equal to 1), each time i transmission slices are sent, the previous row of data of image source a that has been received must be divided into the above i transmission slices and sent out (if a row is not fully stored, it is considered empty and will not be sent out).

[0043] Therefore, when there are two image sources, there must be a transmission slice that contains one row of data from both image source a and image source b. In this case, the transmission slice length can be taken as the sum of the row transmission time of image source a and image source b. The following discusses the row length ratio i in different situations: 1) When i is greater than or equal to 1, the transmission slice length is the sum of the row transmission time of image source a and image source b. The type and content of the transmission slice are determined according to the actual situation. One possible division method is: for example, when Ta:Tb is exactly 1:1, the order of transmission in one transmission cycle is ab, ab, and ab, and it is sent in a cycle; another possible division method is: for example, when Ta:Tb is 3:2, the order of transmission in one cycle is b, ab, and ab, and it is sent in a cycle. No more examples are given here.

[0044] Regarding the calculation method of the line transmission time τ, first, the line transmission time of the two image sources corresponds to τ a and τ bIt should be understood that MIPI transmission images are divided into short packet transmission time (Tsp) and long packet transmission time (Tlp), that is, the row transmission time of the image source is the corresponding Tsp+Tlp. Tsp is only related to the MIPI configuration, and Tlp is also related to the amount of transmitted image data. Then the row transmission time of image source a and image source b are: τ a =Tsp+Tlpa,τ b =Tsp+Tlpb.

[0045] The upper computer calculates the ratio i between the image line length of each image source and the minimum image line length, and corresponds it to the above type, thereby obtaining the transmission slice length and the transmission content in the transmission slice.

[0046] It can be understood that, taking the image row length relationship of Ta:Tb=3:1 as an example. The selected transmission cycle is 1*Ta=3*Tb, that is, in one transmission cycle, 1 row of image source a and 3 rows of image source b are sent respectively, and the subsequent transmission cycles can be transmitted in this way. The ratio of image row length Ta:Tb is simplified to: floor(Ta / Tb):1= 3:1. Corresponding to the above situation 2, that is: transmission slice length = τ a +τ b ; The contents of the transmission piece are b, b, ab, and they are sent cyclically in this cycle.

[0047] Embodiment 2 Step 1: Receive image data from multiple image sources and obtain the image line length of the image source.

[0048] In this embodiment, an example is given in which there are three image sources. First, there are three image sources, namely image source a, image source b, and image source c. The line lengths of the three image sources are obtained, and the image line lengths correspond to Ta, Tb, and Tc. In this embodiment, the frame rate relationship of the three image sources is not limited, and there is no need to classify them according to the frame first.

[0049] Step 2: Determine the transmission period based on the row length relationship of the image source.

[0050] This method performs segmentation based on the relationship between line lengths. First, a transmission cycle can be found, and then the transmission piece length and content distribution within the transmission cycle can be determined. Then, the transmission can be cyclically performed according to the transmission cycle, and finally the smooth implementation of the entire transmission can be ensured.

[0051] Based on this idea, the least common multiple of the row lengths of the image sources ac is selected as a transmission cycle. Specifically, the least common multiple X of the image row lengths Ta, Tb, and Tc is taken, that is, X=A*Ta=B*Tb=C*Tc; where A, B, and C are the number of rows required for the three image sources to take the least common multiple of the row lengths. That is, in one transmission cycle, A rows of image source a, B rows of image source b, and C rows of image source c are sent respectively, and the subsequent transmission cycles can be transmitted in this way in a cycle.

[0052] Step 3: Based on the minimum image row length, perform multi-slice segmentation based on the relationship between the image row lengths of the image source to determine the transmission slice length and content.

[0053] Taking the minimum image line length among the three image sources (in this embodiment, Tc is the minimum value and Ta is the maximum value) as a benchmark, the proportional relationship of the image line lengths of the three image sources Ta:Tb:Tc can be simplified to: floor(Ta / Tc):floor(Tb / Tc):1; where floor() is a floor rounding function.

[0054] It can be understood that the purpose of this technical solution is to divide multiple lines of data from different image sources into different transmission slices in one transmission cycle to reduce the transmission slice length as much as possible. As mentioned above, the minimum transmission slice length is the minimum value of the image row length of the image source that can be supported. Therefore, in order to support the above three image sources, the set transmission slice length should be less than or equal to the minimum row length Tc in the image source. In order to send out all C rows of image c in one transmission cycle, at least C transmission slices are required, each of which contains a row of data from image source c. Subsequently, the data of the remaining image sources only need to be reasonably divided into each transmission slice.

[0055] The data division method of the remaining two image sources can be performed according to the proportional relationship between their image row lengths. It should be stated first that in order to ensure transmission efficiency and reduce the cache space required to store image data, this application stipulates that one row of image cache data can be retained for each image source. Then, for image source a, assuming that the value of floor(Ta / Tc) is 3, for every 3 transmission slices sent, the previous row of data of image source a that has been received must be divided into the above 3 transmission slices and sent out (if a row is not fully stored, it will be regarded as empty and not sent out), and the same is true for image source b. That is, when the proportional relationship between the image source row length and the minimum image row length is i (i is greater than or equal to 1), then for every i transmission slices sent, the previous row of data of the image source that has been received must be divided into the above i transmission slices and sent out (if a row is not fully stored, it will be regarded as empty and not sent out).

[0056] In summary, in order to ensure that the set transmission slice length is as small as possible, image source a and image source b should be divided into different transmission slices as much as possible. The following discusses the image line length ratio relationship in different situations: 1) If the ratio i of the image line length of the two image sources to the minimum image line length is greater than or equal to 2. That is, as long as 2 lines of data are sent in 2 transmission slices, it is certain that the 2 lines of data are not located in the same transmission slice. Therefore, each transmission slice only needs to include 2 lines of data at most, and the transmission slice length can be taken as τ c +MAX(τ a ,τ b ); the type and content of the transmission slice are determined according to the actual situation. One possible division method is: for example, when Ta:Tb:Tc is exactly 2:2:1, the order of transmission in one transmission cycle is ca, cb, ca and cb, and they are sent in a cycle; another possible division method is: for example, when Ta:Tb:Tc is 3:2:1, the order of transmission in one cycle is ca, cb, c, cb, ca and cb, and they are sent in a cycle. No more examples are given here.

[0057] 2) If the ratio i between the image row lengths of the two image sources and the minimum image row length is 1. For an image source where i is 1, that is, each time an image c is sent, the image source data must also be sent accordingly, that is, each transmission slice must contain a row of data of the image source (since the ratio i is calculated using rounding down, not every time the image source has a complete row of data, but it is still considered to exist). Multiple image sources where i is 1 can be regarded as the same image source. For example, when the ratio i between image source b and image source c is 1, image source b and image source c are regarded as one image source, and graphic source a is another image source. You can refer to Example 1 for this. It can be understood that the data of image source b and image source c must exist in each transmission slice, so the transmission slice length = τ a +τ b +τ c .

[0058] Embodiment 3 Step 1: Receive image data from multiple image sources and obtain the image line length of the image source.

[0059] In this embodiment, four image sources are used as an example. First, there are four image sources, namely image source a, image source b, image source c, and image source d. The line lengths of the four image sources are obtained, and the image line lengths correspond to Ta, Tb, Tc, and Td. In this embodiment, the frame rate relationship of the four image sources is not limited, and it is not necessary to classify them according to the frame first.

[0060] Step 2: Determine the transmission period based on the row length relationship of the image source.

[0061] This method performs segmentation based on the relationship between line lengths. First, a transmission cycle can be found, and then the transmission piece length and content distribution within the transmission cycle can be determined. Then, the transmission can be cyclically performed according to the transmission cycle, and finally the smooth implementation of the entire transmission can be ensured.

[0062] Based on this idea, the least common multiple of the row lengths of the image source ad is selected as a transmission cycle. Specifically, the least common multiple X of the image row lengths Ta, Tb, Tc, and Td is taken, that is, X=A*Ta=B*Tb=C*Tc=D*Td; where A, B, C, and D are the number of rows required for the least common multiple of the row lengths of the four image sources. That is, in one transmission cycle, A row of image source a, B row of image source b, C row of image source c, and D row of image source d are sent respectively, and the subsequent transmission cycles can be transmitted in this way in a cycle.

[0063] Step 3: Based on the minimum image row length, perform multi-slice segmentation based on the relationship between the image row lengths of the image source to determine the transmission slice length and content.

[0064] Taking the minimum image line length among the four image sources (in this embodiment, Td is the minimum value and Ta is the maximum value) as a benchmark, the proportional relationship of the image line lengths of the four image sources Ta:Tb:Tc:Td can be simplified to: floor(Ta / Td):floor(Tb / Td):floor(Tc / Td):1; where floor() is a floor rounding function.

[0065] It can be understood that the purpose of this technical solution is to divide multiple lines of data from different image sources into different transmission slices in one transmission cycle to reduce the length of the transmission slice as much as possible. As mentioned above, the minimum length of the transmission slice is the minimum value of the image line length of the image source that can be supported. Therefore, in order to support the above four image sources, the set transmission slice length should be less than or equal to the minimum line length Td in the image source. In order to send out all D lines of image d in one transmission cycle, at least D transmission slices are required, each of which contains a line of data from image source d. Subsequently, the data of the remaining image sources only need to be reasonably divided into each transmission slice.

[0066] The data division method of the remaining three image sources can be performed according to the proportional relationship of their image row lengths. It should be stated first that in order to ensure transmission efficiency and reduce the cache space required to store image data, this application stipulates that one row of image cache data can be reserved for each image source. Then, for image source a, assuming that the value of floor(Ta / Td) is 3, for every 3 transmission slices sent, the received row of data from image source a must be divided into the above 3 transmission slices and sent out (if a row is not fully stored, it will be regarded as empty and not sent out), and the same is true for image source b and image source c. That is, when the proportional relationship between the image source row length and the minimum image row length is i (i is greater than or equal to 1), then for every i transmission slices sent, the received row of data from image source a and image source c must be divided into the above i transmission slices and sent out (if a row is not fully stored, it will be regarded as empty and not sent out).

[0067] In summary, in order to ensure that the set transmission slice length is as small as possible, image source a, image source b and image source c should be divided into different transmission slices as much as possible. The following discusses the transmission time ratio relationship in different situations: 1) If the ratio i of the image line length of the three image sources to the minimum image line length is greater than or equal to 3. That is, as long as 3 lines of data are sent in 3 transmission slices, it is certain that the 3 lines of data are not located in the same transmission slice. Therefore, each transmission slice only needs to include 2 lines of data at most, and the transmission slice length can be taken as τ d +MAX(τ a ,τ b ,τ c ). The type and content of the transmission slice are determined according to the actual situation. One possible division method is: for example, when Ta:Tb:Tc:Td is exactly 3:3:3:1, the order of transmission in one transmission cycle is da, db and dc, and they are sent in a cycle. One possible division method is: for example, when Ta:Tb:Tc:Td is exactly 6:3:3:1, the order of transmission in one transmission cycle is da, db, dc, db, dc and d, and they are sent in a cycle. No more examples are given here.

[0068] 2) If there is a 2 in the ratio i between the image row lengths of the three image sources and the minimum image row length (for example, image source c), and the other ratios i are greater than or equal to 3 (such as image source a and image source b). That is, the two image sources with a ratio i greater than or equal to 3 must not be located in the same transmission slice with each other, but for the image source c with a ratio of 2, it must be sent once in two transmission slices, and must be divided into the same transmission slice with one of the image sources a and b. Therefore, each transmission slice only needs to include at most 3 rows of data (that is, one row of image source d + image source c + any other image source), then the transmission slice length can be taken as τ d +τ c+MAX(τ a ,τ b ). The type and content of the transmission slice are determined according to the actual situation. One possible division method is: for example, when Ta:Tb:Tc:Td is 3:3:2:1, the order of transmission in one transmission cycle is dc, da, dcb, da, dcb and d, and the transmission is cyclic. No more examples are given here.

[0069] 3) If there are two 2s in the ratio i between the image line lengths of the three image sources and the minimum image line length (for example, image source b and image source c), the remaining ratios i are greater than or equal to 3 (image source a). That is, the two image sources with a ratio i of 2 must not be located in the same transmission slice, and image source a must be divided into the same transmission slice with one of image source b and image source c. Therefore, each transmission slice only needs to include at most 3 rows of data (that is, one row of image source d + image source a + any other image source), then the transmission slice length can be taken as τ d +τ a +MAX(τ b ,τ c ). The type and content of the transmission slice are determined according to the actual situation. One possible division method is: for example, when Ta:Tb:Tc:Td is 3:2:2:1, the order of transmission in one transmission cycle is dba, dc, db, dca, db and dc, and the transmission is cyclic. No more examples are given here.

[0070] 4) If the ratio i of the image line length of the three image sources to the minimum image line length is 2, then two of them can be separated into different transmission slices, and the data of the remaining image source must be divided into the same transmission slice as any of the previous two image sources. Therefore, each transmission slice only needs to include 3 rows of data at most (i.e., one row of image source d + any other two image sources), then the transmission slice length can be taken as τ d +MAX(τ a +τ b ,τ a +τ c ,τ b +τ c ). The type and content of the transmission slice are determined according to the actual situation. One possible division method is: when Ta:Tb:Tc:Td is 2:2:2:1, the order of transmission in one transmission cycle is dba and dc, and they are sent in a cycle. No more examples are given here.

[0071] 5) If the ratio i between the image row lengths of the three image sources and the minimum image row length exists at 1. For an image source where i is 1, that is, each time an image d is sent, the image source data must also be sent accordingly, that is, each transmission slice must contain a row of data for the image source (since i is calculated using rounding down, not every time the image source is sent, a complete row of data is present, but it is still considered to exist).

[0072] 5.1) There is an image source with a ratio i of 1 (such as image source b), and the other ratios i are greater than or equal to 2. Each transmission slice only needs to include 3 rows of data at most (i.e., one row of image source d + image source b + any other image source), so the transmission slice length can be taken as τ b +τ d +MAX(τ a ,τ c ). The type and content of the transmission slice are determined according to the actual situation. One possible division method is: when Ta:Tb:Tc:Td is 2:1:2:1, the order of transmission in one transmission cycle is dba and dbc, and they are sent in a cycle. No more examples are given here.

[0073] 5.2) There are two image sources with a ratio i of 1 (such as image source b and image source c). Each transmission slice only needs to contain 4 rows of data at most, so the transmission slice length can be taken as τ a +τ b +τ c +τ d The type and content of the transmission slice are determined according to the actual situation. One possible division method is: when Ta:Tb:Tc:Td is 2:1:1:1, the order of transmission in one transmission cycle is dbca and dbc, and they are sent in a cycle. No more examples are given here.

[0074] Regarding the calculation method of the line transmission time, first, the line transmission time of the four image sources corresponds to τ a , τ b , τ c and τ d It should be understood that MIPI transmission images are divided into short packet transmission time (Tsp) and long packet transmission time (Tlp), that is, the row transmission time τ of the image source is the corresponding Tsp+Tlp, Tsp is only related to the MIPI configuration, and Tlp is also related to the amount of transmitted image data. Then the row transmission time from image source a to image source d is: τ a =Tsp+Tlpa,τ b =Tsp+Tlpb,τ c =Tsp+Tlpc,τ d =Tsp+Tlpd.

[0075] The upper computer calculates the proportional relationship between the image line length of each image source and the minimum image line length, and corresponds it to the above type, thereby obtaining the transmission slice length and the transmission content in the transmission slice.

[0076] For ease of understanding, the following is an actual situation. Step 1: Receive image data from multiple image sources and obtain the image line length of the image source.

[0077] Specifically, in this embodiment, Figure 2 As shown, it is assumed that the image row length relationship of the four image sources is Ta:Tb:Tc:Td=3:1:3:2.

[0078] Step 2: Determine the transmission period based on the row length relationship of the image source.

[0079] The lowest common multiple of the line lengths of image sources ad is selected as a transmission cycle, that is, 2*Ta=6*Tb=2*Tc=3*Td. That is, in one transmission cycle, 2 lines of image source a, 6 lines of image source b, 2 lines of image source c and 3 lines of image source d are sent respectively, and the transmission can be cyclically carried out in subsequent transmission cycles.

[0080] Step 3: Based on the minimum image row length, perform multi-slice segmentation based on the relationship between the image row lengths of the image source to determine the transmission slice length and content.

[0081] Taking the minimum image line length Tb among the four image sources as a reference, the ratio of the image line lengths of the four image sources Ta:Tb:Tc:Td can be simplified to: floor(Ta / Tb):1:floor(Tc / Tb):floor(Td / Tb)=3:1:3:2. This corresponds to the second case in this embodiment, namely: There is one ratio i of 2 between the image line lengths of the three image sources and the minimum image line length, and the remaining ratios i are greater than or equal to 3. Therefore, each transmission slice only needs to include 3 rows of data at most, and the transmission slice length = τ b +τ d +MAX(τ a ,τ c ); A possible division method is bc, bd, ab, bc, bd, abd. Then the transmission slice length can be taken as τ b +τ d +MAX(τ a ,τ c ).

[0082] In this embodiment, the transmission slice length is τ b +τ d +MAX(τ a ,τ c), the prior art is divided into a single piece according to the frame rate as τ a +τ b +τ c +τ d , the length of each transmission line is relatively reduced by (Tsp*3+Tlpb+Tlpd+MAX(Tlpa,Tlpc)) / (Tsp*4+Tlpa+Tlpb+Tlpc+Tlpd), which can be used to support higher-definition image data.

[0083] At the same time, since the transmission slice length is based on the shortest image row length, it is guaranteed that when image sources of any frame rate ratio and any row length ratio are mixed and sent, the stored image row data is about 1 row, which greatly reduces the required temporary storage space. The setting of the transmission slice is dynamically implemented by the host computer, and even in the face of image transmission devices with few slice classification resources, adaptive maximum transmission rate transmission can be achieved.

[0084] Embodiment 4 Step 1: Receive image data from N image sources and obtain the image line length of the image source.

[0085] It can be understood that the transmission method can be applicable to a case where there are N image sources (N≥2).

[0086] In this embodiment, an example is given in which there are N image sources. The N image sources are image source 1 to image source N. The image line lengths corresponding to the N image sources are obtained, and the image line lengths correspond to T1 to TN. In this embodiment, the frame rate relationship of the N image sources is not limited, and classification is not required based on the frame first.

[0087] Step 2: Determine the transmission period based on the row length relationship of the image source.

[0088] This method performs segmentation based on the relationship between line lengths. First, a transmission cycle can be found, and then the transmission piece length and content distribution within the transmission cycle can be determined. Then, the transmission can be cyclically performed according to the transmission cycle, and finally the smooth implementation of the entire transmission can be ensured.

[0089] Based on this idea, the least common multiple of the row lengths of image sources 1-N is selected as a transmission cycle. Specifically, the least common multiple X of the image row lengths T1 to TN is taken, that is, X=A1*T1=A2*Tb=…=AN*TN; where A1 to AN are the number of rows required for the least common multiple of the row lengths of N image sources. That is, in one transmission cycle, A1 rows of image source 1, A2 rows of image source 2, … and AN rows of image source N are sent respectively, and the subsequent transmission cycles can be transmitted in this way in a cycle.

[0090] Step 3: Based on the minimum image row length, perform multi-slice segmentation based on the relationship between the image row lengths of the image source to determine the transmission slice length and content.

[0091] Taking the minimum image row length among the N image sources (in this embodiment, TN is the minimum value and T1 is the maximum value) as a benchmark, the proportional relationship of the image row lengths of the N image sources can be simplified to: floor(T1 / TN):floor(T2 / TN):…:floor(TN-1 / TN):1=i1:i2:i3:…:in-1:1; where floor() is a rounding-down function.

[0092] It can be understood that the purpose of the technical solution is to divide multiple lines of data from different image sources into different transmission slices in one transmission cycle to reduce the transmission slice length as much as possible. As mentioned above, the minimum transmission slice length is the minimum value of the image line length of the image source that can be supported. Therefore, in order to support the above N image sources, the set transmission slice length should be less than or equal to the minimum line length TN in the image source. In order to send all AN lines of image N in one transmission cycle, at least AN transmission slices are required, each of which contains a line of data from image source N, and the data of the remaining image sources can be reasonably divided into each transmission slice. Specifically, first, the received data of the image source with the smallest image line length is divided into the next transmission slice for transmission; the division method of the remaining N-1 image source data can be carried out according to the corresponding ratio relationship i of their image line lengths, and the received data of each of the remaining image sources is divided into any one of the subsequent i transmission slices for transmission. In summary, in order to ensure that the set transmission slice length is as small as possible, the image sources 1-N should be divided into different transmission slices as much as possible.

[0093] In order to facilitate the description, some common English nouns or letters used in the present invention are only used for exemplary reference rather than restrictive interpretation or specific usage, and the protection scope of the present invention should not be limited by their possible Chinese translations or specific letters.

[0094] It should also be noted that, in this document, relational terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

Claims

1. A MIPI-based adaptive image real-time transmission method, characterized in that: The steps include: Using an image transmission device to simultaneously receive data from multiple image sources, and obtaining image line lengths corresponding to the multiple image sources; Determine a transmission period based on a ratio of the lengths of a plurality of the image lines, wherein each of the image sources has at least one line of data to be sent corresponding to the transmission period; In the transmission cycle, the same number of transmission slices are set according to the number of rows of data to be sent contained in the image source corresponding to the minimum image row length in the transmission cycle, and the data to be sent of each image source in the transmission cycle is divided into different transmission slices, which are sent out sequentially by the image transmission device at time intervals of the minimum image row length.

2. The method for real-time adaptive image transmission based on MIPI according to claim 1, characterized in that: The lowest common multiple of the image row lengths of the plurality of image sources is taken as a transmission period, and each of the image sources corresponds to at least one row of data to be sent in the transmission period.

3. The method for real-time adaptive image transmission based on MIPI according to claim 1, characterized in that: An image transmission device is used to simultaneously receive and cache data from a plurality of image sources, and the data cached by each image source is less than 2 lines.

4. The method for real-time adaptive image transmission based on MIPI according to claim 3, characterized in that: Dividing the data to be sent of each of the image sources in the transmission period into different transmission slices comprises: Divide the received data of the image source of the minimum image line length into the next transmission slice for transmission; Based on the corresponding proportional relationship i between the image line length of each of the remaining image sources and the minimum image line length, the received data of each of the remaining image sources is divided into any one of the subsequent i transmission slices for transmission.

5. The method for real-time adaptive image transmission based on MIPI according to claim 4, characterized in that: The proportional relationship i is a floor value of the ratio of the image line length of each of the image sources to the minimum image line length.

6. The method for real-time adaptive image transmission based on MIPI according to claim 5, characterized in that: When there are four image sources, and the ratio i between the image line lengths of the remaining three image sources and the minimum image line length is greater than or equal to 3: Divide the received data of the image source of the minimum image line length into the next transmission slice for transmission; The received data of the remaining three image sources are divided into any one of at least three subsequent transmission slices for transmission without overlap.

7. The method for real-time adaptive image transmission based on MIPI according to claim 5, characterized in that: When there are four image sources, and the ratio i between the image line lengths of the other three image sources and the minimum image line length is 2 greater than or equal to 3 and 1 is 2: Divide the received data of the image source of the minimum image line length into the next transmission slice for transmission; Divide the received data of the corresponding image source whose ratio i is 2 into any one of the following two transmission slices for transmission; The received data corresponding to the two image sources with the ratio i greater than or equal to 3 and not overlapping are divided into any one of at least three subsequent transmission slices for transmission.

8. The method for real-time adaptive image transmission based on MIPI according to claim 5, characterized in that: When there are four image sources, and the ratio i between the image line lengths of the other three image sources and the minimum image line length is 1 greater than or equal to 3 and 2 is 2: Divide the received data of the image source of the minimum image line length into the next transmission slice for transmission; Divide the non-overlapping received data of the two image sources corresponding to the ratio i of 2 into any one of the two subsequent transmission slices for transmission; The received data of the corresponding image source whose ratio i is greater than or equal to 3 is divided into any one of at least 3 subsequent transmission slices for transmission.

9. The method for real-time adaptive image transmission based on MIPI according to claim 5, characterized in that: When there are four image sources, and the ratio i between the image line lengths of the remaining three image sources and the minimum image line length is 2: Divide the received data of the image source of the minimum image line length into the next transmission slice for transmission; Divide the non-overlapping received data of any two corresponding image sources whose ratio i is 2 into any one of the two subsequent transmission slices for transmission; The received data of the remaining image source corresponding to the image source with a ratio i of 2 is divided into any one of the following two transmission slices and sent.

10. The method for real-time adaptive image transmission based on MIPI according to claim 5, characterized in that: When there are four image sources and the ratio i between the image line lengths of the remaining three image sources and the minimum image line length is 1: The received data of the image source with the minimum image row length and the received data of the corresponding image source with the ratio i being 1 are simultaneously divided into the next transmission slice for transmission.

Images