A network push streaming control method, device and equipment based on LT6911C and a storage medium

Abstract

The application discloses a network push stream control method and device based on LT6911C, equipment and storage medium. The HDMI audio and video data signal is transmitted to the LT6911C transcode module through the HDMI interface; the LT6911C transcode module is controlled to convert the HDMI audio and video data signal into the MIPI CSI video data signal and the I2S audio data signal; the RV1126 / RV1109 encoding module is controlled to receive the MIPI CSI video data signal and the I2S audio data signal, and to carry out stream taking coding on the MIPI CSI video data signal and the I2S audio data signal; the audio and video composite data stream after stream taking coding is packed and compressed; and the audio and video composite data stream after packing and compression is sent to a remote network server through a preset network protocol. The application realizes the push stream control of the network video data stream, and has the advantages of universality, generality and strong compatibility.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of network push stream control based on LT6911C, and particularly relates to a network push stream control method and device based on LT6911C, equipment and storage medium. BACKGROUND

[0002] In the field of audio and video program production, with the development of audio and video technology and the large-scale popularity and application of smart phones, 5G and other network devices, audio and video IP transmission has become the main development trend. In the audio and video industry, the modular application of video stream acquisition, editing, management, decoding and production has been very mature. Connecting the above modular business processes together to create an audio and video full-IP solution ecological chain has become a trend in the industry and is unstoppable.

[0003] Starting from the acquisition of video sources, the traditional baseband signal is encoded into an IP stream through an encoder, which can be easily transmitted through wired or wireless networks. If a server is deployed, all devices and IP streams (including NDI) on the link can be easily managed, remotely controlled and parameter configured through the server. IP streams controlled and scheduled by the platform can be used for various purposes, such as decoding, recording and storage, conversion into other protocols for multi-platform live streaming, multi-picture preview and monitoring, and even access to third-party network video stream production systems.

[0004] In the prior art, the network push stream solution based on Huawei HiSilicon occupies the mainstream position in the market. However, due to the market environment limiting Huawei HiSilicon, there is a lack of other new video IP solutions. In addition, the IPC push stream product in the prior art generally uses a Camera as the data source device, which has a large limitation in data source input and is not universal and universal, and cannot meet the application requirements of more occasions. Therefore, it is an urgent problem for technical personnel in the field to invent a reliable and effective network push stream control method based on LT6911C. SUMMARY

[0005] To solve the above problems, the present application provides a network push stream control method, device, equipment and storage medium based on LT6911C.

[0006] To solve the above technical problems, the present application provides a network push stream control method based on LT6911C, which comprises:

[0007] transmitting the HDMI audio and video data signal to the LT6911C transcoding module through the HDMI interface;

[0008] Control the LT6911C transcode module to convert the HDMI audio and video data signal into MIPI CSI video data signal and I2S audio data signal;

[0009] Control the RV1126 / RV1109 encoding module to receive the MIPI CSI video data signal and I2S audio data signal;

[0010] Control the RV1126 / RV1109 encoding module to stream encoding the MIPI CSI video data signal and I2S audio data signal;

[0011] Pack and compress the stream encoded audio and video composite data stream;

[0012] Send the packed and compressed audio and video composite data stream to the remote network server through the preset network protocol.

[0013] Preferably, before the HDMI audio and video data signal is transmitted to the LT6911C transcode module through the HDMI interface, it includes:

[0014] Control the state machine of the LT6911C transcode module to detect whether the input state of the HDMI interface is stable;

[0015] Control the state machine of the LT6911C transcode module to initialize the input of the HDMI interface;

[0016] Control the state machine of the LT6911C transcode module to detect whether the Vedio of the HDMI interface is normal;

[0017] Control the state machine of the LT6911C transcode module to initialize the output of the HDMI interface;

[0018] Control the state machine of the LT6911C transcode module to close all outputs.

[0019] Preferably, before the HDMI audio and video data signal is transmitted to the LT6911C transcode module through the HDMI interface, it further includes:

[0020] Control the state machine of the LT6911C transcode module to poll the real-time state of the HDMI interface;

[0021] When the state machine of the LT6911C transcode module successfully initializes the output of the HDMI interface, transmit the HDMI audio and video data signal to the LT6911C transcode module through the HDMI interface to establish the underlying data stream channel.

[0022] Preferably, the state machine of the LT6911C transcode module is controlled to detect whether the input state of the HDMI interface is stable, comprising:

[0023] detecting whether the TMDS clock of the HDMI interface is stable, if yes, detecting whether the HDMI interface is in an open state, if no, controlling the state machine of the LT6911C transcode module to close all outputs;

[0024] in the detection of whether the HDMI interface is in an open state, if yes, detecting whether the TMDS clock of the HDMI interface changes, if no, setting RXPll enable to control the state machine of the LT6911C transcode module to initialize the input of the HDMI interface;

[0025] in the detection of whether the TMDS clock of the HDMI interface changes, if yes, controlling the state machine of the LT6911C transcode module to close all outputs, if no, detecting whether the output voltage of the HDMI interface is stable;

[0026] in the detection of whether the input resolution of the HDMI interface is stable, if yes, controlling the state machine of the LT6911C transcode module to close all outputs, if no, controlling the state machine of the LT6911C transcode module to initialize the input of the HDMI interface.

[0027] Preferably, the state machine of the LT6911C transcode module is controlled to initialize the input of the HDMI interface, comprising:

[0028] detecting whether the TMDS clock of the HDMI interface is in the range of (30M, 305M), if yes, performing RXPll calibration, and detecting whether RXPll is in a locked state;

[0029] if RXPll is in a locked state, detecting the locked state of RXPll for the second time, and controlling RXPICDR to reset, performing RXPIPHase setting, Eq setting and HDMImodule reset;

[0030] detecting whether the input Lane number of the HDMI interface is correct, if yes, detecting the locked state of RXPll for the third time, if yes, detecting the stable state of field synchronization, and controlling the received FIFO to reset.

[0031] Preferably, the state machine of the LT6911C transcode module is controlled to detect whether the Vedio of the HDMI interface is normal, comprising:

[0032] reading the time slot parameters of the HDMI interface, and judging whether the time slot parameters are normal;

[0033] If yes, the state of the HDMI interface is controlled to be turned on, and the time slot row polarity and column polarity of the HDMI interface are adjusted.

[0034] Preferably, the control of the state machine of the LT6911C transcode module to initialize the output of the HDMI interface comprises:

[0035] The TMDSClock setting, the MIPILaneSwap setting, the MIPIPort setting, the MIPIClock setting, the MIPIDphy setting, the MIPIProtocal setting, the MIPIVideo setting and the AudioI2s setting are saved.

[0036] To solve the above technical problems, the embodiment of the present application provides a network push streaming control device based on LT6911C, which comprises:

[0037] The signal transmission module is configured to transmit the HDMI audio and video data signal to the LT6911C transcode module through the HDMI interface.

[0038] The signal conversion module is configured to control the LT6911C transcode module to convert the HDMI audio and video data signal into the MIPI CSI video data signal and the I2S audio data signal.

[0039] The signal receiving module is configured to control the RV1126 / RV1109 encoding module to receive the MIPI CSI video data signal and the I2S audio data signal.

[0040] The streaming encoding module is configured to control the RV1126 / RV1109 encoding module to perform streaming encoding on the MIPI CSI video data signal and the I2S audio data signal.

[0041] The packing and compression module is configured to pack and compress the audio and video composite data stream after streaming encoding.

[0042] The push streaming control module is configured to send the audio and video composite data stream after packing and compression to a remote network server through a preset network protocol.

[0043] To solve the above technical problems, the embodiment of the present application provides a network push streaming control device based on LT6911C, which comprises: at least one processor, at least one memory and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, the method of the first aspect in the above embodiment is realized.

[0044] To solve the above technical problems, the embodiment of the present application provides a storage medium, which stores computer program instructions, and when the computer program instructions are executed by a processor, the method of the first aspect in the above embodiment is realized.

[0045] To sum up, the embodiment of the present application provides a network push stream control method, device, equipment and storage medium based on LT6911C. The present application transmits HDMI audio and video data signals to the LT6911C transcoding module through the HDMI interface; controls the LT6911C transcoding module to convert the HDMI audio and video data signals into MIPI CSI video data signals and I2S audio data signals; controls the RV1126 / RV1109 encoding module to receive the MIPI CSI video data signals and I2S audio data signals, and performs stream extraction coding on the MIPI CSI video data signals and I2S audio data signals; packs and compresses the audio and video composite data stream after stream extraction coding; and sends the packed and compressed audio and video composite data stream to a remote network server through a preset network protocol. The present application realizes network push stream control based on LT6911C through the RV1126 / RV1109 module and the LT6911C encoding module, and provides a new audio and video IP solution for the restriction of Huawei HiSilicon in the market environment; the present application converts HDMI signals into MIPI CSI signals through the SOC chip LT6911C, and then converts the MIPI CSI signals into network streams through the RV1126 / RV1109, so that the HDMI signal as a source input interface has more universality and universality, and is widely applied in various scenes; in addition, the present application effectively sets the LT6911C transcoding module, so that the IT6616 transcoding module can more accurately realize transcoding control of the HDMI audio and video data signals, and has high reliability. Therefore, the present application realizes push stream control of network video data stream, and has the advantages of universality, universality and strong compatibility. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments of the present application will be briefly introduced. For those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0047] Figure 1 is a flow chart of a network push stream control method based on LT6911C according to an embodiment of the present application.

[0048] Figure 2 is a pin diagram of an LT6911C transcoding module according to an embodiment of the present application.

[0049] Figure 3is a signal output schematic diagram of an LT6911C transcoding module according to an embodiment of the present application.

[0050] Figure 4 is a processing flow chart before HDMI audio and video data signals are transmitted to an LT6911C transcoding module through an HDMI interface according to an embodiment of the present application.

[0051] Figure 5 is a processing flow chart for a state machine of the LT6911C transcoding module to detect whether an input state of the HDMI interface is stable according to an embodiment of the present application.

[0052] Figure 6 is a processing flow chart for a state machine of the LT6911C transcoding module to initialize an input of the HDMI interface according to an embodiment of the present application.

[0053] Figure 7 is a processing flow chart for a state machine of the LT6911C transcoding module to detect whether a Vedio of the HDMI interface is normal according to an embodiment of the present application.

[0054] Figure 8 is a processing flow chart of an RV1126 / RV1109 encoding module according to an embodiment of the present application.

[0055] Figure 9 is a processing flow chart of an RV1126 / RV1109 encoding module to perform stream taking encoding according to an embodiment of the present application.

[0056] Figure 10 is a processing flow chart of an RV1126 / RV1109 encoding module to perform packing compression according to an embodiment of the present application.

[0057] Figure 11 is a structure schematic diagram of an LT6911C-based network push stream control device according to an embodiment of the present application.

[0058] Figure 12 is a structure schematic diagram of an LT6911C-based network push stream control device according to an embodiment of the present application. DETAILED DESCRIPTION

[0059] The features and exemplary embodiments of the various aspects of the present application will be described in detail below with reference to the drawings. In order to make the purposes, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are configured only to explain the present application and are not configured to limit the present application. The present application can be implemented without some of the specific details for those skilled in the art. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0060] It should be noted that, in this paper, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the elements defined by the statement "include" do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0061] Please refer to Figure 1 , Figure 1 A network push streaming control method based on LT6911C is provided for the present application, and the method comprises the following steps:

[0062] S1, transmit the HDMI audio and video data signal to the LT6911C transcode module through the HDMI interface;

[0063] Specifically, in the present application, the HDMI signal is used as the signal source of the audio and video input, the audio and video data signal is mixed together, and output to the LT6911C transcode module. It can be understood that the HDMI signal can be input through a PC, a mobile terminal or an audio and video device with a DHMI interface, etc.

[0064] It should be noted that the present application can also distribute the HDMI signal into multiple HDMI signals through the chip LT86102SXE, wherein one HDMI signal is output to the LT6911C transcode module, and the other HDMI signal is looped out to the display terminal, which is not limited here.

[0065] S2, control the LT6911C transcode module to convert the HDMI audio and video data signal into MIPI CSI video data signal and I2S audio data signal;

[0066] Specifically, the LT6911C transcode module is a high-performance HDMI 1.4 to MIPI DSI or MIPI CSI chip for VR / smartphone / display applications, and the LT6911C has a configurable single-port or dual-port output for MIPI DSI I or MIPI CSI output. In this application, the LT6911C transcode module is used to convert HDMI 1.4 to MIPI CSI single-port output.

[0067] Please refer to Figure 2 , Figure 2 A pin diagram of an LT6911C transcode module provided in the present application.

[0068] Please refer to Figure 3 , Figure 3 A signal output diagram of an LT6911C transcode module provided in the present application.

[0069] Specifically, in this embodiment, the LT6911C transcode module receives HDMI audio and video data signals, converts the HDMI audio and video data signals into MIPI CSI video data signals and I2S audio data signals, and then sends the MIPI CSI video data signals and I2S audio data signals to the RV1126 / RV1109 encoding module through the differential pins MIPI_N1_D3-, MIPI_N1_D3+, MIPI_N1_D2-, MIPI_N1_D2+, MIPI_N1_CLK-, MIPI_N1_CLK+, MIPI_N1_D1-, MIPI_N1_D1+, MIPI_N1_D0-, and MIPI_N1_D0+.

[0070] S3, control the RV1126 / RV1109 encoding module to receive the MIPI CSI video data signals and the I2S audio data signals;

[0071] Specifically, the RV1126 / RK1109 encoding module is a video processing system using a vision processor RV1109 / RV1126 chip, wherein both the RV1126 and the RV1109 are SoCs in the machine vision branch of artificial intelligence, both of which are built-in independent NPUs, the RV1126 can provide 1.5TOPS computing power, and the RV1109 can provide 1.2TOPS computing power.

[0072] Specifically, the application requirements of network video at the present stage are gradually increasing, and various personalized needs are also emerging. Compared with the traditional video IP solution based on HiSilicon chips, the RV1126 / RV1109 system is perfectly compatible with hard decoding FLASH and H265, and can also support 3D playback. Due to the market limitations of HiSilicon chips, it is difficult to keep up with the gradual update of application requirements and personalized needs, and the difficulty of subsequent development by R&D personnel is increasing. For example, the scheme supporting H265 or the scheme supporting 3D playback has been delayed and has not yet appeared, and it is difficult to break through technically.

[0073] Specifically, in the present embodiment, the RV1126 / RV1109 encoding module accepts MIPI CSI video data signals and I2S audio data signals through corresponding pins MIPI_N1_D3-, MIPI_N1_D3+, MIPI_N1_D2-, MIPI_N1_D2+, MIPI_N1_CLK-, MIPI_N1_CLK+, MIPI_N1_D1-, MIPI_N1_D1+, MIPI_N1_D0-, and MIPI_N1_D0+.

[0074] S4, controlling the RV1126 / RV1109 encoding module to stream encode the MIPI CSI video data signals and the I2S audio data signals;

[0075] S5, packaging and compressing the audio and video composite data stream after stream encoding;

[0076] S6, sending the packaged and compressed audio and video composite data stream to a remote network server through a preset network protocol.

[0077] Specifically, in the present embodiment, the RV1126 / RV1109 encoding module is used to encode the received MIPI CSI video data signals into H264 / H265 format, and encode the received I2S audio data signals into AAC format, and then package them into a preset data format and distribute them to the network for user use. Therefore, the work of the RV1126 / RV1109 encoding module is divided into receiving processing part, stream encoding part and packaging sending part.

[0078] Specifically, the RV1126 / RV1109 system accepts video data and audio data signals through the MIPI CSI interface and the I2S interface, encodes the video data and the audio data signals into H264 / H265 format and AAC format respectively, further packs the composite data stream of the audio and the video into FLV format, and finally sends the data stream to a remote network server through the RJ45 interface in the format of the network protocol RTSP or RTMP, or directly plays the corresponding audio and video through the client software such as VLC, potPlayer, ffmpeg and the like, so as to realize the function of IPization of the audio and video data.

[0079] In summary, the application provides a network push stream control method based on LT6911C. In the scheme, HDMI audio and video data signals are transmitted to an LT6911C transcoding module through an HDMI interface; the LT6911C transcoding module is controlled to convert the HDMI audio and video data signals into MIPI CSI video data signals and I2S audio data signals; an RV1126 / RV1109 encoding module is controlled to receive the MIPI CSI video data signals and the I2S audio data signals, and to perform stream taking and encoding on the MIPI CSI video data signals and the I2S audio data signals; the composite data stream of the encoded audio and video is packed and compressed; and the packed and compressed composite data stream of the audio and video is sent to a remote network server through a preset network protocol. The application realizes network push stream control based on LT6911C through the RV1126 / RV1109 module and the LT6911C encoding module, provides a new audio and video IPization solution for the limitation of Huawei HiSilicon in the market environment, and realizes the conversion of HDMI signals into MIPI CSI signals through the SOC chip LT6911C, and the conversion of the MIPI CSI signals into network streams through the RV1126 / RV1109, so that the HDMI signal as a source input interface is more universal and has a wide application scenario. In addition, the application effectively sets the LT6911C transcoding module, so that the IT6616 transcoding module can more accurately realize transcoding control of the HDMI audio and video data signals, and has high reliability. Therefore, the application realizes push stream control of network video data streams, and has the advantages of universality, versatility and strong compatibility.

[0080] On the basis of the above embodiments:

[0081] Please refer to Figure 4 , Figure 4 The application provides a processing flowchart before HDMI audio and video data signals are transmitted to an LT6911C transcoding module through an HDMI interface.

[0082] As a preferred embodiment, before transmitting the HDMI audio and video data signal to the LT6911C transcode module through the HDMI interface, the method further comprises:

[0083] S10, controlling the state machine of the LT6911C transcode module to detect whether the input state of the HDMI interface is stable;

[0084] S11, controlling the state machine of the LT6911C transcode module to initialize the input of the HDMI interface;

[0085] S12, controlling the state machine of the LT6911C transcode module to detect whether the Vedio of the HDMI interface is normal;

[0086] S13, controlling the state machine of the LT6911C transcode module to initialize the output of the HDMI interface;

[0087] S14, controlling the state machine of the LT6911C transcode module to close all outputs.

[0088] Specifically, in the embodiment, the state machine of the LT6911C transcode module starts a loop, sets the HDMI interface state to be closed, sets the HPD signal insertion detection pin to be low, sets the received Phy to be enabled RXPhy, sets the lRXPll to be enabled, sets the HDMI Edid, sets the equalization Eq, sets the clock gate ClockGating, and then sets the HPD signal insertion detection pin to be high, and sets the next step nextStep to be: detecting the DHMI stability.

[0089] As a preferred embodiment, before transmitting the HDMI audio and video data signal to the LT6911C transcode module through the HDMI interface, the method further comprises:

[0090] controlling the state machine of the LT6911C transcode module to poll the real-time state of the HDMI interface;

[0091] When the state machine of the LT6911C transcode module successfully initializes the output of the HDMI interface, transmitting the HDMI audio and video data signal to the LT6911C transcode module through the HDMI interface to establish a data flow channel at the bottom layer.

[0092] Specifically, the LT6911C transcode module is controlled by the RV1126 / 1109 encoding module through the I2C bus at power-on, and corresponding initialization settings are made to the LT6911C transcode module in the process, so that the LT6911C transcode module can convert the HDMI signal into the MIPI CSI signal and the I2S signal. The execution steps of the state machine of the LT6911C transcode module can be divided into five steps: (1) detecting the stability of the HDMI; (2) initializing the HDMI input; (3) detecting the normality of the Video; (4) initializing the HDMI output; and (5) closing all outputs. The state machine of the LT6911C transcode module constantly polls in these five states, and when the "next step" is the corresponding step, the action of the current corresponding step is executed. When the system executes to initialize the HDMI output and succeeds, it indicates that the TMDS signal from the HDMI interface enters the LT6911C transcode module, and is converted into the MIPI CSI signal form and output to the RV1126 / RV1109 encoding module, thereby establishing the underlying data flow channel. In this process, the RV1126 / 1109 encoding module constantly detects the changes of the related states of the LT6911C transcode module, so as to facilitate corresponding processing of signal insertion, signal pullout, signal format number, signal resolution change, signal timing change, etc., and notify the upper-layer APP of the system of the change.

[0093] Please refer to Figure 5 , Figure 5 A processing flowchart for controlling the state machine of the LT6911C transcode module to detect whether the input state of the HDMI interface is stable is provided.

[0094] As a preferred embodiment, controlling the state machine of the LT6911C transcode module to detect whether the input state of the HDMI interface is stable comprises:

[0095] S101, detecting whether the TMDS clock of the HDMI interface is stable, if yes, detecting whether the HDMI interface is in an open state, if no, controlling the state machine of the LT6911C transcode module to close all outputs;

[0096] S102, when detecting whether the HDMI interface is in the open state, if yes, detecting whether the TMDS clock of the HDMI interface changes, if no, setting the RXPll enable to control the state machine of the LT6911C transcode module to initialize the input of the HDMI interface;

[0097] S103, when detecting whether the TMDS clock of the HDMI interface changes, if yes, controlling the state machine of the LT6911C transcode module to close all outputs, if no, detecting whether the output voltage of the HDMI interface is stable;

[0098] S104, if the resolution of the HDMI interface is stable, control the state machine of the LT6911C transcode module to close all outputs, and if not, control the state machine of the LT6911C transcode module to initialize the input of the HDMI interface.

[0099] Referring to Figure 6 , Figure 6 A processing flow chart for controlling the state machine of the LT6911C transcode module to initialize the input of the HDMI interface is provided.

[0100] As a preferred embodiment, the control of the state machine of the LT6911C transcode module to initialize the input of the HDMI interface includes:

[0101] S111, detecting whether the TMDS clock of the HDMI interface is in the range of (30M, 305M), if so, performing RXPll calibration, and detecting whether the RXPll is in a locked state;

[0102] S112, if the RXPll is in the locked state, secondly detecting the locked state of the RXPll, and controlling the RXPICDR to reset, performing RXPIPHase setting, Eq setting and HDMImodule reset;

[0103] S113, detecting whether the number of input lanes of the HDMI interface is correct, if so, thirdly detecting the locked state of the RXPll, and if so, detecting the stable state of the field synchronization, and controlling the received FIFO to reset.

[0104] Referring to Figure 7 , Figure 7 A processing flow chart for controlling the state machine of the LT6911C transcode module to detect whether the video of the HDMI interface is normal is provided.

[0105] As a preferred embodiment, the control of the state machine of the LT6911C transcode module to detect whether the video of the HDMI interface is normal includes:

[0106] S121, reading the time slot parameters of the HDMI interface, and judging whether the time slot parameters are normal;

[0107] S122, if so, controlling the state of the HDMI interface to be opened, and adjusting the time slot row polarity and column polarity of the HDMI interface.

[0108] As a preferred embodiment, the control of the state machine of the LT6911C transcode module to initialize the output of the HDMI interface includes:

[0109] Preserve the TMDSClock setting, MIPILaneSwap setting, MIPIPort setting, MIPIClock setting, MIPIDphy setting, MIPIProtocal setting, MIPIVideo setting and AudioI2s setting.

[0110] Specifically, in the embodiment, after the state machine of the LT6911C transcode module controls the output of the HDMI interface to be initialized, the state machine of the LT6911C transcode module controls all outputs to be closed. When the state machine of the LT6911C transcode module controls all outputs to be closed, it is determined whether the state of the HDMI interface is open. If yes, the sound is closed, the state of the HDMI is set to be closed, and the RxPll is set to be disabled. If not, it is detected again whether the state of the HDMI interface is stable.

[0111] As a preferred embodiment, the LT6911C transcode module converts the HDMI audio and video data signal into the MIPI CSI video data signal and the I2S audio data signal, which includes:

[0112] extracting the video data signal in the HDMI audio and video data signal;

[0113] performing data analysis on the video data signal, and transcoding the video data signal into the MIPI CSI signal;

[0114] transmitting the MIPI CSI signal through the MIPI hardware interface in the YUV422 video format.

[0115] As a preferred embodiment, the LT6911C transcode module converts the HDMI audio and video data signal into the MIPI CSI video data signal and the I2S audio data signal, which further includes:

[0116] extracting the audio data signal in the HDMI audio and video data signal;

[0117] performing data analysis on the audio data signal, and encoding the audio data signal into the I2S signal;

[0118] transmitting the I2S signal through the I2S hardware interface.

[0119] Specifically, the LT6911C transcode module receives the HDMI signal, for the video data signal mixed in the HDMI signal, will be parsed out and transcoded into MIPI CSI signal, through the MIPI hardware interface to the RV1126 / RV1109 encoding module in YUV422 video format output; for the audio data signal mixed in the HDMI signal, will be parsed out and encoded into I2S signal, through the I2S hardware interface output to the RV1126 / RV1109 encoding module.

[0120] As a preferred embodiment, the control RV1126 / RV1109 encoding module to MIPI CSI video data signal and I2S audio data signal for stream encoding includes:

[0121] Control the MIPI CSI signal into the standard MIPI CSI protocol processing;

[0122] The MIPI CSI signal is divided into multiple data streams, wherein the data content, resolution and resolution of the multiple data streams are the same.

[0123] Please refer to Figure 8 , Figure 8 A processing flow chart of the RV1126 / RV1109 encoding module provided in the present application.

[0124] Specifically, after the RV1126 / RV1109 encoding module receives the MIPI CSI video data signal, the MIPI CSI video data signal enters the standard MIPI CSI protocol for processing, and the MIPI CSI video data signal enters two channels of rockchip-MIPI-dphy-RX and rockchip-MIPI-csi2. After the video stream from the rockchip-MIPI-csi2 channel, it enters rkispp0. The function of rkispp0 is to divide one physical data stream into four data streams with the same content and can be used simultaneously, which are the first data stream Bypass ( / dev / video18), the second data stream Scalar0 ( / dev / video19), the third data stream Scalar1 ( / dev / video20) and the fourth data stream Scalar2 ( / dev / video21). The content, resolution and frame rate of the four data streams are the same by default, and the input resolution is 1920x1080@P60. The four data streams can be used according to the different needs of the upper APP. For example, the first data stream Bypass ( / dev / video18) is pushed to the network in the format of 1280x720@P60 data, and the Bypass data stream is taken; the second data stream Scalar0 ( / dev / video19) is recorded to the SD card or USB disk in the format of 1920x1080@P60 data, and the Scalar0 data stream is taken; the third data stream Scalar1 ( / dev / video20) is previewed on the WEB in the format of 720x480@P30 data, and the Scalar1 data stream is taken, etc. Thus, the user layer can take the video data stream sent by HDMI through m_bypass, Scale0, Scale1 and Scale2.

[0125] Specifically, after the RV1126 / RV1109 encoding module receives the I2S audio data signal, the I2S audio data signal directly enters the dummy_codec of the RV1126 / RV1109 encoding module through three directly connected GPIOs in the hardware: I2S2_SCK, I2S2_LRCK and I2S2_SDI, and enters the CPU for processing through DMA. Finally, the interface / proc / asound / card0 / pcm0c that can take the stream is exposed to the user layer. Thus, the user layer can obtain the audio data stream sent by the LT6911C through / proc / asound / card0 / pcm0c.

[0126] Please refer to Figure 9 , Figure 9A processing flowchart of a RV1126 / RV1109 encoding module for stream taking encoding is provided in the present application.

[0127] As a preferred embodiment, the control of the RV1126 / RV1109 encoding module to take stream encoding of the MIPI CSI video data signal and the I2S audio data signal further comprises:

[0128] processing the MIPI CSI video data signal and the I2S audio data signal to form a bottom layer video stream;

[0129] converting the bottom layer video stream into video data in a preset format;

[0130] converting the video data into a video data stream in H264 or H265 format, wherein the preset format includes resolution size and scaling multiple;

[0131] outputting the video data in different preset formats through corresponding interfaces respectively to form device files exposed to an upper layer application;

[0132] the upper layer application opens the device files through input and output control to obtain the video data.

[0133] Specifically, as shown in Figure 9 the bottom layer kernel will expose device files such as m_bypass (i.e., / dev / video18), Scale0 (i.e., / dev / video19), Scale1 (i.e., / dev / video20), Scale2 (i.e., / dev / video21), / proc / asound / card0 / pcm0c to the user layer application in the upper layer, such as mediaserver, V4L2-media, UAC, UVC, which will open the device files through IO ctrl to take stream operation on the corresponding device files. In the current network stream system, mediaserver will be used as the main process to open, set, take stream and close the corresponding device.

[0134] Specifically, the data source interface m_bypass, i.e., the interface / dev / video18, takes the video data stream, and by analogy, it can also take stream from other interfaces such as video19, video20, etc. The memory allocation mode of stream taking is MEMORY_DMABUF, and the stream taking resolution is 1920x1080, and the format is nv12, which is provided to the upper layer application mediaserver.

[0135] Specifically, the I2S audio data signal is taken from a data source interface default, that is, interface / proc / asound / card0 / pcm0c to an audio data stream, the audio channel is a double channel, the sampling rate is 48KHz, and the format of pcm_fltp is provided to the upper layer application mediaserver.

[0136] Specifically, in another preferred embodiment, after the upper layer application opens the device file through input and output control and obtains the video data, the video data can also be processed according to the destination format; wherein the corresponding processing includes at least one of resolution adjustment, video cropping, and video flipping. Through the above technical solution, after obtaining the video data, the corresponding processing such as resolution adjustment, video cropping, and video flipping is performed according to the required destination format, so as to further meet the specific requirements of different servers or clients in presenting the video data, that is, when the presentation requirements of the video data are not in the range of the pre-set format, the corresponding processing can be directly performed according to the destination format of the server or the client.

[0137] Please refer to Figure 10 , Figure 10 A processing flow chart of an RV1126 / RV1109 encoding module provided in the present application is provided.

[0138] Specifically, after the video stream is taken from the corresponding interface, it needs to be converted into RGA first, and then encoded into H264 / H265 format, and then packaged into RTMP and RTSP together with the encoded audio AAC signal and sent to the network.

[0139] Specifically, the reason why the video data needs to be processed according to the destination format is that the resolution of the video, the cropping of the video, and the flipping of the video up, down, left and right may be required to be set by the upper layer application (such as the related configuration of the web), and the processing according to the destination format provides this type of operation and meets the corresponding requirements. It should be noted that when the video data is processed according to the destination format, one of the resolution adjustment, video cropping, and video flipping can be performed, two of the resolution adjustment, video cropping, and video flipping can be performed, or all three of the resolution adjustment, video cropping, and video flipping can be performed. The specific situation is selected according to the actual needs, and will not be described here.

[0140] Specifically, when encoding the MIPI CSI video data signal into the H264 / H265 format, a plurality of parameters related to encoding are involved, such as an encoding level, an encoding resolution, an encoding rate, a gop, a frame rate, and the like. In the embodiment, the RV1126 / RV1109 encoding module encodes the video into H264, with a resolution of 1920x1080, and finally outputs the data to a next node, such as a muxer.

[0141] Specifically, when encoding the I2S audio data signal into the AAC format, a plurality of parameters related to encoding are involved, such as a sampling rate, a sampling bit depth, a code rate, and a sampling format. Finally, the output data is used by a next node, such as a muxer.

[0142] Specifically, the present application needs to composite the node after video encoding and the node after audio encoding into an RTSP node or an RTMP node, and publish to the network. The current system provides the rtsp service by listening to the 554 port, and the client can open the URL: rtsp: / / 192.168.1.100:554 / live / mainstream through the software VLC, for example, assuming that the ip address of the device is 192.168.1.100. It can be understood that the specific actual ip address of the device is set according to the actual situation, which is not specifically limited here.

[0143] Specifically, the present application can also configure the server-side streaming address through the web. Unlike the device address 192.168.1.100, the client interested in audio and video can pull the stream from the server side, instead of from the device, thereby achieving a good balance of network bandwidth.

[0144] Please refer to Figure 11 The embodiment of the present application provides a network streaming control device based on LT6911C, which comprises:

[0145] The signal transmission module 1 is used for transmitting the HDMI audio and video data signal to the LT6911C transcode module through the HDMI interface.

[0146] The signal conversion module 2 is used for controlling the LT6911C transcode module to convert the HDMI audio and video data signal into the MIPI CSI video data signal and the I2S audio data signal.

[0147] The signal receiving module 3 is used for controlling the RV1126 / RV1109 encoding module to receive the MIPI CSI video data signal and the I2S audio data signal.

[0148] The stream encoding module 4 is used to control the RV1126 / RV1109 encoding module to perform stream encoding on MIPI CSI video data signals and I2S audio data signals;

[0149] The packaging and compression module 5 is used to package and compress the audio and video composite data stream after it has been encoded.

[0150] The streaming control module 6 is used to send the packaged and compressed audio and video composite data stream to a remote network server through a preset network protocol.

[0151] In addition, combined Figure 1 The network streaming control method based on LT6911C described in this embodiment of the invention can be implemented by a network streaming control device based on LT6911C. Figure 12 A schematic diagram of the hardware structure of a network streaming control device based on LT6911C provided in an embodiment of the present invention is shown.

[0152] The LT6911C-based network streaming control device may include a processor 401 and a memory 402 storing computer program instructions.

[0153] Specifically, the processor 401 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement embodiments of the present invention.

[0154] Memory 402 may include mass storage for data or instructions. For example, and not limitingly, memory 402 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be internal or external to a data processing device. In a particular embodiment, memory 402 is a non-volatile solid-state memory. In a particular embodiment, memory 402 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0155] The processor 401 implements the LT6911C-based network push streaming control method in any of the above embodiments by reading and executing computer program instructions stored in the memory 402.

[0156] In one example, the LT6911C-based network push streaming control device can further include the communication interface 403 and the bus 410. As shown in the figure, Figure 12 the processor 401, the memory 402, and the communication interface 403 are connected through the bus 410 and complete communication with each other.

[0157] The communication interface 403 is mainly used to realize the communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0158] The bus 410 includes hardware, software or both to couple components of the LT6911C-based network push streaming control device to each other. By way of example, and without limitation, the bus can include an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front-side bus (FSB), a hypertransport (HT) interconnect, an industry standard architecture (ISA) bus, an infiniband interconnect, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standards association local (VLB) bus, or another suitable bus or combination of two or more of these. Where appropriate, the bus 410 can include one or more buses. Although the present embodiments describe and show a particular bus, the present application contemplates any suitable bus or interconnect.

[0159] In addition, in combination with the LT6911C-based network push streaming control method in the above embodiments, the embodiments of the present application can provide a computer readable storage medium to implement. The computer readable storage medium has computer program instructions stored thereon; the computer program instructions are executed by the processor to implement any of the LT6911C-based network push streaming control methods in the above embodiments.

[0160] It should be further noted that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be executed in the order mentioned in the embodiments, or in an order different from the embodiments, or several steps can be executed simultaneously.

[0161] The above is only a specific implementation of the present application, and those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here. It should be understood that the protection scope of the present application is not limited to this, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A network push streaming control method based on LT6911C, characterized in that, The method comprises: transmitting HDMI audio and video data signals to an LT6911C transcoding module through an HDMI interface; controlling the LT6911C transcoding module to convert the HDMI audio and video data signals into MIPI CSI video data signals and I2S audio data signals; controlling an RV1126 / RV1109 encoding module to receive the MIPI CSI video data signals and I2S audio data signals; controlling the RV1126 / RV1109 encoding module to stream encoding the MIPI CSI video data signals and I2S audio data signals; packaging and compressing the stream-encoded audio and video composite data stream; sending the packaged and compressed audio and video composite data stream to a remote network server through a preset network protocol; before the HDMI audio and video data signals are transmitted to the LT6911C transcoding module through the HDMI interface, the method comprises: controlling a state machine of the LT6911C transcoding module to detect whether the input state of the HDMI interface is stable; controlling the state machine of the LT6911C transcoding module to initialize the input of the HDMI interface; controlling the state machine of the LT6911C transcoding module to detect whether the Video of the HDMI interface is normal; controlling the state machine of the LT6911C transcoding module to initialize the output of the HDMI interface; controlling the state machine of the LT6911C transcoding module to close all outputs; before the HDMI audio and video data signals are transmitted to the LT6911C transcoding module through the HDMI interface, the method further comprises: controlling the state machine of the LT6911C transcoding module to poll the real-time state of the HDMI interface; when the state machine of the LT6911C transcoding module successfully initializes the output of the HDMI interface, transmitting the HDMI audio and video data signals to the LT6911C transcoding module through the HDMI interface to establish a bottom-layer data stream channel; the controlling the state machine of the LT6911C transcoding module to detect whether the input state of the HDMI interface is stable comprises: detecting whether the TMDS clock of the HDMI interface is stable, if yes, detecting whether the HDMI interface is in an open state, if no, controlling the state machine of the LT6911C transcoding module to close all outputs; when detecting whether the HDMI interface is in the open state, if yes, detecting whether the TMDS clock of the HDMI interface changes, if no, setting RXPll enable to control the state machine of the LT6911C transcoding module to initialize the input of the HDMI interface; when detecting whether the TMDS clock of the HDMI interface changes, if yes, controlling the state machine of the LT6911C transcoding module to close all outputs, if no, detecting whether the output voltage of the HDMI interface is stable; In the detection of the input resolution of the HDMI interface, if yes, the state machine of the LT6911C transcode module is controlled to close all outputs, and if no, the state machine of the LT6911C transcode module is controlled to initialize the input of the HDMI interface. The control of the state machine of the LT6911C transcode module to initialize the input of the HDMI interface comprises: Detecting whether the TMDS clock of the HDMI interface is in the range of (30M, 305M), if yes, performing RXPll calibration, and detecting whether the RXPll is in a locked state; If the RXPll is in the locked state, the locked state of the RXPll is detected for the second time, and the RXPICDR is controlled to reset, the RXPIPHase is set, the Eq is set, and the HDMImodule is reset; Detecting whether the input Lane number of the HDMI interface is correct, if yes, the locked state of the RXPll is detected for the third time, if yes, the line field synchronization stable state is detected, and the received FIFO is controlled to reset; The control of the state machine of the LT6911C transcode module to detect whether the Video of the HDMI interface is normal comprises: Reading the time slot parameters of the HDMI interface, and judging whether the time slot parameters are normal; If yes, the state of the HDMI interface is controlled to be opened, and the time slot row polarity and column polarity of the HDMI interface are adjusted; The control of the state machine of the LT6911C transcode module to initialize the output of the HDMI interface comprises: Saving the TMDSClock setting, the MIPILaneSwap setting, the MIPIPort setting, the MIPIClock setting, the MIPIDphy setting, the MIPIProtocal setting, the MIPIVideo setting, and the AudioI2s setting.

2. A network push stream control device based on LT6911C, used to implement the network push stream control method based on LT6911C as claimed in claim 1, characterized in that, The device comprises: A signal transmission module configured to transmit an HDMI audio and video data signal to an LT6911C transcode module through an HDMI interface; A signal conversion module configured to control the LT6911C transcode module to convert the HDMI audio and video data signal into an MIPI CSI video data signal and an I2S audio data signal; A signal receiving module configured to control an RV1126 / RV1109 encoding module to receive the MIPI CSI video data signal and the I2S audio data signal; A streaming encoding module configured to control the RV1126 / RV1109 encoding module to perform streaming encoding on the MIPI CSI video data signal and the I2S audio data signal; A packing and compression module configured to pack and compress the audio and video composite data stream after streaming encoding; A streaming control module configured to send the audio and video composite data stream after packing and compression to a remote network server through a preset network protocol.

3. A network push streaming control device based on LT6911C, characterized in that, The device comprises: At least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method of claim 1.

4. A storage medium having stored thereon computer program instructions, characterized in that, The computer program product comprises computer program instructions which, when executed by a processor, implement the method as claimed in claim 1.

Images