Mobile processing device, processing method and system for processing a video signal source

Abstract

This invention relates to a mobile processing device, processing method, and system for processing video signal sources. The mobile processing device is equipped with an Android operating system. The mobile processing device includes an HDMI-IN interface for connecting to a video signal source. Based on the V4L framework, the mobile processing device opens the device file of the HDMI-IN interface in the Android Native layer to acquire raw data from the video signal source. This invention enables the mobile processing device to quickly connect to a video signal source and, through further encoding and other processing of the acquired raw data, to function as a video data relay device.

Description

Technical Field

[0001] This application relates to video signal source processing technology, and more particularly to mobile processing equipment, processing methods and systems for processing video signal sources. Background Technology

[0002] In existing technologies, mobile processing devices running the Android operating system, such as smartphones, typically only have MicroUSB and Type-C interfaces, lacking an HDMI interface. Furthermore, while existing Android TV toolkits can provide functionality for operating the HDMI-IN interface of hardware devices—for example, reading signal sources connected via HDMI-IN—the related APIs provided by the Android TV toolkits can only perform media playback functions on the HDMI-IN interface and do not support other operations. Moreover, it is a closed-source API, meaning developers cannot see the source code to encapsulate and extend it. If developers need to perform multiple operations on the HDMI-IN interface or the connected signal source simultaneously (including subsequent encoding and streaming operations based on the raw signal source data), existing Android technology cannot achieve such functionality. Summary of the Invention

[0003] To address the deficiencies or shortcomings in the existing technology, this application proposes a mobile processing device for processing video signal sources. The mobile processing device is equipped with an Android operating system. The mobile processing device includes an HDMI-IN interface for connecting to a video signal source. The mobile processing device is based on the V4L framework and opens the device file of the HDMI-IN interface in the Android Native layer to acquire raw data from the video signal source.

[0004] According to some embodiments of the present invention, the mobile processing device transcodes the raw data into RGB format and provides the transcoded data to a display output device for display; and / or, the mobile processing device encodes the raw data into network stream data for network transmission.

[0005] According to some embodiments of the present invention, the mobile processing device uses the libyuv image processing library to transcode the raw data into RGB format and provides the transcoded data to the display output device for display; and / or, the mobile processing device uses the mediacodec encoding and decoding library to compress the raw data into H264 format for network transmission.

[0006] According to some embodiments of the present invention, the mobile processing device transcodes the original data into RGB format and processes the transcoded data, the processing including:

[0007] S21, the RGB data of a frame of image is reduced by half according to the data of single-byte blocks using the ScalePlane function of the libyuv image processing library. The reduction by half includes retaining the data of pixels only by alternating rows, taking the target bit data from the RGB three-bit data of one of the retained pixels, and taking the other two bits of data from the RGB data of a pixel adjacent to the pixel in the image that are complementary to the target bit.

[0008] S22, recombine the target bit data and the other two bits of data to form the RGB data of a pixel in the image;

[0009] S23 uses OpenGL's shader scripting functionality to adjust the RGB data positions of the recombined pixels based on the relative positional relationship between the RGB data positions of the recombined pixels and the RGB data positions of the pixels before recombining, so that the RGB data positions of the recombined pixels correspond one-to-one with the RGB data positions of the pixels before recombining.

[0010] According to some embodiments of the present invention, opening the device file of the HDMI-IN interface in the Android Native layer to acquire raw data from the video signal source includes:

[0011] Open the file handle corresponding to the HDMI-IN interface by its name;

[0012] Based on the file handle, obtain the relevant parameters of the HDMI-IN interface through the ioctl interface;

[0013] Select the relevant parameters and configure the HDMI-IN interface accordingly;

[0014] Based on the settings, the raw data of the video signal source is collected.

[0015] According to some embodiments of the present invention, the mobile processing device includes a mobile communication processing device; the mobile communication processing device includes a mobile phone.

[0016] According to some embodiments of the present invention, the mobile processing device provides the processed RGB data to the display output device for display, or encodes the processed RGB data into network stream data for network transmission.

[0017] This application also proposes a video signal source processing method based on the aforementioned mobile processing device, comprising:

[0018] Connect the video signal source via the HDMI-IN interface;

[0019] Based on the V4L framework, the device file of the HDMI-IN interface is opened in the Native layer of Android to capture raw data from the video signal source.

[0020] According to some embodiments of the present invention, the method further includes transcoding the original data into RGB format and providing the transcoded data to a display output device for display; and / or encoding the original data into network stream data for network transmission.

[0021] According to some embodiments of the present invention, the method further includes transcoding the original data to RGB format, and processing the transcoded data, the processing including:

[0022] S21, the RGB data of a frame of image is reduced by half according to the data of single-byte blocks using the ScalePlane function of the libyuv image processing library. The reduction by half includes retaining the data of pixels only by alternating rows, taking the target bit data from the RGB three-bit data of one of the retained pixels, and taking the other two bits of data from the RGB data of a pixel adjacent to the pixel in the image that are complementary to the target bit.

[0023] S22, recombine the target bit data and the other two bits of data to form the RGB data of a pixel in the image;

[0024] S23 uses OpenGL's shader scripting functionality to adjust the RGB data positions of the recombined pixels based on the relative positional relationship between the RGB data positions of the recombined pixels and the RGB data positions of the pixels before recombining, so that the RGB data positions of the recombined pixels correspond one-to-one with the RGB data positions of the pixels before recombining.

[0025] According to some embodiments of the present invention, opening the device file of the HDMI-IN interface in the Android Native layer to acquire raw data from the video signal source includes:

[0026] Open the file handle corresponding to the HDMI-IN interface by its name;

[0027] Based on the file handle, obtain the relevant parameters of the HDMI-IN interface through the ioctl interface;

[0028] Select the relevant parameters and configure the HDMI-IN interface accordingly;

[0029] Based on the settings, the raw data of the video signal source is collected.

[0030] According to some embodiments of the present invention, the method further includes providing the processed RGB data to a display output device for display, or encoding the processed RGB data into network stream data and sending the network stream data to a terminal for the terminal to decode the network stream data and output it to the screen.

[0031] This application also proposes a system for processing video signal sources, including a mobile processing device for processing video signal sources and a terminal; the mobile processing device responds to an input forwarding request, encodes the raw data of the acquired video signal source into a format that matches the receiving mode of a terminal, and sends it to the terminal.

[0032] According to some embodiments of the present invention, the mobile processing device responds to an input forwarding request, encodes the raw data of the video signal source into network stream data and sends it to the terminal, which then outputs it to the screen.

[0033] According to some embodiments of the present invention, the mobile processing device responds to an input forwarding request by providing the processed RGB data to the display output device for display, or encodes the processed RGB data into network stream data and sends it to the terminal, which then outputs it to the screen.

[0034] The above embodiments of the present invention overcome common cognitive biases of those skilled in the art regarding mobile processing devices under the Android operating system. They unusually introduce an HDMI-IN interface and successfully acquire the corresponding raw data. This allows the mobile processing device to quickly acquire the accessed video signal source and, through further encoding and other processing of the acquired raw data, to act as a video data relay device to further transmit the signal source data. For example, this includes directly pushing the content of a host without a display device to a terminal with a display device, such as a network terminal, or allowing the video signal source to be viewed directly on the mobile processing device. Attached Figure Description

[0035] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0036] Figure 1a A schematic diagram illustrating the processing of a video signal source by a mobile processing device according to some embodiments of the present invention is shown.

[0037] Figure 1b A flowchart illustrating a method for processing RGB data according to some embodiments of the present invention is shown.

[0038] Figure 2aThe diagram illustrates the processing of transcoded RGB data according to some embodiments of the present invention.

[0039] Figure 2b This diagram illustrates further processing of pixels in the height direction according to some embodiments of the present invention.

[0040] Figure 3a This diagram illustrates the arrangement of RGB data at a resolution of 6x4.

[0041] Figure 3b Showing the Figure 3a The diagram shows RGB data halved in both width and height.

[0042] Figure 3c Show deletion Figure 3b A schematic diagram of RGB data with a resolution of 3*2 after the italicized part.

[0043] Figure 3d A schematic diagram illustrating the processing of pixels according to other embodiments of the present invention is shown.

[0044] Figure 3e Deletion is shown according to other embodiments of the present invention. Figure 3d A schematic diagram of the data formed after the data in the italic section.

[0045] Figure 3f A flowchart illustrating a method for processing RGB data according to some embodiments of the present invention is shown.

[0046] Figure 3g A flowchart illustrating a video signal source processing method based on the mobile processing device according to some embodiments of the present invention is shown.

[0047] Figure 4a A schematic structural block diagram of an apparatus for processing video signal sources according to some embodiments of the present invention is shown.

[0048] Figure 4b A schematic structural block diagram of an apparatus for processing a video signal source according to further embodiments of the present invention is shown.

[0049] Figure 4c A structural block diagram of a device output unit according to some embodiments of the present invention is shown.

[0050] Figure 5 A schematic structural diagram of an apparatus for processing video signal sources according to some embodiments of the present invention is shown. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0052] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0053] The first aspect of this application provides a mobile processing device for processing video signal sources, the mobile processing device being equipped with an Android operating system. The mobile processing device includes an HDMI-IN interface for connecting to a video signal source. The mobile processing device is based on the V4L framework and opens the device file of the HDMI-IN interface in the Android Native layer to acquire raw data from the video signal source.

[0054] In this application, mobile processing device includes all portable devices capable of data communication during mobility, such as mobile phones.

[0055] Figure 1a A schematic diagram illustrating the processing of a video signal source by a mobile processing device according to some embodiments of the present invention is shown.

[0056] As shown in the figure: The hardware device input source (e.g., a PC, a camera, etc.) is connected to the mobile processing device as a valid signal source via the HDMI-IN interface. The mobile processing device is based on the V4L framework (including V4L1 and V4L2), and opens the device file of the HDMI-IN interface in the Android Native layer to acquire the raw data of the video signal source.

[0057] According to some embodiments of the present invention, opening the device file of the HDMI-IN interface in the Android Native layer to acquire raw data from a video signal source can be achieved in the following way: opening the file handle corresponding to the HDMI-IN interface by the HDMI-IN interface name (e.g., " / dev / videol"); obtaining relevant parameters of the HDMI-IN interface through the ioctl interface based on the file handle, for example, determining what kind of picture format the HDMI-IN interface supports (the picture format includes the aspect ratio and resolution, such as the aspect ratio being 1:1, 4:3, or others), pixel format (e.g., RGB, YUV), and frame rate); selecting relevant parameters (e.g., what kind of data to read, what kind of pixel format, picture size, etc.) and setting the HDMI-IN interface according to the selected relevant parameters; and acquiring raw data from the video signal source according to the settings.

[0058] According to some embodiments of the present invention, the mobile processing device can transcode the original data into RGB format and provide the transcoded data to the display output device for display.

[0059] Furthermore, according to some embodiments of the present invention, the mobile processing device processes the transcoded RGB data. Figure 1b A flowchart illustrating a method for processing RGB data according to some embodiments of the present invention is shown. The method includes:

[0060] S1, take the target bit data from the RGB three-dimensional data of a pixel in a frame image, and take the other two bits of data from the RGB data of a pixel adjacent to the pixel in the image that are complementary to the target bit.

[0061] like Figure 2a As shown, the first row represents four pixels of an image frame, numbered from right to left as pixels one through four. Each pixel includes R, G, and B data (each data consists of 8 bytes). The G data is taken from the first pixel, the B and R data from the second pixel, the G data from the third pixel, and the B and R data from the fourth pixel.

[0062] S2, the target bit data and the other two bits of data are recombined to form the RGB data of a pixel in the image.

[0063] The adjacent data obtained in step S1 above are recombined (i.e. Figure 2a (Data recombination in the middle), for example, the G data of the first pixel and the B and R data of the second pixel are combined to form Figure 2a The RGB data of one pixel in the third row, the G data of the third pixel, and the B and R data of the fourth pixel form the RGB data of another pixel in the third row of the diagram.

[0064] According to other embodiments of the present invention, R data can be taken from the first pixel and B and G data can be taken from the second pixel to form RGB data of a pixel in the third row of the figure, or B data can be taken from the first pixel and R and G data can be taken from the second pixel to form RGB data of a pixel in the third row of the figure.

[0065] S3. Based on the relative positional relationship between the RGB data positions of the recombined pixels and the RGB data positions of the pixels before combination, adjust the RGB data positions of the recombined pixels so that the RGB data positions of the recombined pixels correspond one-to-one with the RGB data positions of the pixels before combination.

[0066] like Figure 2a As shown, the RGB data of the recombined pixel (row 3 in the figure) may have a positional reversal with the RGB data of the pixel before combination (as shown in rows 1 and 2 in the figure). To address this, this application further proposes to adjust the RGB data position of the recombined pixel based on the relative positional relationship between the RGB data positions of the recombined pixel and the RGB data positions of the pixel before combination, so that the RGB data positions of the recombined pixel and the RGB data positions of the pixel before combination correspond one-to-one.

[0067] According to some embodiments of the present invention, the fragment shader configuration can also be modified (i.e., shader optimization as indicated in the figure) so that the RGB data positions of the recombined pixels correspond one-to-one with the RGB data positions of the pixels before combination. For example... Figure 2a As shown, the R and G color channels can be inverted to avoid color inversion.

[0068] According to some embodiments of the present invention, steps S1, S2 and S3 can be performed sequentially on all pixels of the image row by row, or steps S1, S2 and S3 can be performed sequentially on all pixels of the image column by column.

[0069] The above embodiments significantly reduce the amount of RGB data after transcoding, thereby stabilizing frame rate fluctuations and improving the smoothness of subsequent image output. Furthermore, by modifying the fragment shader configuration, the two channels of the two colors are inverted, saving operation steps and running time while ensuring normal image color.

[0070] According to some embodiments of the present invention, pixel data can also be retained only every other row, and steps S1, S2, and S3 can be performed sequentially on the retained pixels row by row. Figure 2b As shown, except for the pixels in the first and third rows of the image, Figure 2a In addition to using this method for point selection, the pixel data in the second and fourth rows are also omitted. This reduces the amount of RGB data after transcoding by half in both the width and height directions.

[0071] According to some embodiments of the present invention, the aforementioned data processing process can be further implemented using specific open-source libraries. These open-source libraries include, for example, the libyuv image processing library. However, the CPU multimedia instructions for libyuv scaling can only process data in byte blocks of 2 to the power of n, such as 1 byte block, 2 byte blocks, or 4 byte blocks. Each pixel in the RGB image format consists of 3 bytes, and the libyuv library cannot use CPU multimedia instructions to accelerate scaling operations on it (the libyuv library has a scaling function ScaleARGB for 4-byte blocks, a scaling function ScalePlane_16 for 2-byte blocks, and a scaling function ScalePlane for single-byte data, but no scaling function for 3-byte blocks like RGB). If the RGB data is first converted to other formats such as ARGB or I420 (a YUV format), and then scaled using libyuv, the conversion itself requires considerable computation, resulting in unsatisfactory performance, as verified by actual testing.

[0072] Therefore, the present invention proposes the following method for processing RGB data, which includes the following steps S21-S23 ( Figure 3f (The corresponding flowchart is shown):

[0073] S21, the RGB data of a frame of image is reduced by half using the ScalePlane function of the ibyuv image processing library, which processes the data into single-byte blocks. The reduction by half includes retaining only the data of pixels that are separated by rows, taking the target bit data from the RGB three-dimensional data of one of the retained pixels, and taking the other two bits of data from the RGB data of a pixel adjacent to the pixel in the image that are complementary to the target bit.

[0074] like Figure 3d As shown, the first row represents six pixels in an image frame, numbered from right to left as pixels one through six. Each pixel includes R, G, and B data. The G data is extracted from the first pixel, the B and R data from the second pixel, the G data from the third pixel, the B and R data from the fourth pixel, the G data from the fifth pixel, and the B and R data from the sixth pixel. Pixels in rows two and four are then deleted.

[0075] S22, the target bit data and the other two bits of data are recombined to form the RGB data of a pixel in the image.

[0076] The adjacent data extracted in step S21 above are recombined. For example, the G data of the first pixel and the B and R data of the second pixel form the RGB data of one pixel, and the G data of the third pixel and the B and R data of the fourth pixel form the RGB data of another pixel.

[0077] Similarly, the above processing can be performed on all adjacent pixels in the image in sequence to obtain a series of reconstructed pixels.

[0078] S23 uses OpenGL's shader scripting functionality to adjust the RGB data positions of the recombined pixels based on the relative positional relationship between the RGB data positions of the recombined pixels and the RGB data positions of the pixels before recombining, so that the RGB data positions of the recombined pixels correspond one-to-one with the RGB data positions of the pixels before recombining.

[0079] The readjusted RGB data can be used to achieve correct rendering and display.

[0080] The implementation process described above will be explained in further detail below.

[0081] The arrangement of RGB data at 6x4 resolution is as follows: Figure 3a As shown, the RGB data is arranged in memory in BGR order.

[0082] If now expect Figure 3a If the RGB data shown is halved in both width and height, it can be deleted. Figure 3b The italicized portion can be deleted to form Figure 3c The RGB data shown is at a resolution of 3*2.

[0083] As described earlier, libyuv's scaling CPU multimedia instructions can only process data in byte blocks of powers of 2. Therefore, for Figure 3b Currently, there is no efficient way to implement the deletion operation in existing technologies. This invention proposes using the libyuv's ScalePlane function (which operates on a single byte basis) for halving processing, which includes deletion... Figure 3d The data in italics, after being deleted, forms the following: Figure 3e The data shown is for a 3*2 dimension.

[0084] contrast' Figure 3c 'and' Figure 3e Two points are evident:

[0085] Firstly, the order of R and G is reversed in both cases;

[0086] Secondly: Figure 3eThe B, G, and R components of each pixel in the image do not belong to the BGR components of the same pixel in the original image, but they are all components of adjacent pixels.

[0087] Regarding the first point, the R component and G component can be swapped without performance loss using OpenGL's shader scripting capabilities (see explanation below).

[0088] Regarding the second point, adjacent pixels in an image have similar colors, and the color components of adjacent pixels can be interchanged without affecting the display effect, or in other words, the difference is basically imperceptible to the naked eye. Therefore, it can be... Figure 3c Considered as Figure 3e They are equivalent.

[0089] The implementation process of S23 is as follows:

[0090] OpenGL, short for 'Open Graphics Library', is used to render 2D and 3D graphics. In video applications, its 2D rendering capabilities are typically used to display video data frames.

[0091] OpenGL supports a shader language, divided into vertex shaders and fragment shaders (mentioned above). Vertex shaders are used to draw the outline of graphics, while fragment shaders output the color of each pixel. Fragment shaders can operate on each pixel of 2D graphics data, performing appropriate calculations on each color component to achieve certain visual effects, such as adjusting brightness, contrast, and the intensity of red, green, and blue color components. They can also implement the function of swapping color components required in this technical solution. Furthermore, the shader program runs on a GPU-accelerated parallel framework, resulting in extremely high efficiency and eliminating the performance bottleneck caused by CPU point-by-point image processing.

[0092] The following shader program example implements the function of swapping the R and G color components when RGB is used as the data input.

[0093]

[0094]

[0095] Surrounded in front Figures 1a-2b The details of the described embodiments are also applicable to, or can be appropriately adapted to, the embodiments described above. For the sake of simplicity, they will not be repeated here.

[0096] Because libyuv can invoke CPU multimedia instructions to accelerate data processing, the above embodiments of the present invention can achieve highly efficient image scaling without causing much latency. Furthermore, by halving the width and height of RGB image data and combining it with OpenGL shader programs, fast thumbnail display of RGB images can be further achieved. Moreover, since the entire processing uses efficient functions or hardware acceleration methods and does not add any extra data conversion steps (where there is no need to use the libyuv transpose format), processing efficiency is maximized, the time spent processing each frame of data is reduced, and the maximum display frame rate is increased.

[0097] According to some embodiments of the present invention, the mobile processing device can use the ibyuv image processing library to transcode the raw data into RGB format, and use OpenGL ES three-dimensional graphics rendering technology and GLSL shader language to finally realize video playback of the signal source on the display output device.

[0098] According to some embodiments of the present invention, the mobile processing device transcodes the raw data into RGB format and provides the transcoded data to a display output device for display; and / or, the mobile processing device encodes the raw data into network stream data for network transmission. For example, the mobile processing device can send the network stream data to a terminal, which decodes the network stream data and outputs it to the screen. Furthermore, as... Figure 1a As shown, the mobile processing device uses the MediaCodec encoding / decoding library to compress the raw data into H.264 format for network transmission. Furthermore, it can utilize network transmission protocols such as RTP / RTSP to slice and packetize each frame of data and send it to the server, achieving a streaming (data sharing) function. When other terminals connect to the server to view this data, a sharing function from the local device to a remote device is formed. The mobile processing device can also provide the processed RGB data to a display output device for display, or encode the processed RGB data into network stream data for network transmission.

[0099] According to some embodiments of the present invention, the mobile processing device includes a mobile communication processing device; the mobile communication processing device includes a mobile phone.

[0100] This application also proposes a system for processing video signal sources, including a mobile processing device for processing video signal sources and a terminal; the mobile processing device is equipped with an Android operating system; the mobile processing device includes an HDMI-IN interface for connecting to a video signal source; the mobile processing device is based on the V4L framework and opens the device file of the HDMI-IN interface in the Android Native layer to acquire raw data from the video signal source. In response to an input forwarding request, the mobile processing device encodes the acquired raw data from the video signal source into a format matching the receiving mode of a terminal and sends it to the terminal.

[0101] The process of encoding the raw data of the acquired video signal source into a format matching the receiving mode of a terminal includes: encoding the raw data of the video signal source into network stream data and sending it to the terminal, where the terminal outputs it to the screen; and encoding the raw data of the video signal source into a format under other communication protocols such as Bluetooth, so as to send the encoded data to the terminal via Bluetooth or other means, where the terminal outputs it to the screen. The forwarding request also includes a screen mirroring request.

[0102] According to some embodiments of the present invention, the mobile processing device responds to an input forwarding request by providing the processed RGB data to the display output device for display, or encodes the processed RGB data into network stream data and sends it to the terminal, which then outputs it to the screen.

[0103] In addition, the mobile processing device can also be surrounded in front. Figures 1a-3f The mobile processing device described in the various embodiments. The preceding description pertains to... Figures 1a-3f The mobile processing device described herein and its various processing details are also included here by reference, and will not be repeated for the sake of simplicity.

[0104] The above embodiments of the present invention overcome common cognitive biases of those skilled in the art regarding mobile processing devices under the Android operating system. They unusually introduce an HDMI-IN interface and successfully acquire the corresponding raw data. This allows the mobile processing device to quickly acquire the accessed video signal source and, through further encoding and other processing of the acquired raw data, to act as a video data relay device to further transmit the signal source data. For example, this includes directly pushing the content of a host without a display device to a terminal with a display device, such as a network terminal, or allowing the video signal source to be viewed directly on the mobile processing device.

[0105] Figure 3gA flowchart illustrating a video signal source processing method based on the mobile processing device according to some embodiments of the present invention is shown. The method includes:

[0106] S31 connects to a video signal source via the HDMI-IN interface;

[0107] S32, based on the V4L framework, opens the device file of the HDMI-IN interface in the Android Native layer to capture raw data from the video signal source.

[0108] According to some embodiments of the present invention, the method further includes transcoding the original data into RGB format and providing the transcoded data to a display output device for display; and / or encoding the original data into network stream data for network transmission. According to some embodiments of the present invention, the method further includes sending the network stream data to a terminal, whereby the terminal decodes the network stream data and outputs it to a screen.

[0109] According to some embodiments of the present invention, the method further includes processing the transcoded RGB data, the processing including:

[0110] S1, take the target bit data from the RGB three-dimensional data of a pixel in a frame image, and take the other two bits of data from the RGB data of a pixel adjacent to the pixel in the image that are complementary to the target bit;

[0111] S2, recombine the target bit data and the other two bits of data to form the RGB data of a pixel in the image;

[0112] S3. Based on the relative positional relationship between the RGB data positions of the recombined pixels and the RGB data positions of the pixels before combination, adjust the RGB data positions of the recombined pixels so that the RGB data positions of the recombined pixels correspond one-to-one with the RGB data positions of the pixels before combination.

[0113] According to some embodiments of the present invention, adjusting the RGB data positions of the recombined pixels so that the RGB data positions of the recombined pixels correspond one-to-one with the RGB data positions of the pixels before combination includes:

[0114] By modifying the fragment shader configuration, the RGB data positions of the recombined pixels are made to correspond one-to-one with the RGB data positions of the pixels before recombination.

[0115] According to some embodiments of the present invention, the method further includes performing steps S1, S2 and S3 sequentially on all pixels of the image row by row, or performing steps S1, S2 and S3 sequentially on all pixels of the image column by column.

[0116] According to some embodiments of the present invention, the method further includes retaining the data of pixels only every other row, and performing steps S1, S2 and S3 sequentially on the retained pixels row by row.

[0117] According to some embodiments of the present invention, a method for processing transcoded RGB data includes:

[0118] S21, the RGB data of a frame of image is reduced by half according to the data of single-byte blocks using the ScalePlane function of the libyuv image processing library. The reduction by half includes retaining the data of pixels only by alternating rows, taking the target bit data from the RGB three-bit data of one of the retained pixels, and taking the other two bits of data from the RGB data of a pixel adjacent to the pixel in the image that are complementary to the target bit.

[0119] S22, recombine the target bit data and the other two bits of data to form the RGB data of a pixel in the image;

[0120] S23 uses OpenGL's shader scripting functionality to adjust the RGB data positions of the recombined pixels based on the relative positional relationship between the RGB data positions of the recombined pixels and the RGB data positions of the pixels before recombining, so that the RGB data positions of the recombined pixels correspond one-to-one with the RGB data positions of the pixels before recombining.

[0121] According to some embodiments of the present invention, opening the device file of the HDMI-IN interface in the Android Native layer to acquire raw data from the video signal source includes:

[0122] Open the file handle corresponding to the HDMI-IN interface by its name;

[0123] Based on the file handle, obtain the relevant parameters of the HDMI-IN interface through the ioctl interface;

[0124] Select the relevant parameters and configure the HDMI-IN interface accordingly;

[0125] Based on the settings, the raw data of the video signal source is collected.

[0126] According to some embodiments of the present invention, the video signal source processing method of the mobile processing device further includes providing the processed RGB data to a display output device for display, or encoding the processed RGB data into network stream data and sending the network stream data to a terminal for the terminal to decode the network stream data and output it to the screen.

[0127] In addition, the above refers to Figure 1a and 3f The mobile processing device described herein and its various processing details are also included here by reference, and will not be repeated for the sake of simplicity.

[0128] Figure 4a A schematic structural block diagram of an apparatus for processing video signal sources according to some embodiments of the present invention is shown. The apparatus for processing video signal sources may be part of the aforementioned mobile processing device, used to implement some or all of the functions of the aforementioned mobile processing device. Figure 4a As shown, the device 400 may include a data access unit 410 and a data acquisition unit 420.

[0129] The data access unit 410 is used to access a video signal source.

[0130] The data acquisition unit 420 is used to open the device file of the HDMI-IN interface in the Native layer of Android based on the V4L framework in order to acquire raw data from the video signal source.

[0131] According to some embodiments of the present invention, such as Figure 4b As shown, the device 2400 may include: a data access unit 2410, a data acquisition unit 2420, a device output unit 2430, and a code stream output unit 2440.

[0132] The data access unit 2410 is used to access a video signal source.

[0133] The data acquisition unit 2420 is used to open the device file of the HDMI-IN interface in the Native layer of Android based on the V4L framework in order to acquire raw data from the video signal source.

[0134] The device output unit 2430 is used to convert the acquired raw data into RGB format and provide it to the output device for display.

[0135] The stream output unit 2440 is used to encode raw data into network stream data for network transmission. For example, the raw data is compressed into H.264 format using the MediaCodec encoding / decoding library and provided to the transmission protocol for streaming.

[0136] According to some embodiments of the present invention, such as Figure 4c As shown, the device output unit 2430 may further include:

[0137] The data extraction unit 3410 is used to extract the target bit data (8 bytes) from the RGB three-dimensional data of a pixel in a frame image, and to extract the other two bits of data that are complementary to the target bit from the RGB data of a pixel adjacent to the pixel in the image.

[0138] The data combination unit 3420 is used to recombine the target bit data and the other two bits of data to form the RGB data of a pixel in the image.

[0139] The position adjustment unit 3430 is used to adjust the RGB data position of the recombined pixel according to the relative positional relationship between the RGB data position of the recombined pixel and the RGB data position of the pixel before combination, so that the RGB data position of the recombined pixel corresponds one-to-one with the RGB data position of the pixel before combination.

[0140] In this application, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0141] Previous reference Figure 1a-3g The relevant details are included here by reference and will not be repeated here.

[0142] Figure 5 A schematic structural diagram of a device 500 for processing video signal sources according to some embodiments of the present invention is shown. Figure 5 As shown, the device includes a processor 51, a memory 52, and a bus 53.

[0143] In some instances, the device may further include an input device 501, an input port 502, an output port 503, and an output device 504. The input port 502, processor 51, memory 52, and output port 503 are interconnected via a bus 53. The input device 501 and output device 504 are connected to the bus 53 via the input port 502 and output port 503, respectively, and thus connected to other components of the device. It should be noted that the output and input interfaces here can also be represented by I / O interfaces. Specifically, the input device 501 (e.g., an HDMI-IN interface) receives input information (e.g., a video signal source) from an external source and transmits the input information to the processor 51 via the input port 502. The processor 51 processes the input information based on computer-executable instructions stored in the memory 52 to generate output information, temporarily or permanently storing the output information in the memory 52, and then transmitting the output information to the output device 504 via the output port 503. The output device 504 outputs the output information to the outside of the device.

[0144] The aforementioned memory 52 includes a large-capacity memory for data or instructions. For example, and not limitingly, memory 52 may include an HDD, floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where suitable, memory 52 may include removable or non-removable (or fixed) media. Where suitable, memory 52 may be internal or external to a device. In a particular embodiment, memory 52 is a non-volatile solid-state memory. In a particular embodiment, memory 52 includes read-only memory (ROM). Where suitable, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0145] Bus 53 includes hardware, software, or both, and couples multiple components together. For example, and not limitingly, bus 53 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. While specific buses are described and illustrated in embodiments of the invention, the invention contemplates any suitable bus or interconnect.

[0146] The processor 51 performs the following actions based on the computer program stored in the memory 52:

[0147] Connect the video signal source via the HDMI-IN interface;

[0148] Based on the V4L framework, the device file of the HDMI-IN interface is opened in the Native layer of Android to capture raw data from the video signal source.

[0149] According to some embodiments of the present invention, the processor 51 performs the following actions based on a computer program stored in the memory 52:

[0150] The method involves transcoding the raw data into RGB format and providing the transcoded data to a display output device for display; and / or encoding the raw data into network stream data for network transmission. According to some embodiments of the invention, the method further includes sending the network stream data to a terminal, whereby the terminal decodes the network stream data and outputs it to a screen.

[0151] According to some embodiments of the present invention, the processor 51, based on the computer program stored in the memory 52, also performs the following actions:

[0152] The original data is converted to RGB format, and the converted RGB data is then processed, including:

[0153] S1, take the target bit data from the RGB three-dimensional data of a pixel in a frame image, and take the other two bits of data from the RGB data of a pixel adjacent to the pixel in the image that are complementary to the target bit;

[0154] S2, recombine the target bit data and the other two bits of data to form the RGB data of a pixel in the image;

[0155] S3. Based on the relative positional relationship between the RGB data positions of the recombined pixels and the RGB data positions of the pixels before combination, adjust the RGB data positions of the recombined pixels so that the RGB data positions of the recombined pixels correspond one-to-one with the RGB data positions of the pixels before combination.

[0156] According to some embodiments of the present invention, adjusting the RGB data positions of the recombined pixels so that the RGB data positions of the recombined pixels correspond one-to-one with the RGB data positions of the pixels before combination includes:

[0157] By modifying the fragment shader configuration, the RGB data positions of the recombined pixels are made to correspond one-to-one with the RGB data positions before recombination.

[0158] According to some embodiments of the present invention, the method further includes performing steps S1, S2 and S3 sequentially on all pixels of the image row by row, or performing steps S1, S2 and S3 sequentially on all pixels of the image column by column.

[0159] According to some embodiments of the present invention, the method further includes retaining the data of pixels only every other row, and performing steps S1, S2 and S3 sequentially on the retained pixels row by row.

[0160] According to some embodiments of the present invention, the processor 51 performs the following actions based on a computer program stored in the memory 52:

[0161] The transcoded RGB data is processed, including:

[0162] The original data is converted to RGB format, and the converted RGB data is then processed, including:

[0163] S21, the RGB data of a frame of image is reduced by half according to the data of single-byte blocks by the libyuv's ScalePlane function. The reduction by half includes retaining the data of pixels only by alternating rows, taking the target bit data from the RGB three-bit data of one of the retained pixels, and taking the other two bits of data from the RGB data of a pixel adjacent to the pixel in the image that are complementary to the target bit.

[0164] S22, recombine the target bit data and the other two bits of data to form the RGB data of a pixel in the image;

[0165] S23 uses OpenGL's shader scripting functionality to adjust the RGB data positions of the recombined pixels based on the relative positional relationship between the RGB data positions of the recombined pixels and the RGB data positions of the pixels before recombining, so that the RGB data positions of the recombined pixels correspond one-to-one with the RGB data positions of the pixels before recombining.

[0166] In addition, the above refers to Figure 1a-3g The mobile processing device described herein and its various processing details are also included here by reference, and will not be repeated for the sake of simplicity.

[0167] According to further embodiments of the present invention, the computer program may be divided into one or more units in various ways and stored in the memory, and executed by the processor to perform the present invention. The one or more units may be a series of computer program instruction segments capable of performing a specific function, the instruction segments describing the execution process of the computer program in the device. The computer program may be based on the foregoing reference... Figures 4a-4c The functions of each unit in the various embodiments described are used to divide the system into multiple units, or include those referenced above. Figures 4a-4c The various units in the described embodiments are not repeated here for the sake of simplicity.

[0168] The processor referred to may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the device, connecting various parts of the device via various interfaces and lines. The device may be a desktop computer, laptop, handheld computer, cloud server, or other computing devices or a part thereof. The device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the schematic diagrams are merely examples of the device and do not constitute a limitation on the device.

[0169] This application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, performs the following steps:

[0170] Connect the video signal source via the HDMI-IN interface;

[0171] Based on the V4L framework, the device file of the HDMI-IN interface is opened in the Native layer of Android to capture raw data from the video signal source.

[0172] According to some embodiments of the present invention, when the computer program is executed by a processor, it further performs the following steps:

[0173] The method involves transcoding the raw data into RGB format and providing the transcoded data to a display output device for display; and / or encoding the raw data into network stream data for network transmission. According to some embodiments of the invention, the method further includes sending the network stream data to a terminal, whereby the terminal decodes the network stream data and outputs it to a screen.

[0174] According to some embodiments of the present invention, when the computer program is executed by a processor, it further performs the following steps:

[0175] The original data is converted to RGB format, and the converted RGB data is then processed, including:

[0176] S1, take the target bit data from the RGB three-dimensional data of a pixel in a frame image, and take the other two bits of data from the RGB data of a pixel adjacent to the pixel in the image that are complementary to the target bit;

[0177] S2, recombine the target bit data and the other two bits of data to form the RGB data of a pixel in the image;

[0178] S3. Based on the relative positional relationship between the RGB data positions of the recombined pixels and the RGB data positions of the pixels before combination, adjust the RGB data positions of the recombined pixels so that the RGB data positions of the recombined pixels correspond one-to-one with the RGB data positions of the pixels before combination.

[0179] According to some embodiments of the present invention, adjusting the RGB data positions of the recombined pixels so that the RGB data positions of the recombined pixels correspond one-to-one with the RGB data positions of the pixels before combination includes:

[0180] By modifying the fragment shader configuration, the RGB data positions of the recombined pixels are made to correspond one-to-one with the RGB data positions before recombination.

[0181] According to some embodiments of the present invention, the method further includes performing steps S1, S2 and S3 sequentially on all pixels of the image row by row, or performing steps S1, S2 and S3 sequentially on all pixels of the image column by column.

[0182] According to some embodiments of the present invention, the method further includes retaining the data of pixels only every other row, and performing steps S1, S2 and S3 sequentially on the retained pixels row by row.

[0183] According to some embodiments of the present invention, when the computer program is executed by a processor, it performs the following steps:

[0184] The original data is converted to RGB format, and the converted RGB data is then processed, including:

[0185] S21, the RGB data of a frame of image is reduced by half according to the data of single-byte blocks by the libyuv's ScalePlane function. The reduction by half includes retaining the data of pixels only by alternating rows, taking the target bit data from the RGB three-bit data of one of the retained pixels, and taking the other two bits of data from the RGB data of a pixel adjacent to the pixel in the image that are complementary to the target bit.

[0186] S22, recombine the target bit data and the other two bits of data to form the RGB data of a pixel in the image;

[0187] S23 uses OpenGL's shader scripting functionality to adjust the RGB data positions of the recombined pixels based on the relative positional relationship between the RGB data positions of the recombined pixels and the RGB data positions of the pixels before recombining, so that the RGB data positions of the recombined pixels correspond one-to-one with the RGB data positions of the pixels before recombining.

[0188] In addition, the above refers to Figure 1a-3g The mobile processing device described herein and its various processing details are also included here by reference, and will not be repeated for the sake of simplicity.

[0189] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0190] The above embodiments of the present invention overcome common cognitive biases of those skilled in the art regarding mobile processing devices under the Android operating system. They unusually introduce an HDMI interface and successfully collect the corresponding raw data. This allows the mobile processing device to quickly access and acquire video signal sources. Furthermore, it enables the mobile processing device to further process the collected raw data through encoding and other methods, acting as a video data relay device to further transmit the signal source data. For example, it can directly push the content of a host without a display device to a terminal with a display device, such as a network terminal, or allow the user to directly view the video signal source on the mobile processing device.

[0191] It should be noted that although the structure of the apparatus of the present invention and the method of its operation are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0192] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Those skilled in the art will recognize that the present invention can have various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A mobile processing device for processing a video signal source, the mobile processing device is equipped with an Android operating system; the mobile processing device comprises an HDMI-IN interface for accessing the video signal source; the mobile processing device opens a device file of the HDMI-IN interface at a Native layer of the Android based on a V4L framework to collect raw data of the video signal source; the mobile processing device transcodes the raw data into an RGB format and processes the transcoded data, the processing comprising: S21, reducing a frame of image RGB data by half through a ScalePlane function of a libyuv image processing library according to single-byte block data, the reducing by half comprising retaining pixel point data only every other line, and taking target bit data in RGB three-bit data of a pixel point, and taking other two-bit data complementary to the target bit in RGB data of a pixel point adjacent to the pixel point in the image; S22, recombining the target bit data and the other two-bit data to form RGB data of a pixel point of the image; S23, adjusting a position of the RGB data of the recombined pixel point according to a relative positional relationship between the position of the RGB data of the recombined pixel point and a position of the RGB data of the pixel point before recombination through a shader script function of OpenGL, so that the position of the RGB data of the recombined pixel point and the position of the RGB data of the pixel point before recombination respectively correspond one by one; the mobile processing device provides the processed RGB data to a display output device for display, or encodes the processed RGB data into network stream data for network transmission; wherein opening the device file of the HDMI-IN interface at the Native layer of the Android to collect the raw data of the video signal source comprises: opening a file handle corresponding to the HDMI-IN interface through a HDMI-IN interface name; obtaining relevant parameters of the HDMI-IN interface through an ioctl interface according to the file handle; selecting the relevant parameters and setting the HDMI-IN interface according to the selected relevant parameters; collecting the raw data of the video signal source according to the setting.

2. The mobile processing device of claim 1, wherein, the mobile processing device transcodes the raw data into the RGB format using the libyuv image processing library and provides the transcoded data to the display output device for display; and / or, the mobile processing device compresses the raw data into an H264 format using a mediacodec codec library for network transmission.

3. The mobile processing device of claim 1, wherein, the mobile processing device comprises a mobile communication processing device.

4. The mobile processing device of claim 3, wherein, the mobile communication processing device is a mobile phone. 5.A video signal source processing method based on the mobile processing device of any one of claims 1-4, comprising: accessing the video signal source through the HDMI-IN interface; Open a device file of the HDMI-IN interface based on a V4L framework in a Native layer of Android to collect raw data of a video signal source; The method further comprises transcoding the raw data into an RGB format, and processing the transcoded data, the processing comprising: S21, reducing the RGB data of a frame of image by half through a ScalePlane function of a libyuv image processing library according to single-byte block data, the reducing by half comprising retaining data of pixel points only every other line, and taking target bit data in RGB three-bit data of a pixel point, and taking other two-bit data complementary to the target bit in RGB data of a pixel point adjacent to the pixel point in the image; S22, recombining the target bit data and the other two-bit data to form RGB data of a pixel point of the image; S23, adjusting a position of the RGB data of the recombined pixel point according to a relative positional relationship between the position of the RGB data of the recombined pixel point and a position of the RGB data of the pixel point before recombination through a shader script function of OpenGL, so that the position of the RGB data of the recombined pixel point and the position of the RGB data of the pixel point before recombination correspond to each other one by one; The method further comprises providing the processed RGB data to a display output device for display, or encoding the processed RGB data into network stream data, and sending the network stream data to a terminal for output to a screen after decoding by the terminal.

6. The method of claim 5, wherein, Open a device file of the HDMI-IN interface based on a V4L framework in a Native layer of Android to collect raw data of a video signal source, comprising: Opening a file handle corresponding to the HDMI-IN interface through a HDMI-IN interface name; Obtaining relevant parameters of the HDMI-IN interface through an ioctl interface according to the file handle; Selecting the relevant parameters and setting the HDMI-IN interface according to the selected relevant parameters; Collecting raw data of a video signal source according to the setting.

7. A system for processing a video signal source, comprising the mobile processing device for processing a video signal source according to any one of claims 1-4 and a terminal; the mobile processing device encoding the collected raw data of the video signal source into a format matching a receiving mode of the terminal and sending to the terminal in response to an input forwarding request.

8. The system for processing a video signal source according to claim 7, the mobile processing device encoding the raw data of the video signal source into network stream data and sending to the terminal in response to an input forwarding request, and the terminal outputting the network stream data to a screen.

9. The system for processing a video signal source of claim 7, wherein, The mobile processing device providing the processed RGB data to a display output device for display, or encoding the processed RGB data into network stream data and sending to the terminal, and the terminal outputting the network stream data to a screen in response to an input forwarding request.

Images