Image tag data transmission method, device, computer equipment and storage medium

Abstract

Embodiments of the present application relate to the field of image data processing technology, and specifically, to a method, device, computer equipment, and storage medium for transmitting image marked data. The image marked data transmission method includes: the sending end performs binary serialization processing on the image data of the marked area; the sending end packages the first binary sequence data and the first original code stream of the original image into a media stream according to a preset code stream content format rule and a preset media stream content format to obtain a packaged media stream data packet; the receiving end decodes the received media stream data packet according to the preset code stream content format rule and the preset media stream content format, and performs image processing on the original image and the marked area based on the second binary sequence data and the second original code stream. A method for transmitting image marked data with higher timeliness, synchronization, and reliability is provided.

Description

Technical Field

[0001] The present application relates to the technical field of image data processing, and in particular to a method, apparatus, computer equipment and storage medium for transmitting image-tagged data. Background Art

[0002] With the development of artificial intelligence such as deep learning and neural network learning, region marking technology has gradually become feasible and applied in the field of video image processing, which is used to automatically identify and mark key areas such as people and specific content in video images. The marking information can be used as a reference for video encoder compression to improve the encoding quality of specific areas, and can also be used to automatically crop image areas in real-time multimedia streaming processing to achieve synchronous video transmission of different proportions, and can also be used for target tracking and identification in multimedia streaming monitoring images.

[0003] Usually, the description information of the marked area can be sent in through the encoder to define a specific data structure and method (these data are also called regions of interest - ROI), but the description information will not be further packaged into the bitstream and passed to the decoder, so it is impossible to obtain the original description information of the marked area after decoding. The MPEG-TS (hereinafter referred to as TS) data format commonly used for real-time multimedia streaming transmission does not define its transmission method. This results in the inability to directly perform image area cropping or screen content identification after receiving the media stream data. It usually requires additional transmission channels and data format encapsulation to realize the transmission of marked area information, but multi-channel transmission will bring about image and marked area data synchronization problems, and increase media cache delay.

[0004] Therefore, there is an urgent need for an image marking method that can retain the original marking information and has higher synchronization with the transmission of the original image information. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, computer device and storage medium for transmitting image-tagged data.

[0006] In a first aspect of an embodiment of the present application, a method for transmitting image mark data is provided, which is applied to a media stream transmission system, wherein the media stream transmission system includes at least a transmitting end and a receiving end, and the method includes at least:

[0007] The sending end performs binary serialization processing on the image data of the marked area to obtain first binary sequence data of the marked area;

[0008] The transmitting end packages the first binary sequence data and the first original code stream of the original image into a media stream according to a preset code stream content format rule and a preset media stream content format to obtain a packaged media stream data packet; wherein the marked area refers to any area in the original image;

[0009] The sending end sends the media stream data packet to the receiving end, and the receiving end decodes the received media stream data packet according to the preset code stream content format rule and the preset media stream content format to obtain the second binary sequence data of the marked area and the second original code stream of the original image;

[0010] The receiving end performs image processing on the original image and the marked area based on the second binary sequence data and the second original code stream.

[0011] In an optional embodiment of the present application, the sending end performs binary serialization processing on the image data of the marked area to obtain first binary sequence data of the marked area, including:

[0012] The sending end extracts binary serialized data of the marked area according to a preset format to obtain the first binary sequence data of the marked area; wherein the first binary sequence data at least includes: at least one of the number of areas, string labels, area coordinates, and area sizes of the marked area.

[0013] In an optional embodiment of the present application, the transmitting end packages the first binary sequence data and the first original code stream of the original image into a media stream according to a preset code stream content format rule and a preset media stream content format to obtain a packaged media stream data packet, including:

[0014] The sending end buffers the first binary sequence data;

[0015] After receiving the first original code stream output by the encoder, the sending end indexes the first original code stream and the first binary sequence data according to the timestamp of the original image, and packages the data with the same timestamp into the media stream according to the preset code stream content format rule and the preset media stream content format to obtain the packaged media stream data packet.

[0016] In an optional embodiment of the present application, after receiving the first original code stream output by the encoder, the transmitting end indexes the first original code stream and the first binary sequence data according to the timestamp of the original image, and packages the data with the same timestamp into the media stream according to the preset code stream content format rule and the preset media stream content format, to obtain the packaged media stream data packet, including:

[0017] If the first original code stream supports data extension, processing the first binary sequence data into first code stream extension data of the first original code stream;

[0018] The first code stream extension data and the first original code stream are packaged into a media stream according to the preset code stream content format rule and the preset media stream content format to obtain the packaged media stream data packet.

[0019] In an optional embodiment of the present application, the receiving end decodes the received media stream data packet according to the preset code stream content format rule and the preset media stream content format to obtain the second binary sequence data of the marked area and the second original code stream of the original image, including:

[0020] The receiving end splits the received media stream data packet into the second binary sequence data of the marked area and the second original code stream of the original image;

[0021] Parsing the second binary sequence data to obtain parsed marking area data;

[0022] Parsing the second original code stream to obtain parsed original image data;

[0023] The original image data and the marked area data are synchronously output according to the timestamp.

[0024] In an optional embodiment of the present application, if the second original code stream supports data extension, the second original code stream is parsed to obtain the parsed original image data and second code stream extension data; wherein the second code stream extension data refers to the first code stream extension data after encoding and decoding.

[0025] In an optional embodiment of the present application, the above-mentioned image mark data transmission method further includes:

[0026] The data length of the mark area data is verified based on a pre-configured verification tool.

[0027] According to a second aspect of the embodiments of the present application, there is provided an image tag data transmission device, which is applied to a media stream transmission system, wherein the media stream transmission system comprises at least a transmitting end and a receiving end, and the device comprises at least:

[0028] A first processing module, configured to perform binary serialization processing on the image data of the marked area at the sending end to obtain first binary sequence data of the marked area;

[0029] A packaging module is used for the sending end to package the first binary sequence data and the first original code stream of the original image into a media stream according to a preset code stream content format rule and a preset media stream content format, so as to obtain a packaged media stream data packet; wherein the marked area refers to any area in the original image;

[0030] A decoding module, configured for a transmitting end to send the media stream data packet to the receiving end, and the receiving end to decode the received media stream data packet according to the preset code stream content format rule and the preset media stream content format to obtain the second binary sequence data of the marked area and the second original code stream of the original image;

[0031] The second processing module is used for the receiving end to perform image processing on the original image and the marked area based on the second binary sequence data and the second original code stream.

[0032] According to a third aspect of an embodiment of the present application, a computer device is provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0033] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of any of the above methods when executed by a processor.

[0034] The image tag data transmission method provided in the embodiment of the present application:

[0035] In the first aspect, the image mark data transmission method provided by the embodiment of the present application performs binary serialization processing on the image data of the marked area to obtain the first binary sequence data of the marked area, which can facilitate the image data of the marked area to be loaded into a media stream such as a video coding stream and an MPEG-TS media stream in a unified format, thereby facilitating the transmission of the image data of the marked area;

[0036] On the second aspect, the image mark data transmission method provided by the embodiment of the present application can facilitate the image data of the marked area to be loaded into the media streams such as the video coding stream and the MPEG-TS media stream in a unified format by performing binary serialization processing on the image data of the marked area, thereby facilitating the transmission of the image data of the marked area. At the same time, the original image marked area information and the media stream can be synchronously transmitted through the extended data area in the video coding stream and the MPEG-TS media stream, thereby solving the problem of the original image data and the marked area data being out of sync in the traditional solution, greatly improving the synchronization of the original image data and the marked area data in the media stream, and further improving the timeliness and reliability of the media stream.

[0037] On the third aspect, the embodiments of the present application provide that the sending end can encapsulate the information of different marked areas into media streams such as MPEG-TS and video frame code streams for transmission together, and the receiving end directly decodes the received media stream data packet according to the preset code stream content format rule and the preset media stream content format to obtain the second binary sequence data of the marked area and the second original code stream of the original image. The second binary sequence data and the second original code stream of the original image can be used to directly perform image cropping or picture content identification, without introducing a multi-channel data synchronization processing process, without adding additional media stream cache, and reducing system complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 A flowchart of an image marking data transmission method provided by an embodiment of the present application;

[0040] Figure 2 An interaction diagram of a sending end and a receiving end in an image marking data transmission method provided by an embodiment of the present application;

[0041] Figure 3 A flowchart of packaging a media stream data packet at a sending end in an image mark data transmission method provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of a marked area in an original image in a method for transmitting image marked data provided by an embodiment of the present application;

[0043] Figure 5 A flowchart of an image marking data transmission method provided by an embodiment of the present application;

[0044] Figure 6 A flowchart of an image marking data transmission method provided by an embodiment of the present application;

[0045] Figure 7 A flowchart of decoding a media stream data packet at a receiving end in an image mark data transmission method provided in an embodiment of the present application;

[0046] Figure 8 A schematic diagram of the structure of an image marking data transmission device provided in one embodiment of the present application;

[0047] Fig. 9 A schematic diagram of the computer device structure provided for one embodiment of the present application. DETAILED DESCRIPTION

[0048] In the process of implementing the present application, the inventors found that there is an urgent need for an image marking method that can retain the original marking information and has a higher synchronization with the transmission of the original image information.

[0049] In response to the above problems, embodiments of the present application provide an image marking data transmission method, apparatus, computer equipment and storage medium to improve the transmission integrity of the marking information and the synchronization with the original image data transmission.

[0050] The solutions in the embodiments of the present application can be implemented using various computer languages, for example, the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0051] In order to make the technical solutions and advantages in the embodiments of the present application more clearly understood, the exemplary embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0052] The following is a brief description of the application environment of the image tag data transmission method provided in the embodiment of the present application:

[0053] An image marking data transmission method provided in an embodiment of the present application is applied to a media stream transmission system, which is used for data processing and interaction between media, such as image editing, video editing, etc. The media stream transmission system at least includes a sending end and a receiving end, and between the sending end and the receiving end. Among them, the sending end refers to a device that contains the original image and marks the original image to obtain a marked area, and the receiving end refers to a device that receives the media stream data packet sent by the sending end. The device can be a server, a computer, a mobile terminal, a wearable device, etc., and the embodiment of the present application does not make specific limitations and can be flexibly adjusted according to actual conditions.

[0054] See also Figure 1-Figure 3 The following embodiments take the above-mentioned media stream transmission system as the execution subject, and apply the method provided in the embodiment of the present application to the above-mentioned media stream transmission system to transmit the original image containing the marked area as an example for specific description. The image marked data transmission method provided in the embodiment of the present application includes the following steps 101 to 104:

[0055] Step 101: The sending end performs binary serialization processing on the image data of the marked area to obtain first binary sequence data of the marked area;

[0056] The transmission and application process of the marker area in the media streaming transmission system is as follows: Figure 2As shown, in a media stream transmission system, the image data of the marked area needs to be accompanied by most of the processing steps of the original image data, such as encoding, media stream packaging, media stream unpacking, post-image processing, etc. In the embodiment of the present application, the sending end performs binary serialization processing on the image data of the marked area, that is, encapsulates the image data of the marked area, binary serializes the image data of each marked area, and then packages it according to the corresponding preset code stream content format specification and the preset media stream content format specification, and uniformly defines the description method of the marked area information, and then loads the image data of the marked area into the media stream through subsequent steps, and the receiving end can correctly restore the original marked area information by unpacking the corresponding binary data information.

[0057] The marked area refers to any area in the original image, for example, it can be any irregular shape such as a rectangle, a circle, etc. In order to more clearly illustrate the image marked data transmission method provided in the embodiment of the present application, the following embodiment takes the marked area as a matrix for example. Figure 4 , including marked area 1 and marked area 2. The marked area is a rectangular area in the original image area, which includes: starting coordinates, rectangular area width, rectangular area height and area label information. A plane rectangular coordinate system is established with the width and height of the original image. The coordinate system takes the upper left corner of the original image as the origin, the positive half axis of the x-axis from left to right, and the positive half axis of the y-axis from the upper left to the bottom. The starting coordinates range from 0 to the width of the original image minus 1 on the x-axis, and range from 0 to the height of the original image minus 1 on the y-axis to ensure the integrity of the marked area information. The width of the key area ranges from 1 to the width of the original image, and the height ranges from 1 to the height of the original image. The label information of the original image area can be represented by a string, which can be used as a content identifier or a reference for area cropping at the media stream receiving end to facilitate the processing and transmission of image data.

[0058] Step 102: The transmitting end packages the first binary sequence data and the first original code stream of the original image into a media stream according to a preset code stream content format rule and a preset media stream content format to obtain a packaged media stream data packet;

[0059] A media stream data packet refers to a media data packet exchanged between devices. The media stream data packet in the embodiment of the present application is illustrated by taking a TS (Transport Stream) packet as an example.

[0060] Step 103: The transmitting end transmits the media stream data packet to the receiving end, and the receiving end decodes the received media stream data packet according to the preset code stream content format rule and the preset media stream content format to obtain the second binary sequence data of the marked area and the second original code stream of the original image;

[0061] Step 104: The receiving end performs image processing on the original image and the marked area based on the second binary sequence data and the second original code stream.

[0062] The second binary sequence data corresponds to the first binary sequence data, and is data obtained by encoding and decoding the first binary sequence data; correspondingly, the second original code stream is code stream data obtained by encoding and decoding the first original code stream.

[0063] In the image marked data transmission method provided by the embodiment of the present application, a sending end performs binary serialization processing on the image data of the marked area to obtain first binary sequence data of the marked area; the sending end packages the first binary sequence data and the first original code stream of the original image into a media stream according to a preset code stream content format rule and a preset media stream content format to obtain a packaged media stream data packet; wherein the marked area refers to any area in the original image; the sending end sends the media stream data packet to the receiving end, and the receiving end decodes the received media stream data packet according to the preset code stream content format rule and the preset media stream content format to obtain second binary sequence data of the marked area and a second original code stream of the original image; the receiving end performs image processing on the original image and the marked area based on the second binary sequence data and the second original code stream.

[0064] In the first aspect, the image mark data transmission method provided by the embodiment of the present application performs binary serialization processing on the image data of the marked area to obtain the first binary sequence data of the marked area, which can facilitate the image data of the marked area to be loaded into a media stream such as a video coding stream and an MPEG-TS media stream in a unified format, thereby facilitating the transmission of the image data of the marked area;

[0065] On the second aspect, the image mark data transmission method provided by the embodiment of the present application can facilitate the image data of the marked area to be loaded into the media streams such as the video coding stream and the MPEG-TS media stream in a unified format by performing binary serialization processing on the image data of the marked area, thereby facilitating the transmission of the image data of the marked area. At the same time, the original image marked area information and the media stream can be synchronously transmitted through the extended data area in the video coding stream and the MPEG-TS media stream, thereby solving the problem of the original image data and the marked area data being out of sync in the traditional solution, greatly improving the synchronization of the original image data and the marked area data in the media stream, and further improving the timeliness and reliability of the media stream.

[0066] On the third aspect, the embodiments of the present application provide that the sending end can encapsulate the information of different marked areas into media streams such as MPEG-TS and video frame code streams for transmission together, and the receiving end directly decodes the received media stream data packet according to the preset code stream content format rule and the preset media stream content format to obtain the second binary sequence data of the marked area and the second original code stream of the original image. The second binary sequence data and the second original code stream of the original image can be used to directly perform image cropping or picture content identification, without introducing a multi-channel data synchronization processing process, without adding additional media stream cache, and reducing system complexity.

[0067] In an optional embodiment of the present application, the above step 101, the sending end performs binary serialization processing on the image data of the marked area to obtain the first binary sequence data of the marked area, includes the following steps:

[0068] The sending end extracts binary serialized data from the marked area according to a preset format to obtain the first binary sequence data of the marked area;

[0069] The first binary sequence data at least includes: at least one of the number of regions, string labels, string labels, region coordinates, and region sizes of the marked regions. The region data refers to the number of marked regions in the original image, the string labels are used to represent the image data of each module in the marked region, the region coordinates are used to represent the specific position of each marked region in the original image, and the region size refers to any parameter that can represent the size of the marked region, such as the width and height of the marked region.

[0070] The data syntax of the first binary sequence data of the generated marking area can be expressed as follows in Table 1:

[0071] Table 1

[0072]

[0073] The definitions of the various fields in the syntax in Table 1 are as follows:

[0074] region_count has a value range of 1 to 255, indicating how many regions are included in the subsequent data;

[0075] The value range of region_label_length is 0 to 255, which indicates the number of label characters of a region, that is, the effective length of the region_label field;

[0076] region_label indicates the region label string. If region_label[0] is '?', it means using the URL parameter format. If it is '<', it means using the XML format. If it is '{', it means using the JSON format.

[0077] region_left corresponds to the x-axis value of the region's starting coordinate, ranging from 0 to the image width minus 1;

[0078] region_top corresponds to the y-axis value of the region's starting coordinate, ranging from 0 to the image height minus 1;

[0079] region_width indicates the width of the region, ranging from 1 to the image width, and should be less than or equal to the image width after adding region_left;

[0080] region_height indicates the height of the region, and its value range is from 1 to the image height. After adding region_top, it should be less than or equal to the image height.

[0081] The sending end of the embodiment of the present application extracts binary serialized data of the marked area according to a preset format, and can obtain first binary sequence data including the number of marked areas, string labels, string labels, area coordinates, area sizes and other information, which can more accurately characterize the specific content of the marked area, thereby improving the accuracy and reliability of the transmission of the marked area image data.

[0082] See also Figure 5 In an optional embodiment of the present application, the above step 102, the transmitting end packages the first binary sequence data and the first original code stream of the original image into a media stream according to a preset code stream content format rule and a preset media stream content format to obtain a packaged media stream data packet, including the following steps 501-502:

[0083] Step 501: The sending end caches the first binary sequence data;

[0084] For example, the cache is stored in a flash memory module or other location at the sending end.

[0085] Step 502: After receiving the first original code stream output by the encoder, the sending end indexes the first original code stream and the first binary sequence data according to the timestamp of the original image, and packages the data with the same timestamp into the media stream according to the preset code stream content format rule and the preset media stream content format to obtain the packaged media stream data packet.

[0086] Since the encoder may have output delays, the embodiment of the present application first caches the marked area data corresponding to the image frame (i.e., the first binary sequence data), and then uses the image frame timestamp for indexing after receiving the first original code stream output by the encoder. The marked area data is synchronously packaged by querying the timestamp, which can avoid the encoder delay in another dimension, further improve the synchronization between the marked area image data and the original image data, and further improve the timeliness and reliability of the image marked data transmission method provided by the embodiment of the present application.

[0087] In an optional embodiment of the present application, in the above step 502, after receiving the first original code stream output by the encoder, the transmitting end indexes the first original code stream and the first binary sequence data according to the timestamp of the original image, and packages the data with the same timestamp into the media stream according to the preset code stream content format rule and the preset media stream content format to obtain the packaged media stream data packet, including the following two cases:

[0088] Case 1: if the first original code stream supports data extension, the first binary sequence data is processed into first code stream extension data of the first original code stream;

[0089] If the code stream supports data extension, execute Figure 3 Generate code stream extension data in and package the code stream extension data with the original code stream output by the encoder. H264 / H265 code stream carries custom extension data in the Supplemental Enhancement Information (SEI) data:

[0090] The marker region information is packaged using the user_data_unregistered syntax in SEI as defined in "ISO / IEC 14496-10" and "ISO / IEC 23008-2".

[0091] When the marker area data contains continuous 0x00 0x00 0x00 or 0x00 0x00 0x01, it is necessary to insert 0x03 to perform contention prevention processing, that is, to process it in the manner of Encapsulate Byte Sequence Payload (EBSP).

[0092] When the processed EBSP data is larger than 239 bytes, it is split into multiple consecutive user_data_unregistered data, and each user_data_unregistered data stores a maximum of 239 bytes of data.

[0093] Correspondingly, the above steps package the first code stream extension data and the first original code stream into a media stream according to the preset code stream content format rule and the preset media stream content format to obtain the packaged media stream data packet.

[0094] Execute media stream data packet (media stream data packet) generation: According to the definition of "ISO / IEC 13818-1", the marker area data is stored in the private data field of the media stream data packet header. At the same time, the private data shall be organized according to the description of Appendix D.2 of `DVB - ETSI TS101 154`. The private data in each media stream data packet consists of 8-bit data type, 8-bit data length and data payload. The marker area data is packaged before the code stream data.

[0095] The private data type is fixed to 193 to facilitate parsing and identification at the receiving end. Each TS data packet has a fixed length of 188 bytes. After deducting the necessary media stream data packet header overhead, the private data length ranges from 0 to 179. If the marker area data length is greater than the private data length, it is split into multiple consecutive media stream data packets for storage.

[0096] Even if the above processing has carried the marker area data in the code stream, the TS data packet still needs to carry the marker area data, which can adapt to the situation where the receiving end cannot directly parse the code stream extension data. At the same time, the area information extension data in the code stream can be directly used for retransmission or persistence.

[0097] The first original code stream supports data extension. By processing the first binary sequence data into the first code stream extended data of the first original code stream through the extended data area in the media stream, the original image marking area information and the media stream can be synchronously transmitted, thereby solving the problem of the original image data and the marking area data being out of synchronization in the traditional solution, greatly improving the synchronization of the original image data and the marking area data in the media stream, and further improving the timeliness and reliability of the media stream.

[0098] The second case: if the first original code stream does not support data extension, then directly follow the above step 502, after the sending end receives the first original code stream output by the encoder, the first original code stream and the first binary sequence data are indexed according to the timestamp of the original image, and the data with the same timestamp are packaged into the media stream according to the preset code stream content format rule and the preset media stream content format to obtain the packaged media stream data packet.

[0099] See also Figure 6 and Figure 7 In an optional embodiment of the present application, the above step 103, the receiving end decodes the received media stream data packet according to the preset code stream content format rule and the preset media stream content format to obtain the second binary sequence data of the marked area and the second original code stream of the original image, includes the following steps 601-604:

[0100] Step 601: The receiving end splits the received media stream data packet into the second binary sequence data of the marked area and the second original code stream of the original image;

[0101] Step 602: Parse the second binary sequence data to obtain parsed marking area data;

[0102] The receiving end needs to parse the marked area data from the media stream data stream first to parse the media stream data packet to split the custom extended data (i.e. the second binary sequence data) and the code stream data (i.e. the second original code stream) in the media stream, and then parse them separately. The working process is as follows: Figure 7 As shown:

[0103] The second binary sequence data can be parsed from the media stream data packet according to the format defined in Appendix D.2 of "DVB - ETSI TS 101 154". It can be determined whether the data type value is 193. If so, the second binary sequence data can be extracted according to the order in which the media stream data packets arrive.

[0104] After obtaining the parsed marked area data, the parsed original image data is bound to the bitstream timestamp (PTS) carried in the TS and sent to the cache.

[0105] Step 603: Parse the second original code stream to obtain parsed original image data;

[0106] Step 604: synchronously output the original image data and the marked area data according to the timestamp.

[0107] After the video image is decoded, the marked area data is queried through the timestamp and then output synchronously. Due to different encoding technologies (such as H264 B frames) and decoder characteristics, decoding may be delayed, and there is a difference between the sequence input to the decoder and the sequence output. The marked area data needs to be cached first, and then synchronized with the timestamp and the frame sequence output by the decoder before being output. The encoder delay can be avoided, the synchronization between the marked area image data and the original image data can be further improved, and the timeliness and reliability of the image marking data transmission method provided in the embodiment of the present application can be further improved.

[0108] In an optional embodiment of the present application, if the second original code stream supports data extension, the second original code stream is parsed to obtain the parsed original image data and the second code stream extension data;

[0109] The second code stream extension data refers to the first code stream extension data after encoding and decoding. If the second original code stream supports data extension, the extension data in the code stream is parsed.

[0110] If the second original code stream does not support data extension, directly parse the media stream data packet to obtain the parsed original image data and marked area data;

[0111] In an optional embodiment of the present application, the above-mentioned image mark data transmission method further includes the following steps:

[0112] The data length of the mark area data is verified based on a pre-configured verification tool.

[0113] When parsing the marked region data, the data can be verified and parsed according to the marked region data binary sequence syntax, for example, the data length can be verified by region_count and region_label_length. If the actual data length is not equal to the sum of region_label_length, the verification fails. After the verification is successful, the subsequent steps are continued. If the complete marked region data has been parsed in the media stream data packet, the code stream extension data can be no longer parsed, so as to save the entire process and improve the transmission efficiency of the marked image data.

[0114] In an optional embodiment of the present application, the receiving end performs image processing on the original image and the marked area based on the second binary sequence data and the second original code stream in the above step 104. The image processing process includes but is not limited to marking the marked area of ​​the original image. Before or after forming the marked area, the original image may also be subjected to the following optimization processing steps:

[0115] Get the original image and convert it into HSI color space, extract the brightness component in HSI color space and enhance the brightness;

[0116] Adjust the contrast of the enhanced brightness component, and perform brightness area division and dynamic range optimization;

[0117] The optimized brightness components are edge enhanced and recombined to generate an enhanced HSI image, and the enhanced HSI image is converted back to RGB color space to obtain an optimized multimedia image.

[0118] In an optional embodiment of the present application, the obtaining of the original image and performing HSI color space conversion refers to extracting the original image data from the multimedia device, converting the obtained image format into a standard format, and performing noise smoothing on the image by mean filtering;

[0119] Normalize the color channel data of the preprocessed original image, and calculate the brightness component I, saturation S and hue H of the image based on the normalized color channel data;

[0120] Get the converted HSI color space and the corresponding HSI image.

[0121] In an optional embodiment of the present application, extracting the brightness component in the HSI color space and performing brightness enhancement refers to extracting the brightness component I from the HSI color space and using it as an independent channel, normalizing the extracted brightness component I, and then smoothing the brightness component through a bilateral filter, and outputting the final brightness component after completing edge smoothing and normalization.

[0122] Apply a brightness conversion function to the brightness component to enhance the contrast and depth of the original brightness:

[0123]

[0124] Where A is the pixel brightness value after brightness conversion, a(x) is the brightness enhancement adjustment parameter, and I(x) is the brightness component;

[0125] A Gaussian pyramid is constructed for the brightness component after brightness conversion. The pyramid is set to n layers to generate images at multiple resolution levels. The initial layer directly takes the converted brightness component. From the second layer onwards, the previous layer image is Gaussian blurred and downsampled layer by layer to generate low-resolution images at each layer until the nth layer is reached.

[0126] After reaching the nth layer, a multi-layer Gaussian pyramid is obtained, each layer represents the low-frequency information of brightness at different spatial scales;

[0127] Construct a Laplacian pyramid based on the Gaussian pyramid, extract high-frequency information from each layer of the Gaussian pyramid image, upsample the Gaussian pyramid image of the l+1th layer by bilinear interpolation, and calculate the Laplacian pyramid image of the lth layer Ll(x,y) using the following formula:

[0128]

[0129] Where Gl(x, y) is the lth layer image of the Gaussian pyramid, Gl+1(x, y) is the l+1th layer image of the Gaussian pyramid, U() is the upsampling operation, which enlarges Gl+1(x, y) to the same resolution as the lth layer Gl(x, y), and method represents the interpolation method used for upsampling;

[0130] The calculated Stored as the high-frequency component of the l-th layer of the Laplacian pyramid;

[0131] The last layer of the Gaussian pyramid is used as the lowest frequency information of the Laplace pyramid, and the high frequency information of each layer is integrated with the low frequency basic brightness information to form a complete Laplace pyramid structure. is the multi-scale brightness component;

[0132] The brightness component of each layer is input into the initial convolution layer of DCNN to capture the basic brightness features through convolution operation, and the convolution structure is processed using the ReLU activation function to extract the basic details of brightness;

[0133] In the middle layer of DCNN, the output of the initial convolutional layer H(z) is processed by multiple layers of convolution, and each layer of convolution is based on the output of the previous layer:

[0134]

[0135] In the formula, represents the brightness feature of the k-th layer output, is the k-th convolution kernel weight, is the brightness feature output from the k-1th layer, is the bias item of the current layer;

[0136] The output of each convolution layer is batch normalized and then the ReLU activation function is used:

[0137]

[0138] In the formula, Represents the brightness feature output after the k-th layer of convolution, batch normalization and activation processing. BN() is the batch normalization operation;

[0139] After multiple layers of convolution, batch normalization and activation function, the deep extracted brightness features are finally obtained;

[0140] Based on the brightness component I, at each pixel x in the image, take N The local brightness mean and standard deviation of pixel x are calculated for the pixels in the N windows, and the local brightness mean and local standard deviation of all pixels are combined to generate the final illumination estimation map T(x). The weight matrix W(x) is generated based on the illumination estimation map.

[0141] The multi-scale brightness components and deep brightness features are adjusted to the same resolution through upsampling and downsampling methods;

[0142] The multi-channel depth brightness features are merged using the mean of all channels and converted into single-channel brightness features. After the format is unified, weighted fusion is performed through the weight matrix to obtain the fused brightness component. Starting from the lowest frequency layer, the Laplacian pyramid components of each layer are upsampled and superimposed in turn to gradually reconstruct the complete brightness component to obtain the enhanced brightness component. .

[0143] In an optional embodiment of the present application, adjusting the contrast of the enhanced brightness component refers to obtaining the enhanced brightness component and performing normalization processing, defining the solution population size of the water cycle algorithm as P, and each solution represents a set of contrast parameters (s, d), where s refers to the contrast scaling factor and d refers to the brightness offset;

[0144] Initialize the river solution and the stream solution, and simulate the process of water flow to find the optimal solution;

[0145] According to the parameters (s, d) of the current solution, the contrast of the normalized brightness component is adjusted:

[0146]

[0147] In the formula, is the adjusted brightness value, is the normalized brightness value;

[0148] Calculate the objective function value J for the adjusted brightness component to evaluate the optimization effect of the current solution:

[0149]

[0150] Where M and N are the width and height of the image, representing the total range of pixels. is the local brightness mean, x represents the horizontal position of the pixel, and y represents the vertical position of the pixel;

[0151] Update the river solution and the stream solution according to the objective function value;

[0152] Set the maximum number of iterations until the maximum number of iterations is reached and stop the iteration. After the water cycle algorithm completes the iteration, select the solution with the smallest objective function value and use its parameters as the final optimal solution. Use the optimal solution to adjust the brightness component and perform range clipping on the optimized brightness value to obtain the brightness component after contrast adjustment. .

[0153] In an optional embodiment of the present application, the brightness region division and dynamic range optimization refers to dividing the brightness component Divide into bright, dark and mid-tone areas, set the threshold H and , if the brightness value , then the pixel is classified as a brightness area. If the brightness value , then the pixel is classified as a dark area. If the brightness value , then the pixel is classified as the midtone area;

[0154] For the brightness area, a nonlinear compression function is used to reduce the output of the brightness of the bright part. For the dark area, a logarithmic transformation is used to enhance the dark details. For the mid-tone area, a linear adjustment is performed to balance the brightness transition.

[0155] Combines adjustments to the highlight, shadow, and midtone areas into a final dynamic range optimized luminance component.

[0156] In an optional embodiment of the present application, the edge enhancement of the optimized brightness component and the recombining to generate the enhanced HSI image refers to using the Laplacian operator to enhance the edge details of the brightness component after dynamic range optimization, keeping the brightness component consistent with the hue and saturation in the original HSI image, and The brightness channel in the original HSI image is replaced, and the enhanced brightness component is recombined with the hue and saturation components in the HSI color space to form the final HSI image.

[0157] In an optional embodiment of the present application, converting the enhanced HSI image back to the RGB color space to obtain an optimized multimedia image means converting the HSI image into an RGB image through the image processing library OpenCV to obtain an optimized multimedia image, saving the generated multimedia image in a standard format and performing data backup.

[0158] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0159] See also Figure 8 An embodiment of the present application provides an image tag data transmission device 800, which is applied to a media stream transmission system. The media stream transmission system at least includes a transmitting end and a receiving end. The image tag data transmission device 800 includes: a first processing module 810, a packaging module 820, a decoding module 830 and a second processing module 840, wherein:

[0160] The first processing module 810 is used for performing binary serialization processing on the image data of the marked area at the sending end to obtain first binary serial data of the marked area;

[0161] The packaging module 820 is used for the transmitter to package the first binary sequence data and the first original code stream of the original image into a media stream according to a preset code stream content format rule and a preset media stream content format to obtain a packaged media stream data packet; wherein the marked area refers to any area in the original image;

[0162] The decoding module 830 is used for the transmitting end to send the media stream data packet to the receiving end, and the receiving end decodes the received media stream data packet according to the preset code stream content format rule and the preset media stream content format to obtain the second binary sequence data of the marked area and the second original code stream of the original image;

[0163] The second processing module 840 is used for the receiving end to perform image processing on the original image and the marked area based on the second binary sequence data and the second original code stream.

[0164] In an optional embodiment of the present application, the first processing module 810 is specifically used for the sending end to extract binary serialized data of the marked area according to a preset format to obtain the first binary sequence data of the marked area; wherein the first binary sequence data includes at least: at least one of the number of areas, string labels, area coordinates, and area sizes of the marked area.

[0165] In an optional embodiment of the present application, the packaging module 820 is specifically used for the sending end to cache the first binary sequence data; after receiving the first original code stream output by the encoder, the sending end indexes the first original code stream and the first binary sequence data according to the timestamp of the original image, and packages the data with the same timestamp into the media stream according to the preset code stream content format rule and the preset media stream content format to obtain the packaged media stream data packet.

[0166] In an optional embodiment of the present application, the packaging module 820 is specifically used to, if the first original code stream supports data extension, process the first binary sequence data into first code stream extension data of the first original code stream; package the first code stream extension data and the first original code stream into the media stream according to the preset code stream content format rule and the preset media stream content format, to obtain the packaged media stream data packet.

[0167] In an optional embodiment of the present application, the packaging module 820 is specifically used for the receiving end to split the received media stream data packet into the second binary sequence data of the marked area and the second original code stream of the original image; parse the second binary sequence data to obtain the parsed marked area data; parse the second original code stream to obtain the parsed original image data; and synchronously output the original image data and the marked area data according to the timestamp.

[0168] In an optional embodiment of the present application, if the second original code stream supports data extension, the second original code stream is parsed to obtain the parsed original image data and second code stream extension data; wherein the second code stream extension data refers to the first code stream extension data after encoding and decoding.

[0169] In an optional embodiment of the present application, the second processing module 840 is further configured to verify the data length of the mark area data based on a pre-configured verification tool.

[0170] For the specific definition of the above-mentioned image marking data transmission device 800, please refer to the definition of the image marking data transmission method above, which will not be repeated here. The various modules in the above-mentioned image marking data transmission device 800 can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0171] In one embodiment, a computer device is provided, the internal structure diagram of which can be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an image marking data transmission method as described above is implemented. It includes: a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, any step in the image marking data transmission method as described above is implemented.

[0172] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any step in the above-mentioned image-marked data transmission method can be implemented.

[0173] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0174] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0175] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0177] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0178] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for transmitting image-marked data, characterized in that: Applied to a media stream transmission system, the media stream transmission system includes at least a sending end and a receiving end, and the method includes at least: The sending end performs binary serialization processing on the image data of the marked area to obtain first binary sequence data of the marked area; The transmitting end packages the first binary sequence data and the first original code stream of the original image into a media stream according to a preset code stream content format rule and a preset media stream content format to obtain a packaged media stream data packet; wherein the marked area refers to any area in the original image; The sending end sends the media stream data packet to the receiving end, and the receiving end decodes the received media stream data packet according to the preset code stream content format rule and the preset media stream content format to obtain the second binary sequence data of the marked area and the second original code stream of the original image; The receiving end performs image processing on the original image and the marked area based on the second binary sequence data and the second original code stream; Before or after forming the marked area, the original image is optimized as follows: Get the original image and convert it into HSI color space, extract the brightness component in HSI color space and enhance the brightness; Extract the brightness component I from the HSI color space and use it as an independent channel. After normalizing the extracted brightness component I, smooth the brightness component through a bilateral filter. After completing edge smoothing and normalization, output the final brightness component. Apply a brightness conversion function to the brightness component to enhance the contrast and depth of the original brightness: ； Where A is the pixel brightness value after brightness conversion, a(x) is the brightness enhancement adjustment parameter, and I(x) is the brightness component; A Gaussian pyramid is constructed for the brightness component after brightness conversion. The pyramid is set to n layers to generate images at multiple resolution levels. The initial layer directly takes the converted brightness component. From the second layer onwards, the previous layer image is Gaussian blurred and downsampled layer by layer to generate low-resolution images at each layer until the nth layer is reached. After reaching the nth layer, a multi-layer Gaussian pyramid is obtained, each layer represents the low-frequency information of brightness at different spatial scales; Construct a Laplacian pyramid based on the Gaussian pyramid, extract high-frequency information from each layer of the Gaussian pyramid image, upsample the Gaussian pyramid image of the l+1th layer by bilinear interpolation, and calculate the Laplacian pyramid image of the lth layer using the following formula: ； In the formula, G l (x, y) is the lth layer image of the Gaussian pyramid, G l+1 (x, y) is the l+1th layer image of the Gaussian pyramid, U() is the upsampling operation, G l+1 (x,y) is enlarged to the lth layer G l (x, y) have the same resolution, method indicates the interpolation method used for upsampling; The calculated Stored as the high-frequency component of the l-th layer of the Laplacian pyramid; The last layer of the Gaussian pyramid is used as the lowest frequency information of the Laplace pyramid, and the high frequency information of each layer is integrated with the low frequency basic brightness information to form a complete Laplace pyramid structure. is the multi-scale brightness component; The brightness component of each layer is input into the initial convolution layer of DCNN to capture the basic brightness features through convolution operation, and the convolution structure is processed using the ReLU activation function to extract the basic details of brightness; In the middle layer of DCNN, the output of the initial convolutional layer H(z) is processed by multiple layers of convolution, and each layer of convolution is based on the output of the previous layer: ； In the formula, represents the brightness feature of the k-th layer output, is the k-th convolution kernel weight, is the brightness feature output from the k-1th layer, is the bias item of the current layer; The output of each convolution layer is batch normalized and then the ReLU activation function is used: ； In the formula, Represents the brightness feature output after the k-th layer of convolution, batch normalization and activation processing. BN() is the batch normalization operation; After multiple layers of convolution, batch normalization and activation function, the deep extracted brightness features are finally obtained; Based on the brightness component I, at each pixel x in the image, take N The local brightness mean and standard deviation of pixel x are calculated for the pixels in the N windows, and the local brightness mean and local standard deviation of all pixels are combined to generate the final illumination estimation map T(x). The weight matrix W(x) is generated based on the illumination estimation map. The multi-scale brightness components and deep brightness features are adjusted to the same resolution through upsampling and downsampling methods; The multi-channel depth brightness features are merged using the mean of all channels and converted into single-channel brightness features. After the format is unified, weighted fusion is performed through the weight matrix to obtain the fused brightness component. Starting from the lowest frequency layer, the Laplacian pyramid components of each layer are upsampled and superimposed in turn to gradually reconstruct the complete brightness component to obtain the enhanced brightness component. .

2. The image mark data transmission method according to claim 1, characterized in that: The sending end performs binary serialization processing on the image data of the marked area to obtain first binary sequence data of the marked area, including: The sending end extracts binary serialized data of the marked area according to a preset format to obtain the first binary sequence data of the marked area; wherein the first binary sequence data at least includes: at least one of the number of areas, string labels, area coordinates, and area sizes of the marked area.

3. The image mark data transmission method according to claim 1, characterized in that: The transmitting end packages the first binary sequence data and the first original code stream of the original image into a media stream according to a preset code stream content format rule and a preset media stream content format to obtain a packaged media stream data packet, including: The sending end buffers the first binary sequence data; After receiving the first original code stream output by the encoder, the sending end indexes the first original code stream and the first binary sequence data according to the timestamp of the original image, and packages the data with the same timestamp into the media stream according to the preset code stream content format rule and the preset media stream content format to obtain the packaged media stream data packet.

4. The image mark data transmission method according to claim 3, characterized in that: After receiving the first original code stream output by the encoder, the transmitting end indexes the first original code stream and the first binary sequence data according to the timestamp of the original image, and packages the data with the same timestamp into the media stream according to the preset code stream content format rule and the preset media stream content format, to obtain the packaged media stream data packet, including: If the first original code stream supports data extension, processing the first binary sequence data into first code stream extension data of the first original code stream; The first code stream extension data and the first original code stream are packaged into a media stream according to the preset code stream content format rule and the preset media stream content format to obtain the packaged media stream data packet.

5. The image mark data transmission method according to claim 4, characterized in that: The receiving end decodes the received media stream data packet according to the preset code stream content format rule and the preset media stream content format to obtain the second binary sequence data of the marked area and the second original code stream of the original image, including: The receiving end splits the received media stream data packet into the second binary sequence data of the marked area and the second original code stream of the original image; Parsing the second binary sequence data to obtain parsed marking area data; Parsing the second original code stream to obtain parsed original image data; The original image data and the marked area data are synchronously output according to the timestamp.

6. The image mark data transmission method according to claim 5, characterized in that: If the second original code stream supports data extension, the second original code stream is parsed to obtain the parsed original image data and second code stream extension data; wherein the second code stream extension data refers to the first code stream extension data after encoding and decoding.

7. The image mark data transmission method according to claim 5, characterized in that: Also includes: The data length of the mark area data is verified based on a pre-configured verification tool.

8. A computer device comprising: The method comprises a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images