Non-heritage tourism news low-delay image communication production and display system and method

Abstract

The application relates to the fields of video image processing and communication technology, and discloses a low-delay image communication acquisition and display system and method for non-heritage tourism news, which comprises a background base map storage module for pre-transmitting a static base map and an interactive instruction file; a region light detection module for calculating a sparse light parameter matrix of a real-time picture; a light smoothing processing module for generating a continuous pixel-level light adjustment map through interpolation; a dynamic foreground extraction and differential transmission module configured to correct the base map by using the adjustment map to build a virtual reference base map, extract a dynamic foreground image fragment through difference, and pack and transmit the light parameter and the foreground fragment; and a terminal synthesis display module for rendering a local base map by using the light parameter, superimposing the foreground fragment to restore the picture, and responding to touch interaction. By transmitting only the light parameter and the foreground fragment, the transmission data volume and delay are significantly reduced, and the problems of background synthesis and semantic interaction under a complex light environment are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video image processing and communication technology, in particular to a low-delay image communication collecting and editing display system and method for intangible cultural heritage tourism news. BACKGROUND

[0002] With the development of digital tourism industry, the collecting and dissemination of intangible cultural heritage tourism news increasingly relies on real-time video communication technology. In the existing news collecting process, the front reporter usually uses a mobile camera acquisition device to shoot the intangible cultural heritage performance scene or folk activities, and transmits the video stream to the user terminal through 4G or 5G mobile network. In order to adapt to the network bandwidth limitation, the traditional scheme mainly uses general video coding standards such as H.264, H.265, etc., to compress the image sequence through intra prediction and inter motion compensation technology, so as to reduce the data code rate in the transmission process, thereby realizing the remote display of intangible cultural heritage pictures.

[0003] The intangible cultural heritage performance scene usually has the characteristics of drastic and complex light and shadow changes, such as switching of stage lights, fluctuation of outdoor light intensity, etc. Although the traditional video coding technology can compress data, when facing frequent changes of full-screen light, the residual data amount of inter prediction is still large, resulting in high delay or stuttering phenomenon of video transmission in the scene of tourist attractions with fluctuating network signal or limited bandwidth. Although there is a foreground extraction transmission scheme based on background difference in the prior art, which tries to transmit only dynamic targets to reduce bandwidth, such scheme is difficult to effectively distinguish between light mutation and object motion. When the scene light changes, the pixel value of the static background will fluctuate, causing the difference algorithm to mistakenly determine the static background as dynamic foreground, so that the background stripping fails, and the reconstruction quality of the picture cannot be guaranteed while maintaining low bandwidth transmission.

[0004] The existing video stream transmission mode is limited to one-way presentation of visual information, and the transmitted video data lacks semantic association with specific cultural entities in the picture. When watching intangible cultural heritage news live, the audience can only passively receive image signals and cannot interact with the characteristic costumes, musical instruments or intangible cultural heritage props appearing in the picture. The separation of video display and information retrieval makes it difficult for users to instantly obtain cultural interpretation of the objects of interest during the viewing process, limiting the information carrying depth of intangible cultural heritage tourism news and the user's participation experience. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a low-delay image communication collecting and editing display system and method for intangible cultural heritage tourism news, which solves the problem that the existing video transmission technology cannot effectively strip the background light changes and dynamic foreground targets in the complex light environment of intangible cultural heritage performance scene, resulting in large video coding data redundancy and lack of semantic interactive information based on target entities in traditional video stream.

[0006] In order to achieve the above object, the present application is realized by the following technical scheme: the present application provides a low-delay image communication collection and display system for intangible cultural heritage tourism news, which comprises a background map storage module, a regional illumination detection module, an illumination smoothing processing module, a dynamic foreground extraction and differential transmission module, and a terminal synthesis display module.

[0007] The background map storage module is configured to control a camera acquisition device to obtain a static environment image of an intangible cultural heritage performance scene, and determine the initial reference background map after image stability analysis. The background map storage module uses a pre-trained target detection neural network model to perform feature extraction and classification prediction on the initial reference background map, identifies target objects with specific cultural attributes in the picture, and obtains the rectangular bounding box coordinates of the target objects. The background map storage module retrieves associated cultural and tourism metadata according to the identified target object identifier, binds the rectangular bounding box coordinates and the cultural and tourism metadata to generate an interactive explanation file, and sends the initial reference background map and the interactive explanation file to the receiving end for storage.

[0008] The regional illumination detection module is connected with the background map storage module and is configured to divide the real-time collected video picture into logical regions. For all logical detection regions, the regional illumination detection module extracts the pixel brightness set of the current real-time video frame and the pixel brightness set of the initial reference background map, and constructs a linear illumination transformation model. The regional illumination detection module uses the least square method to solve the brightness gain coefficient and the brightness bias value of the logical detection region, so that the linear transformation error between the pixel brightness set of the real-time video frame in the logical detection region and the pixel brightness set of the initial reference background map is minimized, and a sparse illumination parameter matrix composed of the brightness gain coefficient and the brightness bias value of all regions is generated.

[0009] The illumination smoothing processing module is connected with the regional illumination detection module and is configured to perform spatial interpolation operation on the sparse illumination parameter matrix to generate a continuous pixel-level illumination adjustment map covering the whole picture. The continuous pixel-level illumination adjustment map includes a continuous gain map and a continuous bias map. For any target pixel point in the whole picture, the illumination smoothing processing module determines the center point of the adjacent logical detection region surrounding the target pixel point, and uses the bilinear interpolation algorithm to calculate the smooth gain value and the smooth bias value of the target pixel point according to the normalized distance weight of the target pixel point relative to the center point of the adjacent logical detection region.

[0010] The dynamic foreground extraction and differential transmission module is connected with the background base map storage module and the illumination smoothing processing module, and is configured to correct the initial reference base map by using the continuous pixel-level illumination adjustment map to generate a virtual reference base map. When generating the virtual reference base map, the dynamic foreground extraction and differential transmission module obtains the original brightness value of each pixel point in the initial reference base map, multiplies the original brightness value by the smoothing gain value of the corresponding coordinate in the continuous gain map, and adds the smoothing bias value of the corresponding coordinate in the continuous bias map, and takes the calculated result as the virtual brightness value of the virtual reference base map at the pixel point. The dynamic foreground extraction and differential transmission module calculates the absolute value of the brightness difference between the current real-time video frame and the virtual reference base map at the same coordinate, and compares the absolute value with the preset tolerance threshold. If the absolute value is greater than the tolerance threshold, the dynamic foreground extraction and differential transmission module determines that the pixel is a dynamic foreground, otherwise, it is determined as a static background. The dynamic foreground extraction and differential transmission module generates a binary foreground mask according to the determination result, and cuts out the image region containing the dynamic foreground pixels from the current real-time video frame as a dynamic foreground image fragment. The dynamic foreground extraction and differential transmission module constructs the sparse illumination parameter matrix, the dynamic foreground image fragment and its position information into a transmission data packet and sends it to the receiving end, and the transmission data packet does not contain image data of the static background.

[0011] The terminal synthesis display module is located at the receiving end and is configured to parse the transmission data packet and perform shading rendering on the locally stored initial reference base map according to the illumination parameters, and superimposes the dynamic foreground image fragment to restore the display picture. In the restoration process, the terminal synthesis display module reconstructs the continuous pixel-level illumination adjustment map locally at the receiving end. For all pixel coordinates of the target display image, the terminal synthesis display module determines whether the pixel coordinate is within the position range of the dynamic foreground image fragment. If yes, the terminal synthesis display module directly extracts the pixel value of the foreground fragment at the pixel coordinate position as the display value; if not, the terminal synthesis display module performs linear modulation calculation on the locally stored initial reference base map by using the reconstructed continuous pixel-level illumination adjustment map, and takes the calculation result as the display value. At the same time, the terminal synthesis display module monitors the screen touch event and obtains the touch point coordinates, traverses the target object bounding box parameters recorded in the interaction instruction file, and determines whether the touch point coordinates fall within the range of any target object. If it is determined to be hit, the terminal synthesis display module calls and displays the cultural and tourism metadata associated with the target object.

[0012] The second aspect of the present application provides a low-delay image communication production and display method for intangible cultural heritage tourism news, comprising the following steps:

[0013] The control camera acquisition device obtains a static environment image of a performance scene as an initial reference base map, a target detection algorithm is used to identify target objects in the base map and generate an interactive description file containing position coordinates and associated information, and the initial reference base map and the interactive description file are sent to a receiving end.

[0014] Real-time video stream of the performance scene is collected and logically divided. For all logical detection regions, linear brightness transformation relationship between the current video frame and the initial reference base map is calculated, brightness gain coefficient and brightness bias value are solved by minimizing linear transformation error, and a sparse illumination parameter matrix is generated.

[0015] The sparse illumination parameter matrix is subjected to spatial interpolation operation, and discrete regional illumination parameters are mapped to a continuous pixel-level illumination adjustment map covering the whole screen. The continuous pixel-level illumination adjustment map includes a continuous gain map recording illumination scaling ratio of each point of the whole screen and a continuous bias map recording illumination offset of each point of the whole screen.

[0016] The initial reference base map is linearly modulated by using the continuous pixel-level illumination adjustment map, and a virtual reference base map consistent with the current real-time illumination distribution is generated. Pixel difference between the current real-time video frame and the virtual reference base map is calculated, and dynamic foreground image fragments are extracted according to the difference threshold. The sparse illumination parameter matrix, the dynamic foreground image fragments and their position information are constructed into a transmission data packet and sent to the receiving end.

[0017] The receiving end analyzes the transmission data packet, reconstructs the continuous pixel-level illumination adjustment map locally, and renders the locally stored initial reference base map to a virtual background according to the continuous pixel-level illumination adjustment map. The decoded dynamic foreground image fragments are superimposed on the corresponding coordinate position of the virtual background, and the final display screen is synthesized. At the same time, screen touch events are monitored, user operations are defined according to the coordinates in the interactive description file, and corresponding non-heritage tourism information is displayed.

[0018] The application provides a low-delay image communication acquisition and display system and method for non-heritage tourism news. The application has the following beneficial effects:

[0019] 1. The application realizes extremely low bandwidth transmission of video data by constructing a transmission mechanism containing a sparse illumination parameter matrix and dynamic foreground image fragments. The system only needs to transmit a static reference base map once in the initial stage, and only transmit illumination change parameters and cropped dynamic foreground regions with extremely small data volume in the subsequent process. This differential transmission strategy effectively removes a large amount of space-time redundant data in the video sequence, significantly reduces network load, and ensures that real-time picture transmission with low delay and high fluency can be realized under the condition that the network environment of the non-heritage tourism site is limited.

[0020] 2、The present application uses regional light detection and spatial smoothing interpolation algorithm to solve the interference caused by complex light changes in the scene to background difference. Through the calculation of brightness gain and bias and pixel-level reconstruction, the receiving end can use the local map to synthesize the virtual background that adapts to the current light changes in the scene. This not only ensures the visual continuity of the synthesized picture, avoids the background mis-detection caused by sudden light changes, but also ensures the accuracy of dynamic foreground extraction, so that the final display picture restores the scene atmosphere while maintaining the clear and complete image details.

[0021] 3、The present application combines target detection algorithm and interactive description file, and gives instant messaging video stream semantic interaction ability. By pre-identifying cultural entities in the static background and binding metadata at the acquisition end, the receiving end can respond to the user's touch operation according to the preset coordinate information. This changes the traditional passive viewing mode of news live broadcast, so that users can click on specific costumes, musical instruments or props in the picture to get detailed non-heritage culture introduction, and improves the information carrying capacity and transmission depth of tourism news. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The system framework diagram of the present application;

[0023] Figure 2 The background map storage module processing flowchart of the present application;

[0024] Figure 3 The regional light detection and light smoothing processing flowchart of the present application;

[0025] Figure 4 The dynamic foreground extraction and difference transmission processing flowchart of the present application;

[0026] Figure 5 The terminal synthesis display and interaction processing flowchart of the present application;

[0027] Figure 6 The method flowchart of the present application.

[0028] Among them, 10, background map storage module; 20, regional light detection module; 30, light smoothing processing module; 40, dynamic foreground extraction and difference transmission module; 50, terminal synthesis display module. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0030] Referring to the drawings Figure 1 , Figure 1 It is a non-heritage tourism news low-delay image communication production and display system structure block diagram according to an embodiment of the present application. The present application provides a non-heritage tourism news low-delay image communication production and display system, comprising:

[0031] The background map storage module 10 is configured to pre-shoot the non-heritage performance scene as a high-definition background map and send it to the receiving end, divide the background map into several detection areas, identify entities with tourism value such as ancient buildings, plaques or non-heritage appliances in the background map, record their coordinates and generate an explanation file containing cultural tour information and send it together.

[0032] The area light detection module 20 is configured to analyze the video captured by the camera in real time, calculate the brightness gain and bias value of each detection area relative to the initial background map, convert the complex stage light changes on site into a group of sparse parameter matrices with extremely small data volume, and capture the on-site environment atmosphere with low data volume.

[0033] The light smoothing processing module 30 is configured to use a bilinear interpolation algorithm to perform spatial transition processing on the sparse brightness parameters to generate a continuous pixel-level light adjustment map covering the entire screen, simulate the natural gradient effect of light in the real physical world, and eliminate the edge discontinuity phenomenon caused by partition detection.

[0034] The dynamic foreground extraction and differential transmission module 40 is configured to use the light adjustment map to correct the original background map to generate a virtual background map consistent with the on-site light and shadow, and perform pixel-level differential comparison between the real-time video and the virtual background map, use the background pixel value cancellation principle to eliminate the interference of stage light changes, accurately lock and separate the dynamic images of non-heritage performers and props, prevent the system from mistakenly judging that only the background wall has become brighter as new content, and thus realize the packaging and sending of only dynamic foreground fragments, position coordinates and light parameters, and abandon the repeated transmission of static backgrounds.

[0035] The terminal synthesis display module 50 is configured to recolor the local background map according to the light parameters and superimpose the dynamic foreground to restore the picture at the receiving end, and use the pre-transmitted explanation file to build a transparent interactive layer aligned with the background map, respond to user touch operations on the background ancient buildings or appliances while playing the performance, and call up associated travel knowledge to realize multimedia interactive display integrating dynamic performance and static tour.

[0036] The specific working principle and logical relationship of each functional module will be described in detail below.

[0037] Referring to the drawings Figure 2 , Figure 2is a schematic diagram of the processing flow of the background image storage module according to an embodiment of the present application. The background image storage module 10 is the initialization link of the entire non-heritage tourism news low-delay image communication production and display system, and its core task is to establish the reference data for subsequent difference calculation and the index data for terminal interaction.

[0038] The background image storage module 10 first controls the camera acquisition device in a fixed position to scan the non-heritage performance scene. Before the performance officially starts and the performers have not yet entered the picture area, the acquisition device obtains multiple frames of continuous live environment images. The image processor inside the background image storage module 10 performs stability analysis on this set of continuous images using inter-frame difference or Gaussian mixture model, eliminates possible transient noise or slight disturbances, and synthesizes or selects an image with the highest clarity and no dynamic foreground interference as the initial reference background image (denoted as ). The background image storage module 10 converts the data format of the initial reference background image to a color space suitable for subsequent processing, such as the YUV color space, to separate the luminance signal and the chrominance signal.

[0039] In order to support the operation of the subsequent regional light detection module 20, the background image storage module 10 divides the initial reference background image into several rectangular detection regions in a logical manner. Specifically, assuming that the horizontal resolution of the initial reference background image is and the vertical resolution is , the background image storage module 10 presets the number of horizontal grid as and the number of vertical grid as , then the pixel range of the detection region in the th row and the th column is defined as:

[0040] ;

[0041] In the formula, the , , indicates the floor operation, and are the horizontal and vertical coordinates of the pixel point, is the index of the row (the th row), is the index of the column (the th column). The background image storage module 10 stores the above region division parameters ( ) as system configuration parameters, ensuring that all subsequent modules perform light parameter calculation based on the same spatial reference, rather than only obtaining the global average brightness.

[0042] On this basis, the background map storage module 10 performs key scene semantic analysis and interactive data construction functions. The background map storage module 10 is configured with a pre-trained target detection neural network model. The target detection neural network model includes a feature extraction network (Backbone), a feature pyramid network (Neck), and a detection head (Head). The background map storage module 10 inputs the pixel matrix of the initial reference map as input data into the target detection neural network model, extracts high-dimensional semantic features of the image through multiple convolution operations. The detection head performs classification prediction and bounding box regression prediction based on the extracted feature map, thereby identifying target objects with specific cultural attributes or tourism value (such as ancient building eaves, plaques, and traditional musical instruments) in complex background textures.

[0043] The output data of the neural network model is a set of vectors containing target class probabilities and position coordinates, which is parsed and defined by the background map storage module 10 as an interactive hotspot set

[0044]

[0045] wherein is the total number of identified objects;

[0046] is the unique identifier of the th object;

[0047] is the position coordinate vector of the object on the map, represented as , representing the horizontal and vertical coordinates of the upper left corner of the rectangular bounding box, respectively, and the width and height.

[0048] Subsequently, the background map storage module 10 indexes the identified target object identifier in the locally preconfigured intangible cultural heritage travel database. The intangible cultural heritage travel database pre-stores text introductions, historical origins, and multimedia guide links corresponding to various intangible cultural heritage instruments and architectural components. The background map storage module 10 associates and binds the above position coordinate data with the retrieved metadata to generate an additional description file independent of the image data. The additional description file adopts a structured data format (such as XML or JSON), which records in detail the identification, pixel coordinate range, and corresponding display content of each interactive target.

[0049] Finally, the background map storage module 10 outputs the processed high-definition initial reference map ​​​The generated additional description file is packaged. Through the communication interface, the data packet is sent to the terminal equipment of the receiving end. The receiving end saves the initial reference base map in the local memory as the canvas of picture reconstruction; and analyzes the additional description file, so as to prepare data for subsequent establishment of a transparent touch response area on the screen which is accurately corresponding to the picture position.

[0050] For the specific sensor control logic in the above image acquisition process, color space conversion matrix operation and basic image encoding compression standard, the person skilled in the art can realize the existing mature technical solutions, which belong to the known technology in the art, and will not be described here.

[0051] Referring to the accompanying drawings Figure 3 , Figure 3 is a schematic diagram of a region light detection and light smoothing process according to an embodiment of the present application. The region light detection module 20 and the light smoothing processing module 30 work together to convert the complex light changes of the live environment into low-dimensional mathematical features and perform high-fidelity reconstruction.

[0052] The region light detection module 20 is connected to the camera acquisition device and receives the non-heritage live video stream in real time. The region light detection module 20 extracts the luminance component (Y component) of the video picture frame by frame, and calls the preset grid division parameters in the background base map storage module 10 to logically divide the current real-time video frame into a plurality of rectangular detection regions consistent with the initial reference base map.

[0053] In order to accurately quantify the light changes, the region light detection module 20 constructs a linear light transformation model in each detection region. Specifically, for the detection region in the i-th row and the j-th column, the region light detection module 20 extracts the pixel brightness set Y of the current real-time video frame in the detection region and the pixel brightness set Y of the initial reference base map. The region light detection module 20 solves the luminance gain coefficient a and the luminance bias value b of the detection region by the least square method, so that the linear transformation error of the two is minimized. The objective function is defined as:

[0054] ;

[0055] In the formula, a and b represent the optimal luminance gain coefficient and the luminance bias value of the detection region in the i-th row and the j-th column (i.e. region

[0056] ​​​​​​​​​​ denotes the parameters when the following objective function (i.e. the sum of squares of linear transformation errors) reaches the minimum value denotes the value of ;

[0057] denotes the summation operation on all pixel points within the detection region at the th row and the th column;

[0058] denotes the current local detection region (or grid block), which contains the coordinate set of all pixel points within the local detection region;

[0059] denotes the pixel coordinate of any pixel point within the region ;

[0060] denotes the pixel brightness value (usually the gray value or Y component) of the current real-time video frame at the coordinate ;

[0061] denotes the pixel brightness value of the initial reference base map (i.e. the pre-stored static background map) at the same coordinate ;

[0062] denotes the linear gain variable to be solved in the least squares fitting calculation process;

[0063] denotes the linear bias variable to be solved in the least squares fitting calculation process. Through the above operation, the region light detection module 20 outputs a sparse light parameter matrix composed of the gain coefficients and the bias values of all detection regions. The data amount of the sparse light parameter matrix is only related to the number of grids, realizing a very high compression ratio capture of the ambient light atmosphere.

[0064] The light smoothing processing module 30 generates a pixel-level light adjustment map covering the full screen using the bilinear interpolation algorithm. For any pixel point in the target image, the light smoothing processing module 30 first determines the four adjacent grid center points (denoted as respectively) surrounding the arbitrary pixel point, and calculates the normalized horizontal distance and the normalized vertical distance of the pixel point relative to the top-left control point . The light smoothing processing module 30 calculates the arbitrary pixel point ​Smooth gain value at pixel point :

[0065] ;

[0066] wherein, represents the luminance gain coefficient of the corresponding control point, , The illumination smoothing processing module 30 calculates the smooth offset value at pixel point using the same interpolation logic. .

[0067] After traversing and calculating all the pixel points in the full picture, the illumination smoothing processing module 30 finally generates two floating-point mapping maps consistent with the resolution of the original video: one is a continuous gain map recording the illumination scaling ratio of each point in the full picture, and the other is a continuous offset map recording the illumination offset of each point in the full picture. The two maps are collectively referred to as pixel-level illumination adjustment maps, which can finely present the natural attenuation and gradual change of the ambient environment light in space, and provide accurate data support for subsequent modification of the static base map to a virtual base map consistent with the real-time illumination.

[0068] For the image luminance component extraction (such as RGB to YUV) and basic matrix operation implementation involved in the above process, those skilled in the art can use existing digital image processing libraries or hardware acceleration instruction sets to implement, which belongs to the known technology in the art, and will not be described here.

[0069] Referring to the accompanying Figure 4 , Figure 4 is a dynamic foreground extraction and differential transmission processing flow diagram according to an embodiment of the application. The dynamic foreground extraction and differential transmission module 40 is a core processing unit for realizing low-delay and low-bandwidth communication, configured to eliminate background redundancy by using the calculated virtual environment, thereby accurately stripping out the dynamic content of the non-heritage performance.

[0070] Specifically, the dynamic foreground extraction and differential transmission module 40 performs pixel-level linear modulation on the initial reference base map using the continuous gain map and the continuous offset map to construct a virtual reference base map consistent with the current performance scene illumination distribution. For a pixel point at any coordinate in the image, the dynamic foreground extraction and differential transmission module 40 calculates its virtual luminance value according to the following formula:

[0071] ;

[0072] wherein, the initial reference background map at coordinates , a lighting gain coefficient corresponding to the coordinate point, a lighting bias value corresponding to the coordinate point. Through the operation, the dynamic foreground extraction and differential transmission module 40 calculates the theoretical brightness distribution that the static background should present under the action of the current lighting parameter, so that the pixel value of the virtual reference background map in the background area is highly approximated to the real live video frame.

[0073] On this basis, the dynamic foreground extraction and differential transmission module 40 acquires the current real-time video frame input by the camera acquisition device and performs differential extraction logic based on background blanking. The dynamic foreground extraction and differential transmission module 40 compares the brightness difference between the current real-time video frame and the virtual reference background map pixel by pixel. Since the lighting of the background area has been synchronized and corrected by the above formula, the background pixel values will cancel each other out and tend to zero when subtracted, while the non-heritage performer and prop areas that are less affected by lighting or have independent textures will retain significant differences. The dynamic foreground extraction and differential transmission module 40 sets a preset judgment threshold , and generates a binary foreground mask through the following judgment logic:

[0074] ;

[0075] wherein, represents the foreground mask judgment result at coordinates , represents the pixel brightness value of the current real-time video frame at coordinates , represents the virtual brightness value of the virtual reference background map at the same coordinates, is a tolerance threshold set according to the noise level of the camera sensor, represents an absolute value operation, 1 represents that the pixel is determined as dynamic foreground, 0 represents that the pixel is determined as static background, and the setting of the tolerance threshold aims to shield the background noise interference in the image acquisition process and the residual error of light adjustment calculation. Specifically, the photosensitive element of the camera will generate inherent thermal noise when converting the light signal into an electrical signal, and the aforementioned light adjustment calculation may have a slight fitting deviation at the edge of the light and shadow. If the brightness fluctuation of these non-target objects exceeds the threshold, they will be incorrectly identified as dynamic foreground by the system. Therefore, the value range of the tolerance threshold is generally set to cover the sum of the maximum noise amplitude of the sensor and the fitting residual peak value. For a general eight-bit depth grayscale video image, the specific value of the tolerance threshold is generally set to be between five percent and ten percent of the total pixel brightness level; when the light in the shooting site is relatively dark and the camera photosensitive gain is relatively high, resulting in an increase in noise, the high value in this range is used to suppress background flicker; while in the scene with sufficient light and high signal-to-noise ratio, the low value in this range is used to ensure that the embroidery texture on the costumes of non-performers and the hair and other small details can be retained, so as to balance between maintaining the purity of the background and the integrity of the foreground details.

[0076] The dynamic foreground extraction and differential transmission module 40 extracts the dynamic foreground image fragments from the current real-time video frame based on the generated foreground mask and performs connected component analysis to extract the minimum bounding rectangle set covering all the pixels marked as 1. The dynamic foreground extraction and differential transmission module 40 crops the image data in these rectangular regions from the current real-time video frame as dynamic foreground image fragments, and records the top-left corner coordinates of each fragment in the original picture. Finally, the dynamic foreground extraction and differential transmission module 40 constructs a transmission data packet. The transmission data packet only contains the following contents: the sparse light parameter matrix generated by the region light detection module 20 (containing the and values of the grid regions), the compressed data stream of the dynamic foreground image fragments, and the vertex coordinate information of each foreground fragment. The transmission data packet does not contain any image data of the static background. In this way, the system avoids repeated transmission of the unchanged background walls, floors and building structures, and significantly reduces the transmission bandwidth demand while maintaining high-definition image quality.

[0077] For the foreground fragment encoding (such as using the macroblock encoding technology in the HEVC or H.264 standard) and the serialization packaging of the data packet involved in the above process, those skilled in the art can use the existing multimedia communication protocol stack to realize it, which belongs to the prior art and will not be described here.

[0078] Referring to the accompanying Figure 5 , Figure 5Fig. 1 is a schematic diagram of a terminal synthesis display and interaction processing flow according to an embodiment of the present application. The terminal synthesis display module 50 is configured to reconstruct a high-fidelity live picture at the receiving end based on compressed data packets, and to display non-heritage cultural and tourism information in response to touch operations.

[0079] The terminal synthesis display module 50 receives data packets from the sending end through a communication interface, and the data packets contain the sparse light parameter matrix generated by the region light detection module 20, the dynamic foreground image fragment data stream generated by the dynamic foreground extraction and differential transmission module 40, the position coordinates of the foreground fragments, and the synchronization time stamp. At the same time, the terminal synthesis display module 50 reads the locally pre-stored initial reference base map and the additional description file containing the definition of interactive hotspots.

[0080] The terminal synthesis display module 50 parses the sparse light parameter matrix to recover the brightness gain coefficient and brightness bias value of all grid regions. In order to eliminate the discontinuity caused by gridding, the terminal synthesis display module 50 performs spatial smoothing interpolation operations consistent with the sending end to reconstruct continuous gain maps and continuous bias maps covering the entire picture in the local memory, thereby obtaining accurate light gain values and light bias values corresponding to all pixel points on the screen.

[0081] On this basis, the terminal synthesis display module 50 performs pixel-level synthesis of the final display frame. The terminal synthesis display module 50 decodes the dynamic foreground image fragments on one hand to obtain a foreground pixel set, and performs light modulation on the initial reference base map on the other hand to generate background pixels. In order to clarify the synthesis logic, the terminal synthesis display module 50 performs numerical filling operations based on layer priority for all pixel coordinates in the target display image. Specifically, the terminal synthesis display module 50 determines whether the current pixel coordinate is within the position coordinate range of any dynamic foreground image fragment. If the result is yes, the terminal synthesis display module 50 directly extracts the pixel value of the decoded foreground fragment at the corresponding position as the display value of the current pixel; if the result is no, the terminal synthesis display module 50 performs background linear modulation calculation, i.e., multiplies the original brightness value of the initial reference base map at the current coordinate by the light gain value corresponding to the current coordinate, and then adds the light bias value corresponding to the current coordinate, and takes the result of the calculation as the display value of the current pixel. Through the above logic, the terminal synthesis display module 50 realizes seamless splicing of dynamic foreground content and virtually corrected background, and completes high-fidelity restoration of the picture without transmitting background image data.

[0082] While the visual picture is being reconstructed, the terminal synthetic display module 50 loads the additional description file received in the initialization stage to build the interactive response logic. The terminal synthetic display module 50 parses the interactive hot spot set recorded in the description file, which contains the unique identifier of each intangible cultural heritage target object and the rectangular bounding box parameters defined by the top-left corner coordinate, width and height.

[0083] When the terminal synthetic display module 50 monitors a screen touch event, the horizontal and vertical coordinates of the touch point are immediately obtained. The terminal synthetic display module 50 traverses the interactive hot spot set to perform a geometric containment judgment on the touch point. Specifically, the terminal synthetic display module 50 successively judges whether the horizontal coordinate of the touch point is greater than or equal to the left-top horizontal coordinate of the target object and less than or equal to the sum of the left-top horizontal coordinate and the width, and whether the vertical coordinate of the touch point is greater than or equal to the left-top vertical coordinate of the target object and less than or equal to the sum of the left-top vertical coordinate and the height. If the above conditions are met at the same time, the terminal synthetic display module 50 determines that the interaction hits, and retrieves the intangible cultural heritage metadata in the local database according to the unique identifier of the target object. The terminal synthetic display module 50 then calls the graphic rendering interface to render a floating display layer containing text introduction or multimedia content on the upper layer of the current display frame.

[0084] For the data packet decompression (such as JSON parsing, entropy decoding), the graphic rendering pipeline (such as OpenGL or Vulkan interface call), and the event listening mechanism of the operating system involved in the above process, the existing mobile terminal development framework or graphic processing library can be used to implement them, which are known in the art and will not be described here.

[0085] Referring to the accompanying drawings Figure 6 , Figure 6 is a flow chart of an intangible cultural heritage tourism news low-delay image communication production and display method according to an embodiment of the present application. The present application provides an intangible cultural heritage tourism news low-delay image communication production and display method, which comprises the following steps:

[0086] S10, control the camera acquisition device to obtain the live environment image, and generate a clear initial reference base map after stability analysis. The initial reference base map is divided into a plurality of logical detection regions, and a target detection neural network is used to identify intangible cultural heritage target objects in the base map to generate an interactive description file containing the position coordinates and associated information of the target objects. The initial reference base map and the interactive description file are sent to the receiving end for storage.

[0087] S20, real-time acquisition of live performance video stream and extraction of luminance component. For all logical detection regions, the linear luminance transformation relationship between the current video frame and the initial reference base map is calculated, and the sparse illumination parameter matrix composed of the luminance gain coefficient and the luminance bias value is obtained through error minimization operation.

[0088] S30, perform spatial interpolation operation on the sparse light parameter matrix to map the discrete grid data into continuous gain map and continuous bias map covering the whole screen, so as to quantify the spatial distribution characteristics of the field environment light at the pixel level.

[0089] S40, linearly modulate the initial reference base map by using the continuous gain map and the continuous bias map to generate a virtual reference base map consistent with the current real-time light. Calculate the pixel difference between the current real-time video frame and the virtual reference base map, and extract the dynamic foreground image fragments according to the difference threshold. Construct the sparse light parameter matrix, the dynamic foreground image fragments and their position information into a transmission data packet and send it to the receiving end.

[0090] S50, the receiving end analyzes the transmission data packet, reconstructs the continuous gain map and the continuous bias map locally, and renders the initial reference base map stored locally into a virtual background according to the continuous gain map and the continuous bias map. Superimpose the decoded dynamic foreground image fragments onto the corresponding coordinate position of the virtual background to synthesize the final high-fidelity display picture. At the same time, monitor the screen touch event, respond to the user operation according to the coordinate definition in the interactive instruction file, and display the corresponding non-heritage cultural and tourism information.

Claims

1. A low-latency image communication acquisition, editing, and display system for intangible cultural heritage tourism news, characterized in that: include: The background image storage module is configured to acquire static environmental images of the performance venue as initial reference images and pre-send them to the receiving end for storage, while recognizing target entities in the background image to generate interactive instruction files containing coordinates and navigation information; The regional illumination detection module is connected to the background base map storage module and is configured to perform partition analysis on the real-time video image, calculate the linear transformation relationship of brightness of each region relative to the initial reference base map, and generate a sparse illumination parameter matrix characterizing the changes in light and shadow on site. The illumination smoothing module is connected to the regional illumination detection module and is configured to perform spatial interpolation transition processing on the sparse illumination parameter matrix to map and generate a continuous pixel-level illumination adjustment map covering the entire screen. The dynamic foreground extraction and differential transmission module is connected to the background base map storage module and the illumination smoothing processing module, respectively. It is configured to use the continuous pixel-level illumination adjustment map to modulate the initial reference base map to construct a virtual reference base map, separate dynamic foreground image fragments based on the difference between the real-time video and the virtual reference base map, and construct a transmission data packet containing a sparse illumination parameter matrix and dynamic foreground image fragments to send to the receiving end. The terminal synthesis and display module, located at the receiving end, is configured to re-render the local initial reference base map using the received sparse illumination parameter matrix, and superimpose the dynamic foreground image fragments to synthesize the display screen, while responding to touch interaction according to the interaction specification document.

2. The low-latency image communication acquisition, editing, and display system for intangible cultural heritage tourism news according to claim 1, characterized in that, When generating the interactive description file, the background map storage module is configured to perform the following operations: use a pre-trained target detection neural network model to extract features and predict the classification of the initial baseline map, identify target objects with specific attributes and obtain their bounding box coordinates; retrieve associated cultural and tourism metadata based on the identified target object identifier, and bind the bounding box coordinates with the cultural and tourism metadata to generate the interactive description file.

3. The low-latency image communication acquisition, editing, and display system for intangible cultural heritage tourism news according to claim 1, characterized in that, When generating the sparse illumination parameter matrix, the region illumination detection module is configured to perform the following operations: for all logical detection regions, extract the pixel brightness set of the current real-time video frame and the pixel brightness set of the initial reference base map; use the least squares method to solve the brightness gain coefficient and brightness bias value of the region, so as to minimize the linear transformation error between the two sets of pixel brightness sets in the region.

4. The low-latency image communication acquisition, editing, and display system for intangible cultural heritage tourism news according to claim 1, characterized in that, The continuous pixel-level illumination adjustment map generated by the illumination smoothing processing module includes a continuous gain map and a continuous bias map; the illumination smoothing processing module is configured to: for any target pixel in the entire image, determine the center point of the adjacent logical detection region surrounding the target pixel, and use a bilinear interpolation algorithm to calculate the smoothing gain value and smoothing bias value of the target pixel according to the distance weight.

5. The low-latency image communication acquisition, editing, and display system for intangible cultural heritage tourism news according to claim 4, characterized in that, When generating a virtual reference base map, the dynamic foreground extraction and differential transmission module is configured to perform the following operations: for all pixels in the initial reference base map, obtain their original brightness values, multiply them by the smoothed gain value of the corresponding coordinate in the continuous gain map, and accumulate the smoothed offset value of the corresponding coordinate in the continuous offset map. The calculated result is used as the virtual brightness value of the virtual reference base map at that pixel.

6. The low-latency image communication acquisition, editing, and display system for intangible cultural heritage tourism news according to claim 1, characterized in that, When extracting dynamic foreground image fragments, the dynamic foreground extraction and differential transmission module is configured to perform the following operations: calculate the absolute value of the brightness difference between the current real-time video frame and the virtual reference base map at the same coordinates; compare the absolute value with a preset tolerance threshold; if the absolute value is greater than the tolerance threshold, determine that the pixel is a dynamic foreground, otherwise determine that it is a static background; generate a binarized foreground mask based on the determination result, and crop out the image region containing the dynamic foreground pixel from the current real-time video frame.

7. The low-latency image communication acquisition, editing, and display system for intangible cultural heritage tourism news according to claim 1, characterized in that, The transmitted data packet contains only the sparse illumination parameter matrix, the compressed data stream of dynamic foreground image fragments, and the vertex coordinate information of the foreground fragments, and does not contain any static background image data.

8. The low-latency image communication acquisition, editing, and display system for intangible cultural heritage tourism news according to claim 1, characterized in that, When restoring the display screen, the terminal synthesis and display module is configured to perform the following operations: reconstruct a continuous pixel-level illumination adjustment map locally at the receiving end; determine whether all pixel coordinates of the target display image are within the location range of the dynamic foreground image fragment; if so, extract the pixel value of the foreground fragment at that location as the display value; if not, use the reconstructed continuous pixel-level illumination adjustment map to perform linear adjustment calculations on the locally stored initial reference base map, and use the calculation result as the display value.

9. The low-latency image communication acquisition, editing, and display system for intangible cultural heritage tourism news according to claim 1, characterized in that, When responding to a touch operation, the terminal synthesis and display module is configured to perform the following operations: monitor screen touch events and obtain the coordinates of the touch point; traverse the bounding box parameters of the target object recorded in the interaction specification file and determine whether the coordinates of the touch point fall within the range of any target object; If a match is found, the cultural and tourism metadata associated with the target object is invoked and displayed.

10. A method for low-latency image communication acquisition, editing, and display of intangible cultural heritage tourism news, as described in any one of claims 1-9, characterized in that, Includes the following steps: The system controls the camera acquisition device to obtain the initial reference base map, generates an interactive specification file containing information about the target object, and sends the initial reference base map and the interactive specification file to the receiving end. The video stream is acquired in real time and divided into regions. The linear transformation relationship of brightness of each region relative to the initial reference base map is calculated to generate a sparse illumination parameter matrix. Spatial interpolation is performed on the sparse lighting parameter matrix to generate a continuous pixel-level lighting adjustment map covering the entire image; The initial reference base map is corrected using a continuous pixel-level illumination adjustment map to generate a virtual reference base map. The difference between the real-time video and the virtual reference base map is calculated to extract dynamic foreground image fragments. A transmission data packet containing a sparse illumination parameter matrix and dynamic foreground image fragments is constructed and sent. The receiving end parses the transmitted data packets, uses lighting parameters to render the local base map, and overlays dynamic foreground image fragments to synthesize the display screen. At the same time, it responds to the user's touch operations according to the interaction specification document.

Images