Cross-border e-commerce live interactive analysis and optimization system based on multi-modal big data

Abstract

The application relates to the technical field of cross-border e-commerce live broadcast data processing, and particularly discloses a cross-border e-commerce live broadcast interaction analysis and optimization system based on multi-modal big data, which collects anchor audio stream, barrage text stream, audience behavior log and video frame sequence; cross-cultural sensitive word units and speech prosody fluctuation sections are extracted from the audio stream; emotional mutation points in the barrage text stream, churn rate jump points in the behavior log and picture heat attenuation areas in the video frame sequence are respectively located, and a multi-modal feature disturbance atlas is generated; the disturbance atlas is quantized to obtain a cross-cultural cognitive friction coefficient; when the cross-cultural cognitive friction coefficient exceeds a threshold value, a cultural buffer element is matched and inserted into the live broadcast stream; the application can quantize the cross-cultural cognitive load degree in real time, intervene in the cultural buffer element before the audience loses, and improve the conversion efficiency of cross-border live broadcast.

Description

Technical Field

[0001] This invention relates to the field of cross-border e-commerce live streaming data processing technology, specifically to a cross-border e-commerce live streaming interactive analysis and optimization system based on multimodal big data. Background Technology

[0002] With the rapid development of cross-border e-commerce live streaming platforms, live streaming marketing has become an important channel for brands to go global. Hosts promote products, while viewers come from diverse cultural backgrounds. During actual live streams, differences exist between the culturally loaded characteristics of the host's language (slang, speaking speed, expression, etc.) and the cultural cognitive habits of the viewers. This can easily lead to comprehension barriers or emotional resistance, resulting in decreased engagement with live chat, increased viewer churn, and ultimately impacting purchase conversion rates.

[0003] Currently, interactive analysis in cross-border e-commerce live streaming mainly relies on manual observation of bullet screen feedback or empirical adjustments based on post-event playback recordings. It lacks the ability to collect and correlate multimodal data such as the streamer's voice, bullet screen sentiment, audience behavior, and video heatmap. Existing technologies struggle to dynamically quantify the cross-cultural cognitive load between the script and audience feedback during live streaming, and cannot specifically adjust the content to mitigate cultural conflicts. Summary of the Invention

[0004] The purpose of this invention is to provide a cross-border e-commerce live streaming interaction analysis and optimization system based on multimodal big data, so as to solve the problems mentioned above.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A cross-border e-commerce live streaming interaction analysis and optimization system based on multimodal big data includes:

[0007] The multimodal data synchronous acquisition module is used to acquire the anchor's audio stream, bullet screen text stream, audience behavior logs, and video frame sequences during the live broadcast.

[0008] The cultural load unit identification module is used to convert the broadcaster's audio stream into temporal discourse units and extract cross-cultural sensitive words and speech prosodic fluctuation segments containing preset cultural load features from the temporal discourse units.

[0009] The spatiotemporal mapping alignment module is used to perform temporal mapping between cross-cultural sensitive words and speech prosody fluctuation segments and emotional mutation points in the bullet screen text stream, churn rate jump points in the audience behavior log, and image thermal decay areas in the video frame sequence, respectively, to generate a multimodal feature perturbation map.

[0010] The cross-cultural cognitive friction coefficient generation module is used to quantify the perturbation density and perturbation amplitude of the multimodal feature perturbation map to obtain the cross-cultural cognitive friction coefficient that characterizes the degree of collective cognitive load of the audience.

[0011] The cultural buffer intervention timing decision module is used to locate the temporal position of the corresponding disturbance source in the multimodal feature disturbance map when the cross-cultural cognitive friction coefficient exceeds a preset threshold, and to match cultural buffer elements with temporal misalignment compensation characteristics from the cultural adaptation library based on the temporal position of the disturbance source.

[0012] The live stream dynamic correction module is used to insert cultural buffer elements into the live stream segment corresponding to the timing position of the disturbance source, and generate an optimized live stream output.

[0013] As a further aspect of the present invention: the extraction of cross-cultural sensitive words and phonetic prosodic fluctuation segments containing preset cultural load features from temporal discourse units specifically includes:

[0014] The broadcaster's audio stream is segmented into discourse based on the fusion of semantic and prosodic boundaries, generating temporal discourse units with timestamps.

[0015] Each temporal discourse unit is matched with a pre-stored cultural load lexicon based on pronunciation similarity, and the discourse units that successfully match are marked as cross-cultural sensitive word units.

[0016] Extract the speech signals of cross-cultural sensitive word units and the preset range before and after them, calculate the local extreme point density and energy envelope fluctuation amplitude of the speech fundamental frequency curve, and determine the continuous interval that meets the preset fluctuation threshold as speech prosodic fluctuation.

[0017] As a further aspect of the present invention: the generation of the multimodal feature perturbation map specifically includes:

[0018] Based on the timestamp of each cross-cultural sensitive word unit, a temporal impact window is formed by expanding forward and backward;

[0019] Within the temporal influence window, the time point with the largest change rate of emotional polarity value in the bullet screen text stream is identified as the emotional mutation point, and the time point with the largest decline in the number of viewers in the audience behavior log is identified as the churn rate jump point.

[0020] Within the temporal influence window, the consecutive frame intervals with the largest decrease in image thermal value in the video frame sequence are extracted as the image thermal attenuation region.

[0021] By overlaying cross-cultural sensitive word units, speech prosody fluctuation segments, emotional abrupt change points, churn rate jump points, and image heat decay areas on the time axis, a multimodal feature perturbation map containing multiple types of perturbation markers is generated.

[0022] As a further aspect of the present invention: the process for determining the thermal attenuation region of the image is as follows:

[0023] Within the temporal influence window, the thermal values ​​of the video frame sequence are extracted frame by frame to obtain the frame thermal value sequence.

[0024] Calculate the thermal descent gradient between adjacent frames in the frame thermal value sequence, and retain frames with positive thermal descent gradients as candidate descent frames;

[0025] Merge temporally continuous candidate descent frames into descent segments, and accumulate the thermal descent gradients of each frame within each descent segment to obtain the total thermal descent of each descent segment.

[0026] Compare the total heat loss of all falling segments, and determine the continuous frame interval corresponding to the falling segment with the largest total heat loss as the heat attenuation area of ​​the image.

[0027] As a further aspect of the present invention: the calculation process of the cross-cultural cognitive friction coefficient is as follows:

[0028] The number of perturbation markers per unit time in the multimodal feature perturbation map is counted, and the number of perturbation markers is used as the perturbation density value;

[0029] The amplitude of change in emotional polarity, the amplitude of change in loss rate, and the amplitude of decrease in image heat corresponding to each perturbation mark are extracted respectively, and the normalized sum of the three is used as the perturbation amplitude value of each perturbation mark.

[0030] Calculate the sum of the perturbation amplitude values ​​of all perturbation markers, and multiply the sum by the perturbation density value to obtain the cross-cultural cognitive friction coefficient.

[0031] As a further aspect of the present invention: the matching process of the cultural buffer element is as follows:

[0032] The moment when the cross-cultural cognitive friction coefficient exceeds a preset threshold is taken as the trigger moment, and the disturbance marker that is closest to the trigger moment in the multimodal feature disturbance map is taken as the temporal position of the disturbance source.

[0033] Analyze the cross-cultural sensitive word units corresponding to the temporal location of the disturbance source, and extract the timestamps of the cross-cultural sensitive word units and the start and end times of the temporal discourse units to which they belong;

[0034] Calculate the time difference between the start time and the end time, and use the time difference as the response misalignment duration;

[0035] Retrieve cultural buffer elements from the cultural adaptation library whose playback duration is less than or equal to the response misalignment duration, and use the retrieved cultural buffer elements as cultural buffer elements with timing misalignment compensation characteristics.

[0036] As a further aspect of the present invention: the marking process of the timing position of the disturbance source is as follows:

[0037] The moment when the cross-cultural cognitive friction coefficient exceeds a preset threshold is recorded as the trigger moment, and all perturbation markers located before the trigger moment are extracted from the multimodal feature perturbation map;

[0038] Calculate the time interval between the timestamp and the trigger time for each disturbance marker, and select the disturbance marker with the smallest time interval as the first candidate disturbance marker;

[0039] Obtain the disturbance amplitude value corresponding to the first candidate disturbance marker, determine whether the disturbance amplitude value exceeds the preset amplitude threshold, and if it does, determine the first candidate disturbance marker as the temporal position of the disturbance source.

[0040] As a further aspect of the present invention: the step of inserting the cultural buffer element into the live stream segment corresponding to the temporal position of the disturbance source to generate the optimized live stream output specifically includes:

[0041] Analyze the timestamp corresponding to the timing position of the disturbance source, and locate the starting frame position to be inserted in the live stream based on the timestamp;

[0042] Obtain the playback duration of the cultural buffer element, and retain the continuous frame interval corresponding to the playback duration as the segment to be replaced, starting from the position of the starting frame;

[0043] The cultural buffer element is gradually merged with the segment to be replaced through audio and video tracks, so that the beginning part of the cultural buffer element is connected to the frame before the starting frame.

[0044] The integrated cultural buffer elements are used to replace the segments to be replaced and embedded into the live stream to generate optimized live output.

[0045] The beneficial effects of this invention are:

[0046] (1) This invention quantifies the multimodal perturbation characteristics between the broadcaster's rhetoric and the audience's feedback through the cross-cultural cognitive friction coefficient. It can accurately identify the moment when the audience's cognitive load accumulates due to cultural differences and intervene with cultural buffering elements before the conversion rate actually declines, thus achieving preventive intervention against cross-cultural communication barriers. Compared with existing post-analysis techniques, this invention improves the response speed of live streaming optimization from minutes to seconds, reduces the audience churn rate caused by cultural conflicts, and improves the conversion efficiency of cross-border live streaming.

[0047] (2) This invention uses a time-series misalignment compensation mechanism to precisely insert cultural buffer elements into the live stream segment corresponding to the time sequence position of the disturbance source, avoiding the sense of disjointedness caused to the viewing experience by simply switching screens or inserting advertisements. Through progressive fusion processing, the cultural buffer elements form a smooth transition with the original live stream content, which alleviates the cross-cultural cognitive load while maintaining the immersion and continuity of the live stream, and improves the content acceptance of the target market audience. Attached Figure Description

[0048] The invention will now be further described with reference to the accompanying drawings.

[0049] Figure 1 This is a system block diagram of the present invention;

[0050] Figure 2 This is a flowchart illustrating the generation of the cross-cultural cognitive friction coefficient in this invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Please see Figure 1 As shown, this invention is a cross-border e-commerce live streaming interaction analysis and optimization system based on multimodal big data, comprising:

[0053] The multimodal data synchronous acquisition module is used to acquire the anchor's audio stream, bullet screen text stream, audience behavior logs, and video frame sequences during the live broadcast.

[0054] The cultural load unit identification module is used to convert the broadcaster's audio stream into temporal discourse units and extract cross-cultural sensitive words and speech prosodic fluctuation segments containing preset cultural load features from the temporal discourse units.

[0055] The spatiotemporal mapping alignment module is used to perform temporal mapping between cross-cultural sensitive words and speech prosody fluctuation segments and emotional mutation points in the bullet screen text stream, churn rate jump points in the audience behavior log, and image thermal decay areas in the video frame sequence, respectively, to generate a multimodal feature perturbation map.

[0056] The cross-cultural cognitive friction coefficient generation module is used to quantify the perturbation density and perturbation amplitude of the multimodal feature perturbation map to obtain the cross-cultural cognitive friction coefficient that characterizes the degree of collective cognitive load of the audience.

[0057] The cultural buffer intervention timing decision module is used to locate the temporal position of the corresponding disturbance source in the multimodal feature disturbance map when the cross-cultural cognitive friction coefficient exceeds a preset threshold, and to match cultural buffer elements with temporal misalignment compensation characteristics from the cultural adaptation library based on the temporal position of the disturbance source.

[0058] The live stream dynamic correction module is used to insert cultural buffer elements into the live stream segment corresponding to the timing position of the disturbance source, and generate an optimized live stream output.

[0059] The multimodal data synchronization acquisition module is used to acquire the broadcaster's audio stream, bullet screen text stream, audience behavior logs, and video frame sequences during the live broadcast, specifically including:

[0060] An audio acquisition service is deployed in the live streaming server to capture the broadcaster's voice signal in real time through the microphone array of the broadcaster's terminal device, and encode the voice signal into continuous audio data packets to form the broadcaster's audio stream; at the same time, the barrage server is polled through the live streaming application programming interface to obtain the barrage text sent by the audience in real time at fixed time intervals, and each barrage text and its sending timestamp are packaged to generate a barrage text stream.

[0061] Deploy a log collection agent in the live streaming server to obtain event logs of viewers entering and leaving the live streaming room, liking, following, and exiting the live streaming room by subscribing to the real-time message queue of the live streaming room. Arrange the event logs in chronological order of occurrence to generate viewer behavior logs. At the same time, set up a video frame extraction node in the live streaming encoder to extract video frame images from the original live streaming video stream at a fixed number of frames per second, and mark the corresponding timestamp for each frame image to generate a video frame sequence.

[0062] The collected audio streams from the broadcasters, text streams from the bullet screens, audience behavior logs, and video frame sequences are uniformly stored in a distributed message queue and aligned according to the timestamps of each data stream to form a time-synchronized multimodal data set.

[0063] In the cultural load unit identification module, the broadcaster's audio stream is converted into temporal discourse units, and cross-cultural sensitive words and speech prosodic fluctuation segments containing preset cultural load features are extracted from the temporal discourse units, specifically including:

[0064] First, the broadcaster's audio stream undergoes speech segmentation. After acquiring the broadcaster's audio stream, speech recognition is performed to obtain the corresponding text content, and fundamental frequency trajectory and energy trajectory are extracted from the audio stream. The text obtained from speech recognition is initially segmented according to the semantic boundaries at the end of sentences to obtain the temporal positions of semantic segmentation points. Simultaneously, the moment when the fundamental frequency value drops to the silence threshold in the fundamental frequency trajectory is detected as the prosodic boundary. The temporal positions of the semantic segmentation points are compared with the temporal boundary temporal positions. If the time difference between the two is less than 0.3 seconds, the time position is determined as a speech segmentation point. The broadcaster's audio stream is segmented into continuous audio segments according to all speech segmentation points. Each audio segment corresponds to a temporal speech unit, and each temporal speech unit is marked with a start timestamp and an end timestamp.

[0065] Secondly, cross-cultural sensitive word units are labeled. A culture-loaded lexicon is pre-constructed, containing words that are prone to misunderstanding or resistance in the target market's cultural context. Each word also stores its standard text form and its corresponding International Phonetic Alphabet (IPA) sequence. Each temporal discourse unit is converted into candidate text through speech recognition, and the candidate text is then converted into a phonetic sequence. The phonetic sequence of the candidate text is compared phonetically with the phonetic sequence of each word in the culture-loaded lexicon. The number of phonemes that need to be replaced, inserted, or deleted between the two phonetic sequences is calculated, and the minimum number of phonemes is taken as the pronunciation difference score. If the pronunciation difference score is less than or equal to 2, the temporal discourse unit is considered to have successfully matched the word in the culture-loaded lexicon, and the temporal discourse unit is labeled as a cross-cultural sensitive word unit.

[0066] Finally, the prosodic fluctuation segments are extracted. Audio signals marked as cross-cultural sensitive word units are acquired, and the fundamental frequency curve and energy envelope curve are extracted from these signals. All local maxima and minima are detected on the fundamental frequency curve, and the number of local extrema per unit time is counted as the extremum density. The average energy of all sampling points on the energy envelope curve is calculated, and the absolute value of the difference between the energy value of each sampling point and the average value is calculated. Sampling points whose absolute difference exceeds 30% of the average energy value are marked as energy fluctuation points, and the duration of consecutive energy fluctuation points is counted as the energy fluctuation duration. Continuous audio intervals with an extremum density exceeding 5 per second and an energy fluctuation duration exceeding 0.5 seconds are extracted and identified as prosodic fluctuation segments.

[0067] In the spatiotemporal mapping alignment module, cross-cultural sensitive words and speech prosodic fluctuation segments are temporally mapped to emotional abrupt change points in the bullet screen text stream, churn rate jump points in the audience behavior log, and image thermal decay regions in the video frame sequence, respectively, to generate a multimodal feature perturbation map, specifically including:

[0068] First, construct the temporal impact window. Obtain the start and end timestamps corresponding to each cross-cultural sensitive word unit. Extend five seconds forward from the start timestamp as the window's start boundary, and extend five seconds backward from the end timestamp as the window's end boundary. The time interval covered by the window's start and end boundaries constitutes the temporal impact window corresponding to that cross-cultural sensitive word unit.

[0069] Secondly, locate the sentiment abrupt change point within the temporal influence window. Obtain all bullet comment texts within the temporal influence window from the bullet comment text stream. Input each bullet comment text into a sentiment polarity value calculation program, which outputs a sentiment polarity value between -1 and +1. Arrange the bullet comment texts in chronological order, calculate the absolute value of the difference between the sentiment polarity value of each bullet comment text and the sentiment polarity value of the previous bullet comment text, and divide this absolute value by the time interval between the two bullet comment texts to obtain the sentiment polarity change rate corresponding to that time point. The time point with the largest sentiment polarity change rate within the temporal influence window is identified as the sentiment abrupt change point.

[0070] Next, locate the churn rate spike within the time-series impact window. Obtain the viewer count records within the time-series impact window from the viewer behavior log. These records contain the number of online viewers in the live stream at each time point. Arrange the viewer count records chronologically and calculate the difference between the viewer count at each time point and the viewer count at the previous time point. Divide this difference by the time interval between the two time points to obtain the churn rate change slope for that time point. The time point with the smallest and negative churn rate change slope within the time-series impact window is identified as the churn rate spike point, representing the moment when viewer churn is fastest.

[0071] Subsequently, the thermal decay region is determined within the temporal influence window. For each frame in the video frame sequence located within the temporal influence window, the brightness value of each pixel is extracted. The image is divided into 256 rectangular regions (16 rows, 16 columns). The average brightness value of all pixels within each rectangular region is calculated as the thermal value of that region. The thermal values ​​of all regions are summed and divided by 256 to obtain the thermal value of that frame. The thermal values ​​of all frames are arranged in chronological order to obtain a frame thermal value sequence. For each frame in the frame thermal value sequence, the difference between the thermal value of that frame and the thermal value of the next frame is calculated. If the difference is positive, the frame is marked as a candidate decreasing frame. Temporally consecutive candidate decreasing frames are merged into a decreasing segment, and each decreasing segment contains several consecutive frames. For each decreasing segment, the thermal decay difference of each frame within the decreasing segment is summed to obtain the total thermal decay of the decreasing segment. Compare the total heat loss of all falling segments, and determine the continuous interval from the start frame to the end frame corresponding to the falling segment with the largest total heat loss as the heat attenuation area of ​​the image.

[0072] Finally, a multimodal feature perturbation map is generated. Using the time axis as a reference, cross-cultural sensitive word units are marked as first-type perturbation points, speech prosodic fluctuation segments are marked as first-type perturbation intervals, emotional abrupt change points are marked as second-type perturbation points, churn rate jump points are marked as third-type perturbation points, and areas of image heat decay are marked as fourth-type perturbation intervals. All perturbation points and intervals are plotted on the same time axis according to their respective temporal positions, forming a multimodal feature perturbation map containing multiple types of perturbation markers.

[0073] Please see Figure 2 As shown, in the cross-cultural cognitive friction coefficient generation module, the perturbation density and amplitude of the multimodal feature perturbation map are quantified to obtain the cross-cultural cognitive friction coefficient, which characterizes the degree of collective cognitive load of the audience. Specifically, it includes:

[0074] First, the perturbation density value is calculated. Taking the current moment as the endpoint, a 30-second period is extracted as a statistical window. All perturbation markers appearing within this statistical window are extracted from the multimodal feature perturbation map. These perturbation markers include emotional abrupt change points, churn rate spikes, and areas of image heat decay. Each dot-like marker is counted as one perturbation marker, and each continuous interval marker is also counted as one perturbation marker. The total number of perturbation markers is counted, and this number is divided by 30 seconds to obtain the perturbation density value, denoted as D, with units of per second.

[0075] Secondly, three amplitude values ​​are extracted for each perturbation marker. For each perturbation marker, the corresponding raw amplitude is obtained according to its type: if the marker is an emotional abrupt change point, the absolute value of the emotional polarity change rate corresponding to that point is taken as the emotional polarity change amplitude. If the marker represents a jump in churn rate, then the absolute value of the slope of the churn rate change corresponding to that point is taken as the magnitude of the churn rate change. If the marked area represents a region of thermal decay, then the total thermal decrease in that area is taken as the thermal decrease rate of the image. For amplitude types not included in the disturbance flag, their corresponding amplitude values ​​are set to zero.

[0076] Then, normalization was performed on the three amplitudes respectively. The minimum value of the amplitude of all sentiment polarity changes within the statistical window was calculated for each. and maximum value For each perturbation marker, the magnitude of the change in emotional polarity The normalized change in sentiment polarity is calculated using the following formula. : ;

[0077] Similarly, calculate the minimum value of the change in churn rate across all statistical windows. and maximum value The normalized churn rate variation for each perturbation marker was obtained. ; Calculate the minimum value of the heat drop across all images within the statistics window. and maximum value The normalized thermal decrease of each perturbation marker was obtained. If the maximum value of a certain type of amplitude within the statistical window is equal to the minimum value, then all normalized values ​​for that type are set to zero.

[0078] Next, the disturbance amplitude value for each disturbance marker is calculated. The first The sum of the normalized emotional polarity change, the normalized loss rate change, and the normalized image heat decrease of each perturbation marker is: .

[0079] Then, the sum of the disturbance amplitude values ​​of all disturbance markers is calculated. This will count the disturbance amplitude value of each disturbance marker within the statistical window. By summing, we get .

[0080] Finally, the perturbation density value Sum of the disturbance amplitude values Multiplying them together yields the cross-cultural cognitive friction coefficient. The specific calculation formula is as follows:

[0081] ;

[0082] in To count the total number of disturbance markers within the window. The value represents the perturbation density. For the first The perturbation amplitude value of each perturbation marker. This is the cross-cultural cognitive friction coefficient. It represents the level of collective cognitive load caused by cross-cultural rhetoric among the audience during the current period. The higher the value, the heavier the audience's cognitive load, and the more necessary it is to intervene with cultural buffering elements.

[0083] In the cultural buffer intervention timing decision module, when the cross-cultural cognitive friction coefficient exceeds a preset threshold, the module locates the temporal position of the corresponding disturbance source in the multimodal feature disturbance map, and matches cultural buffer elements with temporal misalignment compensation characteristics from the cultural adaptation library based on the temporal position of the disturbance source. Specifically, this includes:

[0084] First, the trigger moment and candidate perturbation marker set are determined. The cross-cultural cognitive friction coefficient is calculated in real time and compared with a preset threshold. The preset threshold is determined based on the average friction coefficient corresponding to moments of significant increase in audience churn rate in historical live streaming data, and its value is between 0.6 and 0.8. When the cross-cultural cognitive friction coefficient exceeds the preset threshold, the corresponding time point is recorded as the trigger moment. All perturbation markers whose time positions are before the trigger moment are extracted from the multimodal feature perturbation map, forming a candidate perturbation marker set. Each perturbation marker includes its type attribute, timestamp information, and corresponding perturbation amplitude value.

[0085] Secondly, the most recent perturbation markers are selected. For each perturbation marker in the candidate perturbation marker set, the time interval between the timestamp of the perturbation marker and the trigger time is calculated. The time interval is calculated by subtracting the timestamp of the perturbation marker from the trigger time, in seconds. All perturbation markers are sorted in ascending order of the calculated time intervals, and the perturbation marker with the smallest time interval is selected as the first candidate perturbation marker. If multiple perturbation markers have the same minimum time interval as the trigger time, these perturbation markers are all included in the first candidate perturbation marker set.

[0086] Next, the disturbance amplitude threshold is verified and the temporal location of the disturbance source is determined. The disturbance amplitude value corresponding to the first candidate disturbance marker is obtained. This disturbance amplitude value is obtained by normalizing and summing the amplitude of emotional polarity change, the amplitude of churn rate change, and the amplitude of image heat decrease, with a value ranging from 0 to 3. The disturbance amplitude value is compared with a preset amplitude threshold, which is determined based on the statistical distribution of disturbance amplitudes in normal live streaming interactive scenarios and is set to 1.5. If the disturbance amplitude value of the first candidate disturbance marker exceeds 1.5, then the first candidate disturbance marker is determined as the temporal location of the disturbance source; if the disturbance amplitude value of the first candidate disturbance marker does not exceed 1.5, then the first candidate disturbance marker is excluded, and the disturbance marker with the smallest time interval is selected from the remaining candidate disturbance markers as the new first candidate disturbance marker. This disturbance amplitude value judgment process is repeated until a disturbance marker with a disturbance amplitude value exceeding the preset amplitude threshold is found as the temporal location of the disturbance source.

[0087] Next, the discourse information corresponding to the temporal position of the perturbation source is analyzed. Based on the timestamp of the perturbation source's temporal position, the cross-cultural sensitive word unit corresponding to that timestamp is searched back in the multimodal feature perturbation map. The temporal discourse unit to which this cross-cultural sensitive word unit belongs is obtained, and its start and end timestamps are extracted from the metadata corresponding to this temporal discourse unit. The start timestamp is the time position of the first speech frame of the discourse unit, and the end timestamp is the time position of the last speech frame of the discourse unit.

[0088] Subsequently, the response misalignment duration is calculated. The duration of the discourse unit is obtained by subtracting the start timestamp from the end timestamp of the temporal discourse unit, and this duration is used as the response misalignment duration. This response misalignment duration characterizes the time delay between the appearance of cross-cultural sensitive words and the collective cognitive load response of the audience, in units of 0.1 seconds.

[0089] Finally, cultural buffer elements with temporal misalignment compensation characteristics are matched. A pre-built cultural adaptation library stores various cultural buffer elements, each containing a type label, content data, and playback duration attribute. All cultural buffer elements with a playback duration less than or equal to the response misalignment duration are retrieved from the cultural adaptation library. If multiple matching cultural buffer elements exist, the cultural buffer element with the highest matching degree is selected from the cultural adaptation library based on the cross-cultural sensitive word type corresponding to the temporal position of the disturbance source. The matching degree is determined by the correlation between the applicable scenario label of the cultural buffer element and the semantic category of the cross-cultural sensitive word. The finally selected cultural buffer element is used as the output result with temporal misalignment compensation characteristics.

[0090] In the live stream dynamic correction module, cultural buffer elements are inserted into the live stream segment corresponding to the temporal position of the disturbance source to generate optimized live stream output. Specifically, this includes:

[0091] First, the timestamp corresponding to the temporal position of the disturbance source is parsed, and the position of the starting frame to be inserted is located in the live stream based on this timestamp. The time point marked by the temporal position of the disturbance source is extracted from the multimodal feature disturbance map, and this time point is accurate to milliseconds. The frame rate information of the live video stream is obtained. Assuming the frame rate is 25 frames per second, the time interval between each frame is 40 milliseconds. The time point of the temporal position of the disturbance source is divided by the time interval between each frame to obtain the corresponding frame number. If the calculation result is not an integer, it is rounded down. The video frame pointed to by this frame number is determined as the starting frame to be inserted. At the same time, the encoding order position of this starting frame in the live stream is recorded as the reference point for subsequent replacement operations.

[0092] Secondly, determine the range of the segment to be replaced. Obtain the playback duration of the cultural buffer element, which is stored in the cultural adaptation library in milliseconds. Based on the frame rate of the live video stream, convert the playback duration of the cultural buffer element into the corresponding number of frames, i.e., divide the playback duration by the time interval between each frame and round up to obtain the number of consecutive frames that need to be replaced. Starting from the position of the starting frame, select consecutive video frames with the same number of frames as the starting frame, and mark the interval formed by these frames from the starting frame to the ending frame as the segment to be replaced. At the same time, record the audio track segment corresponding to the segment to be replaced to ensure audio and video synchronization.

[0093] Next, a progressive fusion process is performed on the audio and video tracks of the cultural buffer element and the segment to be replaced. The video content of the cultural buffer element is resampled according to the frame rate of the segment to be replaced, making its frame rate consistent with the live stream. The fusion transition length is set to 5 frames. A transparency gradient parameter is established between the frame before the starting frame and the starting frame, so that the transparency of the first frame of the cultural buffer element is 100%, while the image of the previous frame gradually fades out. In the specific implementation, for each frame in the transition area, the fusion weight is calculated. The first frame at the starting frame position fully displays the content of the cultural buffer element, the weight of the cultural buffer element content in the second frame is 80%, the weight of the content of the previous frame is 20%, and so on for subsequent frames until the transition ends and the cultural buffer element occupies the entire area. The audio track also uses linear cross-attenuation, superimposing the audio of the cultural buffer element and the original audio in the transition area. The volume of the audio of the cultural buffer element gradually increases from zero to normal volume, while the volume of the original audio gradually decreases from normal to 0.

[0094] Finally, the integrated cultural buffer elements are embedded into the live stream to replace the segments to be replaced. The progressively integrated video frame sequence and corresponding audio frame sequence of the cultural buffer elements are written chronologically to the position of the segment to be replaced in the live stream, overwriting the original video and audio frame data. The encoder regenerates the video stream based on the updated frame sequence, ensuring continuous timestamps without jumps, generating the optimized live stream output. This optimized live stream is pushed to the distribution server, and viewers will receive a smooth transition with cultural buffer intervention, thus alleviating the cognitive load caused by cross-cultural rhetoric.

[0095] The working principle of this invention is as follows: First, the audio stream of the broadcaster, the text stream of the bullet comments, the audience behavior logs, and the video frame sequence during the live broadcast are collected. Then, the broadcaster's audio stream is segmented into temporal discourse units, from which cross-cultural sensitive word units containing preset cultural load features and speech prosodic fluctuation segments are extracted. Next, a temporal influence window is constructed based on the timestamp of the cross-cultural sensitive word units. Within this window, emotional abrupt change points in the bullet comment text stream, churn rate jump points in the audience behavior logs, and image heat decay areas in the video frame sequence are located. The above perturbation markers are overlaid with the cross-cultural sensitive word units and speech prosodic fluctuation segments on the time axis to generate a multimodal feature perturbation map. Then, the perturbation density and perturbation amplitude of the perturbation map are quantified to obtain the cross-cultural cognitive friction coefficient. When the coefficient exceeds a preset threshold, the temporal position of the corresponding perturbation source in the perturbation map is located, and a cultural buffer element with temporal misalignment compensation characteristics is matched from the cultural adaptation library based on the position. Finally, the cultural buffer element is inserted into the live stream segment corresponding to the temporal position of the perturbation source to generate the optimized live output.

[0096] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A cross-border e-commerce live streaming interaction analysis and optimization system based on multimodal big data, characterized in that: include: The multimodal data synchronous acquisition module is used to acquire the anchor's audio stream, bullet screen text stream, audience behavior logs, and video frame sequences during the live broadcast. The cultural load unit identification module is used to convert the broadcaster's audio stream into temporal discourse units and extract cross-cultural sensitive words and phonetic prosodic fluctuation segments from the temporal discourse units. Specifically, it includes: performing discourse segmentation on the broadcaster's audio stream based on the fusion of semantic boundaries and prosodic boundaries to generate temporal discourse units with timestamps. Each temporal discourse unit is matched with a pre-stored cultural load lexicon based on pronunciation similarity, and the discourse units that successfully match are marked as cross-cultural sensitive word units. Extract the speech signals of cross-cultural sensitive word units and the preset range before and after them, calculate the local extreme point density and energy envelope fluctuation amplitude of the speech fundamental frequency curve, and determine the continuous interval that meets the preset fluctuation threshold as the speech prosodic fluctuation segment. The spatiotemporal mapping alignment module is used to perform temporal mapping between cross-cultural sensitive words and speech prosody fluctuation segments and emotional mutation points in the bullet screen text stream, churn rate jump points in the audience behavior log, and image thermal decay areas in the video frame sequence, respectively, to generate a multimodal feature perturbation map. The cross-cultural cognitive friction coefficient generation module is used to quantify the perturbation density and perturbation amplitude of the multimodal feature perturbation map to obtain the cross-cultural cognitive friction coefficient that characterizes the degree of collective cognitive load of the audience. The cultural buffer intervention timing decision module is used to locate the temporal position of the corresponding disturbance source in the multimodal feature disturbance map when the cross-cultural cognitive friction coefficient exceeds a preset threshold, and to match cultural buffer elements with temporal misalignment compensation characteristics from the cultural adaptation library based on the temporal position of the disturbance source. The live stream dynamic correction module is used to insert cultural buffer elements into the live stream segment corresponding to the timing position of the disturbance source, and generate an optimized live stream output.

2. The cross-border e-commerce live streaming interaction analysis and optimization system based on multimodal big data as described in claim 1, characterized in that, The generation of the multimodal feature perturbation map specifically includes: Based on the timestamp of each cross-cultural sensitive word unit, a temporal impact window is formed by expanding forward and backward; Within the temporal influence window, the time point with the largest change rate of emotional polarity value in the bullet screen text stream is identified as the emotional mutation point, and the time point with the largest decline in the number of viewers in the audience behavior log is identified as the churn rate jump point. Within the temporal influence window, the consecutive frame intervals with the largest decrease in image thermal value in the video frame sequence are extracted as the image thermal attenuation region. By overlaying cross-cultural sensitive word units, speech prosody fluctuation segments, emotional abrupt change points, churn rate jump points, and image heat decay areas on the time axis, a multimodal feature perturbation map containing multiple types of perturbation markers is generated.

3. The cross-border e-commerce live streaming interaction analysis and optimization system based on multimodal big data according to claim 2, characterized in that, The process for determining the thermal attenuation area of ​​the image is as follows: Within the temporal influence window, the thermal values ​​of the video frame sequence are extracted frame by frame to obtain the frame thermal value sequence. Calculate the thermal descent gradient between adjacent frames in the frame thermal value sequence, and retain frames with positive thermal descent gradients as candidate descent frames; Merge temporally continuous candidate descent frames into descent segments, and accumulate the thermal descent gradients of each frame within each descent segment to obtain the total thermal descent of each descent segment. Compare the total heat loss of all falling segments, and determine the continuous frame interval corresponding to the falling segment with the largest total heat loss as the heat attenuation area of ​​the image.

4. The cross-border e-commerce live streaming interaction analysis and optimization system based on multimodal big data as described in claim 1, characterized in that, The calculation process for the cross-cultural cognitive friction coefficient is as follows: The number of perturbation markers per unit time in the multimodal feature perturbation map is counted, and the number of perturbation markers is used as the perturbation density value; The amplitude of change in emotional polarity, the amplitude of change in loss rate, and the amplitude of decrease in image heat corresponding to each perturbation mark are extracted respectively, and the normalized sum of the three is used as the perturbation amplitude value of each perturbation mark. Calculate the sum of the perturbation amplitude values ​​of all perturbation markers, and multiply the sum by the perturbation density value to obtain the cross-cultural cognitive friction coefficient.

5. The cross-border e-commerce live streaming interaction analysis and optimization system based on multimodal big data according to claim 1, characterized in that, The matching process for the cultural buffer elements is as follows: The moment when the cross-cultural cognitive friction coefficient exceeds a preset threshold is taken as the trigger moment, and the disturbance marker that is closest to the trigger moment in the multimodal feature disturbance map is taken as the temporal position of the disturbance source. Analyze the cross-cultural sensitive word units corresponding to the temporal location of the disturbance source, and extract the timestamps of the cross-cultural sensitive word units and the start and end times of the temporal discourse units to which they belong; Calculate the time difference between the start time and the end time, and use the time difference as the response misalignment duration; Retrieve cultural buffer elements from the cultural adaptation library whose playback duration is less than or equal to the response misalignment duration, and use the retrieved cultural buffer elements as cultural buffer elements with timing misalignment compensation characteristics.

6. The cross-border e-commerce live streaming interaction analysis and optimization system based on multimodal big data according to claim 5, characterized in that, The process of marking the temporal location of the disturbance source is as follows: The moment when the cross-cultural cognitive friction coefficient exceeds a preset threshold is recorded as the trigger moment, and all perturbation markers located before the trigger moment are extracted from the multimodal feature perturbation map; Calculate the time interval between the timestamp and the trigger time for each disturbance marker, and select the disturbance marker with the smallest time interval as the first candidate disturbance marker; Obtain the disturbance amplitude value corresponding to the first candidate disturbance marker, determine whether the disturbance amplitude value exceeds the preset amplitude threshold, and if it does, determine the first candidate disturbance marker as the temporal position of the disturbance source.

7. The cross-border e-commerce live streaming interaction analysis and optimization system based on multimodal big data according to claim 1, characterized in that, The step of inserting cultural buffer elements into the live stream segment corresponding to the temporal position of the disturbance source to generate optimized live stream output specifically includes: Analyze the timestamp corresponding to the timing position of the disturbance source, and locate the starting frame position to be inserted in the live stream based on the timestamp; Obtain the playback duration of the cultural buffer element, and retain the continuous frame interval corresponding to the playback duration as the segment to be replaced, starting from the position of the starting frame; The cultural buffer element is gradually merged with the segment to be replaced through audio and video tracks, so that the beginning part of the cultural buffer element is connected to the frame before the starting frame. The integrated cultural buffer elements are used to replace the segments to be replaced and embedded into the live stream to generate optimized live output.

Images