An adaptive bit rate transmission method, system, device and storage medium

Abstract

An adaptive bitrate transmission method, system, device, and storage medium relate to the field of streaming media transmission optimization. The method involves acquiring multimodal data of the current live stream and determining the product switching time based on the multimodal data; before the product switching time, acquiring first audio data of the current live stream and identifying the product type of the new product in the first audio data; after detecting the product switching, extracting key features of the new product and determining the key display area of ​​the new product based on the key features; obtaining a hierarchical bitrate optimization strategy based on the product type and key display area of ​​the new product; determining a bitrate transmission strategy based on the hierarchical bitrate optimization strategy and the real-time network transmission rate of the current live stream; and transmitting the live video corresponding to the new product according to the bitrate transmission strategy. This allows the bitrate to adapt to the display requirements of the new product in a timely manner.

Description

Technical Field

[0001] This application relates to the field of streaming media transmission optimization, specifically to an adaptive bitrate transmission method, system, device, and storage medium. Background Technology

[0002] With the booming development of live-streaming e-commerce, more and more merchants are choosing to showcase and sell their products through live streaming. Compared to traditional e-commerce, live-streaming e-commerce offers stronger interactivity and immediacy, allowing consumers to understand product details more intuitively and improving the accuracy of their purchasing decisions. High-quality live-streaming footage has a significant impact on product display effectiveness and user purchase intentions; this requires a sufficiently high video bitrate, which is the amount of video data transmitted per second.

[0003] Currently, mainstream live streaming systems employ adaptive bitrate adjustment schemes based on network conditions. These systems monitor real-time network transmission rates and reduce the bitrate to ensure smooth transmission when the real-time network transmission rate is limited, while increasing the bitrate to improve image quality when the real-time network transmission rate is not limited.

[0004] However, in actual live streaming, the frequent switching of products by the host may cause the bitrate adjustment to lag behind the speed of product switching. This results in the image quality not being able to adapt to the new content for a period of time after the switch, and the bitrate not being able to adapt to the display requirements of the new products in a timely manner. Summary of the Invention

[0005] This application provides an adaptive bitrate transmission method, system, device, and storage medium, which enables the bitrate to adapt to the display needs of new products in a timely manner.

[0006] The first aspect of this application provides an adaptive bit rate transmission method, specifically including:

[0007] Acquire the multimodal data of the current live stream, and determine the product switching time based on the multimodal data;

[0008] Before the product switching time, the first audio data of the current live broadcast is obtained, and the product type of the new product is identified in the first audio data;

[0009] After detecting a product switch, extract the key features of the new product and determine the key display area of ​​the new product based on the key features;

[0010] Based on the product type and key display area of ​​the new product, a hierarchical bitrate optimization strategy is obtained. Based on the hierarchical bitrate optimization strategy and the real-time network transmission rate of the current live broadcast, a bitrate transmission strategy is determined.

[0011] The live video corresponding to the new product is transmitted according to the bitrate transmission strategy.

[0012] The above technical solution achieves several key advantages. First, by acquiring multimodal data from the live stream and determining the product switching time based on this data, accurate prediction of product switching can be realized. Before the product switching time, live stream audio data is acquired and new product types are identified. After detecting the product switching, key features of the new product are extracted and key display areas are determined, enabling precise positioning of the new product's display focus. Based on this, a layered bitrate optimization strategy is developed according to the product type and key display areas. By combining this strategy with real-time network transmission rates to determine the bitrate transmission strategy, adaptive transmission of the new product's display content is achieved. This approach effectively predicts product switching timing, quickly perceives the characteristics of new product content, and thus promptly adapts to the display needs of the new product.

[0013] Optionally, the key features include color distribution features and material texture features, and determining the key display area of ​​the new product based on the key features specifically includes:

[0014] Multi-level hierarchical clustering is performed on the color distribution features to obtain multiple cluster centers. The color value and pixel ratio of each cluster center are extracted. A color distribution histogram is generated using the color value as the horizontal axis and the pixel ratio as the vertical axis. The center color value of the cluster center with the highest pixel ratio is selected, and a preset range centered on the center color value is used as the main color range of the new product.

[0015] Construct a gray-level co-occurrence matrix of the material texture features, construct a texture feature vector based on the contrast, uniformity and correlation of the gray-level co-occurrence matrix, and calculate the modulus of the texture feature vector as the overall image complexity of the new product.

[0016] Based on the main color range and overall image complexity, the key display areas for the new product are determined.

[0017] By employing the aforementioned technical solution, firstly, color features are analyzed through multi-level hierarchical clustering. This not only extracts the dominant color tone information but also reflects the distribution weight of each color region through a color distribution histogram, providing a color-dimensional basis for determining key display areas. Secondly, by constructing a gray-level co-occurrence matrix and extracting features such as contrast, uniformity, and correlation, complex texture information is quantified into an overall image complexity index, providing a texture-dimensional basis for determining key display areas. Finally, by combining the features of both the dominant color tone range and the overall image complexity, key display areas of the product can be more comprehensively identified and located, thus providing a more accurate basis for subsequent bitrate allocation.

[0018] Optionally, the step of constructing the gray-level co-occurrence matrix of the material texture features, and constructing a texture feature vector based on the contrast, uniformity, and correlation of the gray-level co-occurrence matrix, specifically includes:

[0019] Convert the image of the new product to a grayscale image;

[0020] According to the preset pixel spacing and direction, multiple adjacent pixel combinations in the grayscale image are determined, the occurrence frequency of grayscale value combinations corresponding to each adjacent pixel combination is counted, and a grayscale co-occurrence matrix is ​​generated. The rows and columns of the grayscale co-occurrence matrix are all grayscale values, and the elements in the grayscale co-occurrence matrix are the occurrence frequency of the corresponding grayscale value combinations in the rows and columns.

[0021] Divide each element value of the gray-level co-occurrence matrix by the sum of all element values ​​to obtain the normalized matrix;

[0022] Multiply each element value in the normalized matrix by the square of the difference between the corresponding row and column gray values, and then sum all the products as the contrast of the gray co-occurrence matrix.

[0023] The uniformity of the gray-level co-occurrence matrix is ​​obtained by calculating the sum of squares of all elements in the normalized matrix.

[0024] Calculate the row standard deviation and column standard deviation in the normalized matrix, as well as the covariance of the normalized matrix. Divide the covariance by the product of the row standard deviation and the column standard deviation to obtain the correlation of the gray-level co-occurrence matrix.

[0025] The contrast is used as the x-component, the uniformity as the y-component, and the correlation as the z-component to construct a three-dimensional vector, thus obtaining the texture feature vector.

[0026] By adopting the above technical solution, firstly, converting the image to grayscale simplifies computational complexity, and constructing a grayscale co-occurrence matrix using preset pixel spacing and orientation effectively captures local texture information of the image. Secondly, matrix normalization eliminates the influence of pixel count on statistical results, making feature extraction more universal. Then, texture features are quantified from three dimensions: contrast, uniformity, and correlation. Contrast reflects the image's sharpness and the degree of local variation; uniformity reflects the regularity and consistency of the texture; and correlation characterizes the directional features of the texture. Finally, by combining these three features into a three-dimensional vector, a comprehensive representation of texture features is achieved, providing a reliable mathematical foundation for subsequent calculations of overall image complexity.

[0027] Optionally, determining the key display area of ​​the new product based on the main color range and overall image complexity specifically includes:

[0028] The live video screen corresponding to the new product is divided into a preset number of sub-regions;

[0029] Calculate the pixel percentage of the dominant color in each sub-region within the dominant color range, and use each pixel percentage as the color importance of the corresponding sub-region;

[0030] Calculate the difference between the local image complexity of each sub-region and the overall image complexity, and use each difference as the texture importance of the corresponding sub-region;

[0031] The color importance and texture importance of each sub-region are weighted and summed to obtain the overall importance of each sub-region;

[0032] Sub-regions whose overall importance exceeds a preset threshold will be designated as key display areas.

[0033] By adopting the above technical solution, firstly, dividing the video frame into equally sized sub-regions provides a basic unit for evaluating the importance of each region. Secondly, color importance is obtained by calculating the pixel proportion of the dominant color within each sub-region, reflecting the salience of the region in terms of color features; texture importance is obtained by calculating the difference between the local complexity and the overall complexity of the sub-region, reflecting the uniqueness of the sub-region in terms of texture features. Then, by weighted summing of color importance and texture importance, a comprehensive evaluation of the importance of sub-regions is achieved, considering both color and texture features. Finally, by setting a threshold, regions with high overall importance are selected as key display areas, providing a precise basis for subsequent bitrate allocation.

[0034] Optionally, the step of obtaining a layered bitrate optimization strategy based on the product type and key display area of ​​the new product, and determining a bitrate transmission strategy based on the layered bitrate optimization strategy and the real-time network transmission rate of the current live stream, specifically includes:

[0035] Extract the lowest bitrate corresponding to the product type of the new product from the preset product type bitrate table as the initial bitrate;

[0036] The ratio of the current live stream resolution to the standard resolution is used as the resolution coefficient. The initial bitrate is multiplied by the resolution coefficient to obtain the base bitrate.

[0037] The bitrate of the key display area is set to a preset multiple of the base bitrate to obtain the first transmission bitrate, and the bitrate of the non-key display area is set to the base bitrate to obtain the second transmission bitrate.

[0038] Obtain the real-time network transmission rate of the current live stream, and optimize the first transmission bitrate of the key display area and the second transmission bitrate of the non-key display area based on the real-time network transmission rate to obtain a bitrate transmission strategy.

[0039] By adopting the above technical solution, the initial bitrate is first determined based on a preset product type bitrate table, ensuring the basic display quality requirements for different product types. Secondly, a resolution coefficient is introduced to dynamically adjust the initial bitrate, allowing bitrate allocation to adapt to different resolution scenarios. Then, by employing differentiated bitrate configuration strategies for critical and non-critical display areas, priority is given to key areas. Finally, by acquiring the network transmission rate in real time and optimizing the transmission bitrate for each area accordingly, bitrate allocation can both highlight key areas and adapt to dynamic changes in network conditions.

[0040] Optionally, the step of optimizing the first transmission bitrate of the critical display area and the second transmission bitrate of the non-critical display area based on the real-time network transmission rate to obtain a bitrate transmission strategy specifically includes:

[0041] Convert the real-time network transmission rate into the available total bitrate;

[0042] When the available total bitrate is greater than or equal to the first transmission bitrate, the first transmission bitrate is used as the target transmission bitrate for the critical display area, and the difference between the available total bitrate and the first transmission bitrate is used as the target transmission bitrate for the non-critical display area.

[0043] When the available total bitrate is less than the first transmission bitrate, if the available total bitrate is greater than the preset survival bitrate, the preset survival bitrate is used as the target transmission bitrate for the non-critical display area, and the difference between the available total bitrate and the preset survival bitrate is used as the target transmission bitrate for the critical display area. If the available total bitrate is less than or equal to the preset survival bitrate, the available total bitrate is used as the target transmission bitrate for the critical display area, and the target transmission bitrate for the non-critical display area is set to zero.

[0044] The transmission of the live video corresponding to the new product according to the bitrate transmission strategy includes:

[0045] The live video corresponding to the new product is transmitted according to the target transmission bitrate of the key display area and the target transmission bitrate of the non-key display area.

[0046] By adopting the above technical solution, firstly, by converting the real-time network transmission rate into the available total bitrate, specific resource constraints are provided for bitrate allocation. Secondly, three bitrate allocation strategies are designed under different network conditions: when network conditions are good, the optimal bitrate for critical display areas is prioritized, and the remaining bandwidth is used to improve the display quality of non-critical areas; when network conditions are average, a preset survival bitrate is introduced to ensure the basic availability of non-critical areas, and the remaining bandwidth is allocated to critical display areas; when network conditions are poor, all available bandwidth is concentrated on ensuring the display effect of critical display areas. This solution, through a dynamic bitrate allocation mechanism, can prioritize the protection of critical areas under different network conditions while taking into account the overall display effect, effectively solving the display quality problem under network fluctuation conditions. Finally, by applying the determined target transmission bitrate to the actual transmission process, the bitrate allocation scheme is implemented.

[0047] Optionally, the multimodal data includes video data, second audio data, and user interaction data. Determining the product switching time based on the multimodal data specifically includes:

[0048] Detect the product display area in the video data, compare the pixel change of the corresponding product display area in adjacent frames, and mark the moment when the pixel change exceeds a first preset threshold as the first candidate moment.

[0049] Extract audio segments of a preset duration before and after each of the first candidate moments from the second audio data, calculate the volume change value in adjacent audio segments, and mark the moment when the volume change value exceeds a second preset threshold as the second candidate moment;

[0050] Calculate the corresponding user comment increment based on the user interaction data before and after each second candidate time with a preset duration;

[0051] The moment when the increase in user comments exceeds a third preset threshold is determined as the product switching moment; when the increase in user comments does not exceed the third preset threshold, speech recognition is performed on each audio segment to obtain text content, and when the text content contains preset product switching-related instructional phrases, the moment when the product switching-related instructional phrases are recognized is determined as the product switching moment.

[0052] By employing the above technical solution, the first candidate time point is obtained by detecting pixel changes in the product display area of ​​the video data, thus achieving preliminary screening of potential product switching times. Secondly, by analyzing volume changes in audio clips before and after these candidate times, a second candidate time point is further selected, improving the accuracy of the judgment. Then, by analyzing the increase in comments in user interaction data, the occurrence of product switching is verified from the perspective of user feedback. Finally, when user feedback is not obvious, speech recognition technology is used to identify the anchor's instructive language as a supplementary basis for product switching judgment. This solution, through layer-by-layer screening and complementary verification of multimodal data, significantly improves the accuracy and reliability of product switching time judgment, providing a precise time reference for subsequent adaptive bitrate transmission.

[0053] A second aspect of this application provides an adaptive code rate transmission system, comprising:

[0054] The acquisition module is used to acquire the multimodal data of the current live broadcast and determine the product switching time based on the multimodal data;

[0055] The identification module is used to acquire the first audio data of the current live broadcast before the product switching time, and identify the product type of the new product in the first audio data;

[0056] The analysis module is used to extract the key features of the new product after detecting a product switch, and to determine the key display area of ​​the new product based on the key features;

[0057] The optimization module is used to obtain a layered bitrate optimization strategy based on the product type and key display area of ​​the new product, and to determine the bitrate transmission strategy based on the layered bitrate optimization strategy and the real-time network transmission rate of the current live broadcast.

[0058] The transmission module is used to transmit the live video corresponding to the new product according to the bitrate transmission strategy.

[0059] The above technical solution achieves several key advantages. First, by acquiring multimodal data from the live stream and determining the product switching time based on this data, accurate prediction of product switching can be realized. Before the product switching time, live stream audio data is acquired and new product types are identified. After detecting the product switching, key features of the new product are extracted and key display areas are determined, enabling precise positioning of the new product's display focus. Based on this, a layered bitrate optimization strategy is developed according to the product type and key display areas. By combining this strategy with real-time network transmission rates to determine the bitrate transmission strategy, adaptive transmission of the new product's display content is achieved. This approach effectively predicts product switching timing, quickly perceives the characteristics of new product content, and thus promptly adapts to the display needs of the new product.

[0060] A third aspect of this application provides an electronic device including a processor, a memory, a user interface, and a network interface, wherein the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any of the foregoing.

[0061] A fourth aspect of this application provides a computer-readable storage medium storing instructions that, when executed, perform the method described in any of the preceding descriptions. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the architecture of an adaptive bit rate transmission system disclosed in an embodiment of this application;

[0063] Figure 2 This is a flowchart illustrating an adaptive bit rate transmission method disclosed in an embodiment of this application;

[0064] Figure 3 This is a schematic diagram of a co-occurrence matrix generation based on a grayscale image disclosed in an embodiment of this application;

[0065] Figure 4 This is a schematic diagram of a module of an adaptive bit rate transmission system provided in an embodiment of this application;

[0066] Figure 5 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application.

[0067] Explanation of reference numerals in the attached figures: 901, processor; 902, communication bus; 903, user interface; 904, network interface; 905, memory. Detailed Implementation

[0068] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0069] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0070] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0071] Figure 1 An exemplary system architecture for an adaptive bit rate transmission system is shown.

[0072] like Figure 1 As shown, system architecture 10 may include electronic device 11, network 12, and streaming media server 13. Network 12 serves as the medium for providing a communication link between electronic device 11 and streaming media server 13. Network 12 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0073] Users can use electronic device 11 to interact with streaming media server 13 via network 12 to receive or send live video data, etc. Various live streaming applications can be installed on electronic device 11, such as live viewing applications, live streaming push applications, etc.

[0074] Electronic device 11 is hardware and can be various electronic devices with a display screen, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0075] The streaming media server 13 can be a server that provides various services, such as a streaming media server that performs adaptive bitrate transmission of live video. The streaming media server 13 can analyze and process the received live video data, determine the product switching time and key display areas, and dynamically adjust the bitrate transmission strategy according to the real-time network transmission rate, and transmit the processed video data to the electronic device 11.

[0076] The following detailed explanation uses the electronic device side as an example.

[0077] This embodiment discloses an adaptive bit rate transmission method. Figure 2 This is a flowchart illustrating an adaptive bit rate transmission method disclosed in an embodiment of this application, as shown below. Figure 2 As shown, the method includes steps S101 to S105:

[0078] S101: Obtain the multimodal data of the current live stream and determine the product switching time based on the multimodal data.

[0079] In this embodiment, multimodal data refers to data from different sensing channels, including video data, second audio data, and user interaction data. Specifically, video data is image data from the live stream, second audio data is the sound data of the host explaining the product, and user interaction data includes comments sent by users while watching the live stream, used to calculate the incremental user comments.

[0080] Specifically, the electronic device first calculates the pixel change of the product area in adjacent frames of the screen by acquiring video data during the live broadcast. At the same time, it extracts the volume difference between adjacent segments when the host is explaining the product from the second audio data, and counts the number of comments from users watching the live broadcast as multimodal data. When the pixel change exceeds a first preset threshold, it is marked as the first candidate moment. For each first candidate moment, when the volume difference exceeds a second preset threshold, it is marked as the second candidate moment. For each second candidate moment, when the increase in the number of comments first exceeds a third preset threshold, it is determined as the product switching moment. This accurately identifies the specific time point when the host switches products, providing an important time reference for subsequent adaptive bitrate transmission.

[0081] Based on the above embodiments, as an optional embodiment, the step of determining the product switching time based on multimodal data may include steps S1011 to S1014:

[0082] S1011: Detect the product display area in the video data, compare the pixel change of the corresponding product display area in adjacent frames, and mark the moment when the pixel change exceeds the first preset threshold as the first candidate moment.

[0083] Specifically, the electronic device locates the product display area in each frame of the video data from the multimodal data, calculates the difference between the pixel values ​​at the same position in the product display area of ​​the current frame and the previous frame, and accumulates all the differences to obtain the pixel change. When the pixel change exceeds a first preset threshold, the moment is taken as the first candidate moment. When the host switches to display a product, the image of the product display area will change significantly, resulting in a large pixel change between adjacent frames. Therefore, by detecting the pixel change, the moment when the product switch may occur can be preliminarily screened, providing candidate time points for subsequent accurate judgment.

[0084] S1012: Extract audio segments of preset duration before and after each first candidate moment from the second audio data, calculate the volume change value in adjacent audio segments, and mark the moment when the volume change value exceeds the second preset threshold as the second candidate moment.

[0085] Specifically, the electronic device extracts second audio data within a preset duration range before and after each first candidate moment from the second audio data in the multimodal data as audio segments, for example, extracting 10 seconds of second audio data before and after each first candidate moment. The volume difference between the current audio segment and the previous audio segment is calculated to obtain a volume change value. When this volume change value exceeds a second preset threshold, the moment is marked as a second candidate moment. Since broadcasters typically exhibit noticeable changes in tone or volume when switching product displays, detecting volume changes can further verify moments when product switching may occur.

[0086] S1013: Calculate the corresponding user comment increment based on the user interaction data before and after each second candidate time point with a preset duration.

[0087] Specifically, the electronic device obtains the total number of user comments within a preset time range before and after each second candidate time from the user interaction data in the multimodal data. For example, it obtains the number of comments sent by users within 30 seconds before and after the second candidate time. The difference between the total number of user comments in the current time period and the total number of comments in the previous time period is calculated to obtain the user comment increment. This increment is used to determine the intensity of user response to product switching. Since users often generate more comments and interactions when the host showcases new products, such as asking about product prices and specifications, a significant increase in the number of comments can serve as an important basis for determining the product switching time.

[0088] S1014: The moment when the increase in user comments exceeds the third preset threshold is determined as the product switching moment. When the increase in user comments does not exceed the third preset threshold, speech recognition is performed on each audio segment to obtain text content. When the text content contains preset product switching-related instructional phrases, the moment when the product switching-related instructional phrases are recognized is determined as the product switching moment.

[0089] Specifically, the electronic device first determines whether the increase in user comments exceeds a third preset threshold. When the increase exceeds the threshold, the corresponding time is directly determined as the product switching time. When the increase does not exceed the threshold, each audio segment is converted into text content, and the text content is checked to see if it contains preset product switching related instructions such as "Next, let's look at this item" or "Next product". When these instructions are detected for the first time, the corresponding time is determined as the product switching time.

[0090] S102: Before the product switching time, acquire the first audio data of the current live broadcast and identify the product type of the new product in the first audio data.

[0091] In this embodiment, the first audio data represents a segment of voice content spoken by the broadcaster before the product switching moment, used to predict the new product information to be displayed. Product type refers to the classification of the new product category, such as clothing, bags, accessories, etc.

[0092] Specifically, before determining the product switching time, the electronic device acquires the host's audio explanation as the first audio data. This audio data is then processed through speech recognition and analysis to convert it into text, extracting keywords related to the product type. Based on these keywords, the device identifies the product type of the new product the host is about to showcase. For example, in a clothing livestream, the converted text content of the host's first audio data might be: "Next, I'll introduce a new dress this year." The keyword extracted is "dress," thus determining the product type as "clothing."

[0093] S103: After detecting a product switch, extract the key features of the new product and determine the key display area of ​​the new product based on the key features.

[0094] In the embodiments of this application, key features include color distribution features and material texture features. The color distribution features include the distribution of different colors in each region, and the material texture features are characterized by a gray-level co-occurrence matrix, including features such as contrast, uniformity, and correlation.

[0095] The key display area refers to the main area in the live broadcast where new products are showcased.

[0096] Specifically, after determining the product switching time, the electronic device first divides the screen displaying the new product into multiple regions. For each region, color clustering is performed, and the number of pixels for each color is counted. The distribution characteristics of different colors in each region are obtained by calculating the ratio of the number of pixels in each cluster to the total number of pixels in the region. Then, grayscale conversion is performed on each region, and a second grayscale co-occurrence matrix is ​​constructed. The contrast is obtained by weighted summation of the squared differences between each element and its corresponding row and column numbers in the second grayscale co-occurrence matrix. The uniformity is obtained by summing the squared values ​​of each element. The correlation is obtained by weighted summation of the products of each element and its corresponding row and column numbers, thus obtaining the material texture features. When the proportion of color pixels in a region exceeds a first preset threshold, and the magnitude of the feature vector formed by contrast, uniformity, and correlation exceeds a second preset threshold, the region is marked as a key display region, thus completing the determination of the key display region for the new product in the live broadcast.

[0097] Based on the above embodiments, as an optional embodiment, the step of determining the key display area of ​​the new product according to key features may include steps S1031 to S1033:

[0098] S1031: Perform multi-level hierarchical clustering on the color distribution features to obtain multiple cluster centers. Extract the color value and pixel ratio of each cluster center. Use the color value as the horizontal axis and the pixel ratio as the vertical axis to generate a color distribution histogram. Select the center color value of the cluster center with the highest pixel ratio, and use the preset range centered on the center color value as the main color range of the new product.

[0099] Specifically, the electronic device performs multi-level hierarchical clustering operations on the color distribution features in the new product image, dividing pixels with similar colors into multiple cluster centers, extracting the color value corresponding to each cluster center and the ratio of the number of pixels contained in that cluster to the total number of pixels, plotting a color distribution histogram using the color value and pixel ratio as the horizontal and vertical axes respectively, selecting the central color value corresponding to the cluster center with the highest pixel ratio, and expanding the central color value to both sides of a preset range as the main color range of the new product.

[0100] S1032: Construct the gray-level co-occurrence matrix of material texture features, construct texture feature vectors based on the contrast, uniformity and correlation of the gray-level co-occurrence matrix, and calculate the modulus of the texture feature vectors as the overall image complexity of the new product.

[0101] Specifically, the electronic device first converts the RGB value of each pixel in the new product image into a grayscale value of 0-255. The grayscale values ​​are divided into multiple levels according to a preset range and numbered accordingly. An empty matrix with the number of rows and columns equal to the number of grayscale level numbers is created as the initial grayscale co-occurrence matrix. Each pixel in the image of the new product image is traversed, and the grayscale level number of the current pixel is used as the row number, and the grayscale level number of its right-hand neighboring pixel is used as the column number. The count is accumulated at the corresponding position in the initial grayscale co-occurrence matrix. After traversing all pixels, a complete grayscale co-occurrence matrix is ​​obtained. Each element value of the grayscale co-occurrence matrix is ​​divided by the sum of all element values ​​to obtain a normalized matrix. The squared difference between each element and its corresponding row and column numbers in the normalized matrix is ​​weighted and summed to obtain the contrast as the x-component. The squared sum of each element value is obtained to obtain the uniformity as the y-component. The weighted sum of the product of each element and its corresponding row and column numbers is obtained to obtain the correlation as the z-component. A three-dimensional vector is constructed as a texture feature vector, and the modulus of the texture feature vector is calculated to obtain the overall image complexity.

[0102] Based on the above embodiments, as an optional embodiment, the step of constructing a texture feature vector based on the contrast, uniformity, and correlation of the gray-level co-occurrence matrix may include steps S10321 to S10327:

[0103] S10321: Convert the image of the new product to a grayscale image.

[0104] In this embodiment of the application, the grayscale image represents a single-channel image in which each pixel of the new product image contains only brightness information, wherein the value range of each pixel is a continuous value from the darkest to the brightest.

[0105] Specifically, after acquiring the RGB color image of the new product, the electronic device extracts the red component (R value), green component (G value), and blue component (B value) of each pixel in the image. These three components are weighted according to the human eye's sensitivity to different colors and summed to obtain the gray value of the pixel. This calculation process is repeated for all pixels in the image to complete the grayscale conversion.

[0106] S10322: Determine multiple adjacent pixel combinations in the grayscale image according to the preset pixel spacing and direction, count the occurrence times of the grayscale value combinations corresponding to each adjacent pixel combination, and generate a grayscale co-occurrence matrix. The rows and columns of the grayscale co-occurrence matrix are all grayscale values, and the elements in the grayscale co-occurrence matrix are the occurrence times of the corresponding grayscale value combinations in the rows and columns.

[0107] Specifically, the electronic device pre-sets the pixel spacing and direction as the selection rules for adjacent pixels. For example, the selection rule is to select adjacent pixels according to a spacing of 1 pixel and the horizontal direction. According to the selection rule, each pixel in the grayscale image is traversed to determine the corresponding adjacent pixels. The grayscale value of the current pixel is used as the row number of the matrix, and the grayscale values ​​of the adjacent pixels are used as the column numbers of the matrix. The count value is accumulated at the corresponding position in the matrix to count the number of times this grayscale value combination appears in the image. The statistical process is repeated for all pixels to generate a grayscale co-occurrence matrix so that each element in the matrix records the frequency of occurrence of the corresponding row and column grayscale value combination.

[0108] like Figure 3 As shown, Figure 3 This is a schematic diagram of generating a co-occurrence matrix based on a grayscale image, as disclosed in an embodiment of this application. The left side of the diagram shows a grayscale image of a new product, and the right side shows the grayscale co-occurrence matrix. Taking the first row of the grayscale image of the new product as an example, when the pixel spacing is set to one pixel and the direction is horizontal, a pixel with a grayscale value of 1 forms a grayscale value combination with its adjacent pixel with a grayscale value of 1 on the right, and the element value of the first row and first column of the grayscale co-occurrence matrix is ​​incremented by one; a pixel with a grayscale value of 1 forms a grayscale value combination with its adjacent pixel with a grayscale value of 2 on the right, and the element value of the first row and second column of the grayscale co-occurrence matrix is ​​incremented by one; a pixel with a grayscale value of 2 forms a grayscale value combination with its adjacent pixel with a grayscale value of 3 on the right, and the element value of the second row and third column of the grayscale co-occurrence matrix is ​​incremented by one. After traversing all pixels, the element value at each position in the grayscale co-occurrence matrix represents the total number of times the combination of pixels in the corresponding row and corresponding column occurs.

[0109] S10323: Divide each element value of the gray-level co-occurrence matrix by the sum of all element values ​​to obtain the normalized matrix.

[0110] Specifically, the electronic device first calculates the sum of all element values ​​in the gray-level co-occurrence matrix, and then divides each element value in the gray-level co-occurrence matrix by the sum to obtain a normalized new matrix.

[0111] S10324: Multiply each element value in the normalized matrix by the square of the difference between the corresponding row and column gray values, and then sum all the products as the contrast of the gray-level co-occurrence matrix.

[0112] Specifically, for each element in the normalized matrix, the electronic device multiplies the element's value by the square of the difference between its row and column indices to obtain the corresponding product term. Then, the product terms of all elements are summed to obtain the contrast ratio. A higher contrast ratio indicates a greater difference in grayscale values ​​between adjacent pixels in the image, and more pronounced texture features.

[0113] S10325: Calculate the sum of squares of all elements in the normalized matrix to obtain the uniformity of the gray-level co-occurrence matrix.

[0114] Specifically, the electronic device squares each element value in the normalization matrix, then sums the squared values ​​of all elements to obtain the uniformity. The greater the uniformity, the more concentrated the gray values ​​of adjacent pixels in the image are, and the more uniform the texture distribution; the smaller the uniformity, the more dispersed the gray values ​​are, and the more irregular the texture distribution is.

[0115] S10326: Calculate the row standard deviation and column standard deviation in the normalized matrix, as well as the covariance of the normalized matrix. Divide the covariance by the product of the row standard deviation and the column standard deviation to obtain the correlation of the gray-level co-occurrence matrix.

[0116] Specifically, the electronic device first calculates the standard deviation of each row and each column in the normalized matrix. Then, it calculates the covariance of the normalized matrix, which is the weighted sum of the products of each element's value and its row and column indices, minus the product of the row mean and column mean. Finally, it divides the covariance by the product of the row and column standard deviations to obtain the correlation value. The closer the correlation value is to 1, the stronger the correlation between the grayscale values ​​of adjacent pixels in the image; the closer the correlation value is to 0, the weaker the correlation between the grayscale values.

[0117] S10327: Construct a three-dimensional vector with contrast as the x-component, uniformity as the y-component, and correlation as the z-component to obtain the texture feature vector.

[0118] Specifically, the electronic device uses the calculated contrast value as the x-axis component of the vector, the uniformity value as the y-axis component of the vector, and the correlation value as the z-axis component of the vector to construct a vector in three-dimensional space.

[0119] S1033: Determine the key display areas for new products based on the main color range and overall image complexity.

[0120] Specifically, the electronic device identifies multiple color regions within the main color range of an image. These color regions typically reflect the main appearance features of the product. The electronic device first converts the RGB value of each pixel in each color region to a grayscale value between 0 and 255. The grayscale values ​​are then divided into multiple levels according to a preset range and assigned corresponding numbers. An empty matrix with the number of rows and columns equal to the number of grayscale level numbers is created as the second initial grayscale co-occurrence matrix. The device iterates through each pixel in the image for each color region, using the grayscale level number of the current pixel as the row number and the grayscale level number of its right-hand neighboring pixel as the column number. The count is incremented at the corresponding position in the initial second grayscale co-occurrence matrix. After traversing all pixels, a complete second grayscale co-occurrence matrix is ​​obtained. Each pixel in the second grayscale co-occurrence matrix is ​​then... The element value is divided by the sum of all element values ​​to obtain the second normalized matrix. The weighted sum of the squared differences between each element and its corresponding row and column numbers in the second normalized matrix is ​​used to obtain the second contrast as the x component. The sum of the squares of each element value is used to obtain the second uniformity as the y component. The weighted sum of the products of each element and its corresponding row and column numbers is used to obtain the second correlation as the z component. A three-dimensional vector is constructed as the second texture feature vector. The modulus of the second texture feature vector is calculated to obtain the regional image complexity of each color region. When the regional image complexity is greater than a preset threshold, it indicates that the region contains more product details. Color regions whose ratio of regional image complexity to overall image complexity is greater than the preset complexity threshold are marked as key display areas.

[0121] Based on the above embodiments, as an optional embodiment, the step of determining the key display area of ​​the new product according to the main color range and the overall image complexity may include steps S10331 to S10335:

[0122] S10331: Divide the live video screen corresponding to the new product into a preset number of sub-areas.

[0123] Specifically, the electronic device divides the live video frame into equal horizontal and vertical sub-regions according to a preset number, resulting in grid-like sub-regions of equal size. Each sub-region contains the same number of pixels, forming a regular region division structure.

[0124] S10332: Calculate the percentage of pixels in the dominant color range of each sub-region, and use the percentage of each pixel as the color importance of the corresponding sub-region.

[0125] Specifically, the electronic device counts the number of pixels in each sub-region whose color values ​​fall within the range of the main color tone. This count is then divided by the total number of pixels in that sub-region to obtain the pixel percentage of the main color tone. This percentage is used as the color importance of the sub-region. The larger the percentage, the more likely the sub-region is to contain features of a new product.

[0126] S10333: Calculate the difference between the local image complexity and the overall image complexity of each sub-region, and use each difference as the texture importance of the corresponding sub-region.

[0127] Specifically, the electronic device converts the image of each sub-region into multi-level grayscale values, establishes a statistical matrix, and uses the grayscale values ​​of each pixel and the pixel to its right as the row and column indices of the matrix, respectively. The grayscale co-occurrence matrix of the sub-region is obtained by accumulating the counts at the corresponding positions. The weighted summation of the squared differences between each element and its row and column indices in the sub-region grayscale co-occurrence matrix yields the sub-region contrast value. The summation of the squared values ​​of each element yields the sub-region uniformity value. The weighted summation of the products of each element and its row and column indices yields the sub-region correlation value. The sub-region contrast, uniformity, and correlation values ​​are combined to form a sub-region texture feature vector. The modulus of this sub-region texture feature vector is calculated to obtain the local image complexity of each sub-region. The difference between the local complexity value and the overall complexity value of each sub-region is used as the texture importance of the corresponding sub-region.

[0128] S10334: The weighted sum of the color importance and texture importance of each sub-region is calculated to obtain the overall importance of each sub-region.

[0129] Specifically, the electronic device multiplies the color importance by the corresponding weighting coefficient and the texture importance by the corresponding preset weighting coefficient, and then adds them together to obtain the overall importance of each sub-region.

[0130] S10335: Designate sub-regions with a comprehensive importance exceeding a preset threshold as key display areas.

[0131] Specifically, the electronic device compares the overall importance of each sub-region with a pre-set threshold, and selects the sub-regions with an overall importance greater than the threshold to form the key display area.

[0132] S104: Based on the product type and key display area of ​​the new product, obtain a layered bitrate optimization strategy. Based on the layered bitrate optimization strategy and the real-time network transmission rate of the current live broadcast, determine the bitrate transmission strategy.

[0133] In this embodiment, the layered bitrate optimization strategy refers to dividing different areas of the live broadcast image into different encoding layers and assigning different transmission bitrates to each layer for data transmission. The bitrate transmission strategy refers to adjusting the transmission bitrate of each encoding layer based on the network transmission rate and transmitting data according to the adjusted transmission bitrate.

[0134] Specifically, the electronic device uses a preset mapping relationship between product type and encoding quality to determine the minimum encoding quality required for key display areas. The encoding quality of key display areas with a difference greater than the minimum encoding quality is designated as a high bitrate encoding layer, while the encoding quality of other areas is designated as a low bitrate encoding layer. A first quantization parameter, a first frame rate, and a first resolution are set for the high bitrate encoding layer, and a second quantization parameter, a second frame rate, and a second resolution are set for the low bitrate encoding layer. The first quantization parameter is less than the second quantization parameter, the first frame rate is higher than the second frame rate, and the first resolution is higher than the second resolution, thus obtaining a layered bitrate optimization strategy. Then, based on the current real-time network transmission rate of the live broadcast, an overall bitrate upper limit is set. Within this upper limit, the target bitrate of each layer is calculated according to a preset ratio (e.g., 70% for high bitrate encoding layer and 30% for low bitrate encoding layer). The bitrate generated by encoding is adjusted step by step to make it close to the target bitrate. When the bitrate generated by encoding exceeds the target bitrate, the quantization parameter is first increased step by step according to a preset step size until the maximum threshold is reached. If further adjustment is still needed, the frame rate is decreased step by step according to a preset multiplier until the minimum threshold is reached. Finally, the resolution is decreased step by step according to a preset level to finally generate the bitrate transmission strategy. For example, when the network transmission rate is 2Mbps, the target bitrate of the high bitrate coding layer is 1.4Mbps. If the bitrate generated by encoding is 1.8Mbps, then adjust it in the following order: first, increase the quantization parameter step by step in increments of 2 (e.g., from 22 to 24, 26, with a maximum of no more than 30). If it still exceeds this limit, then decrease the frame rate step by step in increments of 0.8 (e.g., from 30fps to 24fps, 20fps, with a minimum of no less than 15fps). If necessary, decrease the resolution step by step in increments of 1080P, 720P, 480P.

[0135] Based on the above embodiments, as an optional embodiment, the step of optimizing the hierarchical bitrate strategy and the current live streaming real-time network transmission rate may include steps S1041 to S1044:

[0136] S1041: Extract the lowest bitrate corresponding to the product type of the new product from the preset product type bitrate table as the initial bitrate.

[0137] Specifically, the electronic device stores the correspondence between different product types and minimum bitrates in a preset product type bitrate table. Product types include clothing, jewelry, cosmetics, etc. Each product type has a corresponding minimum bitrate value. The electronic device extracts the minimum bitrate value corresponding to the product type of a new product as the initial bitrate. This initial bitrate can ensure the basic display quality of key display areas.

[0138] S1042: Use the ratio of the current live stream resolution to the standard resolution as the resolution coefficient, and multiply the initial bitrate by the resolution coefficient to obtain the base bitrate.

[0139] Specifically, the electronic device calculates the ratio of the current live screen resolution to the preset standard resolution to obtain the resolution coefficient. The initial bitrate is multiplied by this resolution coefficient to obtain the base bitrate. The introduction of the resolution coefficient allows the bitrate to be adjusted accordingly as the live screen resolution changes, thereby ensuring the encoding quality at different resolutions.

[0140] S1043: Set the bitrate of the critical display area to a preset multiple of the base bitrate to obtain the first transmission bitrate, and set the bitrate of the non-critical display area to the base bitrate to obtain the second transmission bitrate.

[0141] Specifically, the electronic device amplifies the base bitrate using a preset bitrate multiplier to obtain the first transmission bitrate. This preset multiplier is greater than 1, ensuring that the key display areas obtain higher encoding quality. At the same time, the base bitrate is directly used as the second transmission bitrate, so that the non-key display areas maintain a basic display effect, thereby achieving key protection for the key display areas.

[0142] S1044: Obtain the real-time network transmission rate of the current live stream, optimize the second transmission bitrate of the key display area and the second transmission bitrate of the non-key display area based on the real-time network transmission rate, and obtain the bitrate transmission strategy.

[0143] Specifically, the electronic device obtains the real-time network transmission rate of the current live broadcast. When the real-time network transmission rate is greater than the sum of the first transmission bitrate and the second transmission bitrate, the two transmission bitrates are kept unchanged. When the real-time network transmission rate is less than the sum of the first transmission bitrate and the second transmission bitrate, the first transmission bitrate is given priority, and the remaining bandwidth is used as the second transmission bitrate to form the final bitrate transmission strategy.

[0144] Based on the above embodiments, as an optional embodiment, the step of optimizing the first transmission bitrate of the key display area and the second transmission bitrate of the non-key display area based on the real-time network transmission rate to obtain the bitrate transmission strategy may include steps S10441 to S1043:

[0145] S10441: Converts real-time network transmission rate to available total bitrate.

[0146] Specifically, the electronic device multiplies the acquired real-time network transmission rate by a preset conversion coefficient to obtain the total available bitrate. This conversion coefficient is less than 1, which is used to reserve a certain amount of network bandwidth redundancy to avoid stuttering caused by network fluctuations and ensure that the bitrate allocation scheme has sufficient fault tolerance.

[0147] S10442: When the available total bitrate is greater than or equal to the first transmission bitrate, the first transmission bitrate is used as the target transmission bitrate for the critical display area, and the difference between the available total bitrate and the first transmission bitrate is used as the target transmission bitrate for the non-critical display area.

[0148] Specifically, the electronic device determines whether the available total bitrate is greater than or equal to the first transmission bitrate. When the available total bitrate is greater than or equal to the first transmission bitrate, the first transmission bitrate is directly used as the target transmission bitrate for the critical display area to ensure that the critical display area obtains the expected encoding quality. At the same time, the difference between the available total bitrate and the first transmission bitrate is calculated as the target transmission bitrate for the non-critical display area to achieve reasonable utilization of the remaining bandwidth.

[0149] S10443: When the available total bit rate is less than the first transmission bit rate, if the available total bit rate is greater than the preset survival bit rate, the preset survival bit rate is used as the target transmission bit rate for non-critical display areas, and the difference between the available total bit rate and the preset survival bit rate is used as the target transmission bit rate for critical display areas. If the available total bit rate is less than or equal to the preset survival bit rate, the available total bit rate is used as the target transmission bit rate for critical display areas, and the target transmission bit rate for non-critical display areas is set to zero.

[0150] Specifically, when the electronic device determines that the available total bitrate is less than the first transmission bitrate, it further determines the relationship between the available total bitrate and the preset survival bitrate. When the available total bitrate is greater than the preset survival bitrate, the preset survival bitrate is used as the target transmission bitrate for non-critical display areas to ensure basic image quality, and the difference between the available total bitrate and the preset survival bitrate is used as the target transmission bitrate for critical display areas. When the available total bitrate is less than or equal to the preset survival bitrate, all available total bitrate is used as the target transmission bitrate for critical display areas, while the target transmission bitrate for non-critical display areas is set to zero to ensure priority transmission for critical display areas under extreme bandwidth conditions.

[0151] S105: Transmit the live video corresponding to the new product according to the bitrate transmission strategy.

[0152] Specifically, the electronic device encodes video for critical and non-critical display areas separately according to the target transmission bitrate determined in the bitrate transmission strategy. For critical display areas, high-quality encoding is performed using their corresponding target transmission bitrate; for non-critical display areas, basic-quality encoding is performed using their corresponding target transmission bitrate. After encoding, the video data from the critical and non-critical display areas are transmitted over the network according to their respective target transmission bitrates, achieving differentiated video transmission effects.

[0153] This embodiment also discloses an adaptive bit rate transmission system. Figure 4This is a schematic diagram of a module of an adaptive bit rate transmission system disclosed in an embodiment of this application. The system includes:

[0154] The acquisition module is used to acquire the multimodal data of the current live broadcast and determine the product switching time based on the multimodal data;

[0155] The identification module is used to acquire the first audio data of the current live broadcast before the product switching time, and identify the product type of the new product in the first audio data;

[0156] The analysis module is used to extract the key features of the new product after detecting a product switch, and to determine the key display area of ​​the new product based on the key features;

[0157] The optimization module is used to obtain a layered bitrate optimization strategy based on the product type and key display area of ​​the new product, and to determine the bitrate transmission strategy based on the layered bitrate optimization strategy and the real-time network transmission rate of the current live broadcast.

[0158] The transmission module is used to transmit the live video corresponding to the new product according to the bitrate transmission strategy.

[0159] Optionally, the analysis module is further configured to perform multi-level hierarchical clustering on the color distribution features to obtain multiple cluster centers, extract the color value and pixel ratio of each cluster center, generate a color distribution histogram using the color value as the horizontal axis and the pixel ratio as the vertical axis, select the center color value of the cluster center with the highest pixel ratio, and use a preset range centered on the center color value as the main color range of the new product; construct a gray-level co-occurrence matrix of the material texture features, construct a texture feature vector based on the contrast, uniformity, and correlation of the gray-level co-occurrence matrix, calculate the modulus of the texture feature vector as the overall image complexity of the new product; and determine the key display area of ​​the new product based on the main color range and the overall image complexity.

[0160] Optionally, the analysis module is further configured to convert the image of the new product into a grayscale image; determine multiple adjacent pixel combinations in the grayscale image according to a preset pixel spacing and direction; count the occurrence frequency of grayscale value combinations corresponding to each adjacent pixel combination; generate a grayscale co-occurrence matrix, wherein the rows and columns of the grayscale co-occurrence matrix are all grayscale values, and the elements in the grayscale co-occurrence matrix are the occurrence frequency of the corresponding grayscale value combinations in the rows and columns; divide each element value of the grayscale co-occurrence matrix by the sum of all element values ​​to obtain a normalized matrix; and multiply each element value in the normalized matrix by the corresponding row sum. The contrast of the gray-level co-occurrence matrix is ​​obtained by squared differences of gray values ​​represented by columns and summing all products. The uniformity of the gray-level co-occurrence matrix is ​​obtained by summing the squares of all elements in the normalized matrix. The row standard deviation and column standard deviation of the normalized matrix, as well as the covariance of the normalized matrix, are calculated and divided by the product of the row standard deviation and the column standard deviation. The correlation of the gray-level co-occurrence matrix is ​​obtained by constructing a three-dimensional vector with the contrast as the x-component, the uniformity as the y-component, and the correlation as the z-component.

[0161] Optionally, the analysis module is further configured to divide the live video frame corresponding to the new product into a preset number of sub-regions; calculate the pixel proportion of the dominant color of each sub-region within the range of the dominant color, and use each pixel proportion as the color importance of the corresponding sub-region; calculate the difference between the local image complexity and the overall image complexity of each sub-region, and use each difference as the texture importance of the corresponding sub-region; perform a weighted summation of the color importance and texture importance of each sub-region to obtain the comprehensive importance of each sub-region; and designate the sub-regions with a comprehensive importance exceeding a preset threshold as the key display areas.

[0162] Optionally, the optimization module is further configured to: extract the lowest bitrate corresponding to the product type of the new product from a preset product type bitrate table as the initial bitrate; use the ratio of the resolution of the current live stream to the standard resolution as a resolution coefficient, multiply the initial bitrate by the resolution coefficient to obtain the base bitrate; set the bitrate of the key display area to a preset multiple of the base bitrate to obtain a first transmission bitrate, and set the bitrate of the non-key display area to the base bitrate to obtain a second transmission bitrate; obtain the real-time network transmission rate of the current live stream, and optimize the first transmission bitrate of the key display area and the second transmission bitrate of the non-key display area based on the real-time network transmission rate to obtain a bitrate transmission strategy.

[0163] Optionally, the optimization module is further configured to convert the real-time network transmission rate into an available total bitrate; when the available total bitrate is greater than or equal to the first transmission bitrate, the first transmission bitrate is used as the target transmission bitrate for the critical display area, and the difference between the available total bitrate and the first transmission bitrate is used as the target transmission bitrate for the non-critical display area; when the available total bitrate is less than the first transmission bitrate, if the available total bitrate is greater than a preset survival bitrate, the preset survival bitrate is used as the target transmission bitrate for the non-critical display area, and the difference between the available total bitrate and the preset survival bitrate is used as the target transmission bitrate for the critical display area; if the available total bitrate is less than or equal to the preset survival bitrate, the available total bitrate is used as the target transmission bitrate for the critical display area, and the target transmission bitrate for the non-critical display area is set to zero; the transmission of the live video corresponding to the new product according to the bitrate transmission strategy includes: transmitting the live video corresponding to the new product according to the target transmission bitrate of the critical display area and the target transmission bitrate of the non-critical display area.

[0164] Optionally, the acquisition module is further configured to detect the product display area in the video data, compare the pixel changes of the corresponding product display area in adjacent frames, and mark the time when the pixel change exceeds a first preset threshold as a first candidate time; extract audio segments of preset duration before and after each first candidate time from the second audio data, calculate the volume change value in adjacent audio segments, and mark the time when the volume change value exceeds a second preset threshold as a second candidate time; calculate the corresponding user comment increment based on user interaction data of preset duration before and after each second candidate time; determine the time when the user comment increment exceeds a third preset threshold as the product switching time; when the user comment increment does not exceed the third preset threshold, perform speech recognition on each audio segment to obtain text content, and when the text content contains preset product switching-related instructional phrases, determine the time when the product switching-related instructional phrases are recognized as the product switching time.

[0165] It should be noted that the above embodiments of the apparatus are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0166] This embodiment also discloses an electronic device, referring to... Figure 5The electronic device may include: at least one processor 901, at least one communication bus 902, user interface 903, network interface 904, and at least one memory 905.

[0167] The communication bus 902 is used to enable communication between these components.

[0168] The user interface 903 may include a display screen and a camera. Optional user interfaces may also include standard wired interfaces and wireless interfaces.

[0169] The network interface 904 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0170] The processor 901 may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and calling data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array. The processor may integrate one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0171] The memory 905 may include random access memory (RAM) or read-only memory. Optionally, the memory may include a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor. As shown in the figure, the memory, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for an adaptive bit rate transmission method.

[0172] exist Figure 5 In the electronic device shown, the user interface is mainly used to provide an input interface for the user and to obtain the user input data; while the processor can be used to call an application program stored in the memory that provides an adaptive bit rate transmission method. When executed by one or more processors, the electronic device executes one or more methods as described in the above embodiments.

[0173] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0174] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some service interfaces; indirect couplings or communication connections between apparatuses or units may be electrical or other forms.

[0176] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0178] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0179] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will be readily apparent to those skilled in the art upon consideration of the disclosure in this specification. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. An adaptive code rate transmission method, characterized in that, The method includes: acquiring multimodal data of the current live stream, wherein the multimodal data includes video data, second audio data, and user interaction data; The process involves detecting the product display area in the video data, comparing the pixel changes in the corresponding product display area in adjacent frames, and marking the moment when the pixel change exceeds a first preset threshold as a first candidate moment; extracting audio segments of preset duration before and after each first candidate moment from the second audio data, calculating the volume change value in adjacent audio segments, and marking the moment when the volume change value exceeds a second preset threshold as a second candidate moment; calculating the corresponding user comment increment based on user interaction data of preset duration before and after each second candidate moment; and determining the moment when the user comment increment exceeds a third preset threshold as the product switching moment. When the increase in user comments does not exceed a third preset threshold, speech recognition is performed on each audio segment to obtain text content. When the text content contains preset product switching-related instructional phrases, the time corresponding to the identification of the product switching-related instructional phrases is determined as the product switching time. Before the product switching time, the first audio data of the current live broadcast is obtained, and the product type of the new product is identified in the first audio data. After detecting the product switching, the key features of the new product are extracted, and the key display area of ​​the new product is determined based on the key features. The minimum bitrate corresponding to the product type of the new product is extracted from a preset product type bitrate table as the initial bitrate; the ratio of the current live stream resolution to the standard resolution is used as a resolution coefficient, and the initial bitrate is multiplied by the resolution coefficient to obtain the base bitrate; the bitrate of the key display area is set to a preset multiple of the base bitrate to obtain the first transmission bitrate, and the bitrate of the non-key display area is set to the base bitrate to obtain the second transmission bitrate; the real-time network transmission rate of the current live stream is obtained, and the first transmission bitrate of the key display area and the second transmission bitrate of the non-key display area are optimized based on the real-time network transmission rate to obtain the bitrate transmission strategy; The live video corresponding to the new product is transmitted according to the bitrate transmission strategy.

2. The method according to claim 1, characterized in that, The key features include color distribution features and material texture features. Determining the key display area of ​​the new product based on these key features specifically includes: Multi-level hierarchical clustering is performed on the color distribution features to obtain multiple cluster centers. The color value and pixel ratio of each cluster center are extracted. A color distribution histogram is generated using the color value as the horizontal axis and the pixel ratio as the vertical axis. The center color value of the cluster center with the highest pixel ratio is selected, and a preset range centered on the center color value is used as the main color range of the new product. Construct a gray-level co-occurrence matrix of the material texture features, construct a texture feature vector based on the contrast, uniformity and correlation of the gray-level co-occurrence matrix, and calculate the modulus of the texture feature vector as the overall image complexity of the new product. Based on the main color range and overall image complexity, the key display areas for the new product are determined.

3. The method according to claim 2, characterized in that, The construction of the gray-level co-occurrence matrix of the material texture features, and the construction of the texture feature vector based on the contrast, uniformity, and correlation of the gray-level co-occurrence matrix, specifically includes: Convert the image of the new product to a grayscale image; According to the preset pixel spacing and direction, multiple adjacent pixel combinations in the grayscale image are determined, the occurrence frequency of grayscale value combinations corresponding to each adjacent pixel combination is counted, and a grayscale co-occurrence matrix is ​​generated. The rows and columns of the grayscale co-occurrence matrix are all grayscale values, and the elements in the grayscale co-occurrence matrix are the occurrence frequency of the corresponding grayscale value combinations in the rows and columns. Divide each element value of the gray-level co-occurrence matrix by the sum of all element values ​​to obtain the normalized matrix; Multiply each element value in the normalized matrix by the square of the difference between the corresponding row and column gray values, and then sum all the products as the contrast of the gray co-occurrence matrix. The uniformity of the gray-level co-occurrence matrix is ​​obtained by calculating the sum of squares of all elements in the normalized matrix. Calculate the row standard deviation and column standard deviation in the normalized matrix, as well as the covariance of the normalized matrix. Divide the covariance by the product of the row standard deviation and the column standard deviation to obtain the correlation of the gray-level co-occurrence matrix. The contrast is used as the x-component, the uniformity as the y-component, and the correlation as the z-component to construct a three-dimensional vector, thus obtaining the texture feature vector.

4. The method according to claim 2, characterized in that, The step of determining the key display area of ​​the new product based on the main color range and the overall image complexity specifically includes: The live video screen corresponding to the new product is divided into a preset number of sub-regions; Calculate the pixel percentage of the dominant color in each sub-region within the dominant color range, and use each pixel percentage as the color importance of the corresponding sub-region; Calculate the difference between the local image complexity of each sub-region and the overall image complexity, and use each difference as the texture importance of the corresponding sub-region; The color importance and texture importance of each sub-region are weighted and summed to obtain the overall importance of each sub-region; Sub-regions whose overall importance exceeds a preset threshold will be designated as key display areas.

5. The method according to claim 1, characterized in that, The method of optimizing the first transmission bitrate of the critical display area and the second transmission bitrate of the non-critical display area based on the real-time network transmission rate to obtain a bitrate transmission strategy specifically includes: Convert the real-time network transmission rate into the available total bitrate; When the available total bitrate is greater than or equal to the first transmission bitrate, the first transmission bitrate is used as the target transmission bitrate for the critical display area, and the difference between the available total bitrate and the first transmission bitrate is used as the target transmission bitrate for the non-critical display area. When the available total bitrate is less than the first transmission bitrate, if the available total bitrate is greater than the preset survival bitrate, the preset survival bitrate is used as the target transmission bitrate for the non-critical display area, and the difference between the available total bitrate and the preset survival bitrate is used as the target transmission bitrate for the critical display area. If the available total bitrate is less than or equal to the preset survival bitrate, the available total bitrate is used as the target transmission bitrate for the critical display area, and the target transmission bitrate for the non-critical display area is set to zero. The transmission of the live video corresponding to the new product according to the bitrate transmission strategy includes: The live video corresponding to the new product is transmitted according to the target transmission bitrate of the key display area and the target transmission bitrate of the non-key display area.

6. An adaptive bit rate transmission system, characterized in that, Specifically, it includes: The acquisition module is used to acquire multimodal data of the current live stream; wherein, the multimodal data includes video data, second audio data, and user interaction data; the acquisition module is also used to detect the product display area in the video data, compare the pixel change of the corresponding product display area in adjacent frames, and mark the time when the pixel change exceeds a first preset threshold as a first candidate time; extract audio segments of preset duration before and after each first candidate time from the second audio data, calculate the volume change value in adjacent audio segments, and mark the time when the volume change value exceeds a second preset threshold as a second candidate time; and calculate the corresponding user interaction data based on the user interaction data of preset duration before and after each second candidate time. The system calculates the increase in user comments; it determines the moment when the increase in user comments exceeds a third preset threshold as the product switching moment; when the increase in user comments does not exceed the third preset threshold, it performs speech recognition on each audio segment to obtain text content; when the text content contains preset product switching-related instructional phrases, it determines the moment when the product switching-related instructional phrases are recognized as the product switching moment; the recognition module is used to acquire the first audio data of the current live stream before the product switching moment, and identify the product type of the new product in the first audio data; the analysis module is used to extract the key features of the new product after detecting the product switching, and determine the key display area of ​​the new product based on the key features; An optimization module is used to extract the lowest bitrate corresponding to the product type of the new product from a preset product type bitrate table as the initial bitrate; use the ratio of the resolution of the current live broadcast to the standard resolution as a resolution coefficient, multiply the initial bitrate by the resolution coefficient to obtain the base bitrate; set the bitrate of the key display area to a preset multiple of the base bitrate to obtain the first transmission bitrate, and set the bitrate of the non-key display area to the base bitrate to obtain the second transmission bitrate; obtain the real-time network transmission rate of the current live broadcast, and optimize the first transmission bitrate of the key display area and the second transmission bitrate of the non-key display area based on the real-time network transmission rate to obtain a bitrate transmission strategy; The transmission module is used to transmit the live video corresponding to the new product according to the bitrate transmission strategy.

7. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions. The user interface and the network interface are both used to communicate with other devices. The processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the method as described in any one of claims 1 to 5.

Images