Television quick shopping interaction system based on ultrasonic recognition

Abstract

The invention discloses a television quick shopping interaction system based on ultrasonic recognition, and particularly relates to the technical field of artificial intelligence, user interaction data and screen content data are collected, ultrasonic echo signals and video content data are obtained through an ultrasonic sensing array and a video stream obtaining unit, the ultrasonic echo signals are processed, and the video content data are obtained through a video stream processing unit. The method comprises the following steps of: extracting a first track and a position feature of a hand in a three-dimensional space, identifying a shopping pointing gesture predefined by a user by utilizing a pre-trained deep learning model, analyzing acquired real-time video stream data, identifying shopping visual elements in a picture and pixel-level position information of the shopping visual elements on a screen, and displaying the shopping visual elements in the picture and the pixel-level position information of the shopping visual elements. And according to the gesture features of the user and the visual element information of the commodity, judging a shopping intention of the user, determining a target commodity pointed by the intention, and controlling screen display of the smart television and user interaction to complete a quick shopping operation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a TV-based quick shopping interactive system based on ultrasonic recognition. Background Technology

[0002] With the advancement of technology, television has evolved from a single entertainment tool into a smart terminal that integrates multiple functions such as information acquisition, social interaction, and online shopping. Especially in the shopping field, the traditional TV shopping model has evolved from telephone ordering and remote control selection to QR code scanning.

[0003] However, existing TV shopping interaction methods still suffer from limitations in user experience, including clumsy interaction methods, inefficient and ambiguous input of intent, and a disconnect between product information and video content. Users mostly rely on traditional remote controls, navigating through complex graphical interfaces using directional keys to select target products, a cumbersome process that severely disrupts the immersive viewing experience. Although voice search solutions have emerged, users need to accurately describe the product name, resulting in low recognition rates in noisy environments or when the description is unclear, and it struggles to handle directional intents such as "this item on the screen." Furthermore, it cannot achieve real-time and accurate association between products and dynamic video images; users must manually search for products appearing on screen, a time-consuming and error-prone process. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide a TV quick shopping interactive system based on ultrasonic recognition to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a TV quick shopping interaction system based on ultrasonic recognition, comprising a data acquisition module, a gesture recognition module, a product positioning module, a shopping intent matching module, and a shopping interaction module;

[0006] The data acquisition module is used to simultaneously acquire user interaction data and screen content data, and to acquire ultrasonic echo signals and video content data through an ultrasonic sensor array and a video stream acquisition unit.

[0007] The gesture recognition module is connected to the ultrasonic sensor array in the data acquisition module. It processes the ultrasonic echo signal, extracts the first trajectory and position features of the hand in three-dimensional space, and uses a pre-trained deep learning model to recognize the user's predefined shopping pointing gesture.

[0008] The product positioning module is connected to the video stream acquisition unit in the data acquisition module, and is used to analyze the acquired real-time video stream data to identify the purchasable visual elements in the picture and their pixel-level position information on the screen.

[0009] The shopping intent matching module is connected to the gesture recognition module and the product positioning module respectively. It is used to analyze the user's gesture features and the visual element information of the product to determine the user's shopping intent and identify the target product to which the intent is directed.

[0010] The shopping interaction module is connected to the shopping intent matching module and is used to receive the target product information determined by the shopping intent matching module, and control the screen display and user interaction of the smart TV to complete the quick shopping operation.

[0011] Preferably, as a preferred embodiment of the TV quick shopping interactive system based on ultrasonic recognition described in this invention, it includes the data acquisition module for synchronously acquiring user interaction data and screen content data, and acquiring ultrasonic echo signals and video content data through an ultrasonic sensor array and a video stream acquisition unit;

[0012] The ultrasonic sensing array consists of at least two ultrasonic transmitters and at least two ultrasonic receivers, wherein the ultrasonic transmitters periodically transmit ultrasonic signals of a specific frequency and waveform to the user interaction area.

[0013] The ultrasonic receiver synchronously listens to and collects the ultrasonic echo signal reflected from the user's hand, converts the sound wave signal into an electrical signal, and performs pre-amplification and filtering analog processing. Through analog-to-digital conversion, it forms a digital ultrasonic echo data stream.

[0014] The transmitter and receiver, as well as multiple transmitters and receivers, are synchronized using a high-precision clock to ensure accurate measurement of signal propagation time.

[0015] The video stream acquisition unit is used to acquire the video content data currently being played on the smart TV in real time, and further includes:

[0016] Compressed digital video frame data is obtained from the video decoder of the smart TV and transmitted at a set frame rate. The data includes pixel information and timestamps for each video frame. The video frame data acquisition unit synchronizes the acquisition of video frame data with the acquisition of ultrasonic signals to ensure that subsequent modules can associate gestures at a specific moment with video images at the same moment during processing.

[0017] Preferably, as a preferred embodiment of the TV quick shopping interactive system based on ultrasonic recognition described in this invention, it includes a gesture recognition module connected to an ultrasonic sensor array in the data acquisition module, processing the ultrasonic echo signal, extracting the first trajectory and position features of the hand in three-dimensional space, and using a pre-trained deep learning model to recognize the user's predefined shopping pointing gestures, specifically including the following:

[0018] The raw digitized ultrasonic echo data received from the ultrasonic sensor array is preprocessed, including noise reduction and filtering, and echo signal enhancement. Based on the preprocessed ultrasonic echo signal, an ultrasonic positioning algorithm is used to calculate the real-time position of the user's hand in three-dimensional space to extract the three-dimensional position sequence of the hand. Velocity vector sequence and acceleration vector sequence ;

[0019] Using three-dimensional position sequences, velocity vector sequences, and acceleration vector sequences as input to a deep learning model, the model recognizes user-defined shopping pointing gestures, and further includes:

[0020] At each time step Real-time three-dimensional position of the hand Instantaneous velocity vector Combination with instantaneous acceleration vector This forms a high-dimensional feature vector. ;

[0021] continuous time steps eigenvectors Form a sequence of time-series feature vectors As input to a deep learning model: Where M is the length of the time sequence, representing the time window in which the gesture lasts;

[0022] The time series feature vector sequence Input into the pre-trained deep learning model In this process, feature extraction and pattern recognition are performed on the input sequence. Through nonlinear transformation and inter-layer computation, a probability distribution vector representing different predefined gesture categories is finally output. ,in, , representing the predicted probability for C different gesture categories, and , Let be the probability of recognizing it as the j-th gesture;

[0023] According to the probability distribution vector The category with the highest prediction probability is selected as the final gesture recognition result. When the highest prediction probability Exceeding the preset confidence threshold Then it confirms that the user's predefined shopping pointing gesture has been recognized. The gesture information is then output to subsequent modules for processing.

[0024] The ultrasonic positioning algorithm includes the following steps:

[0025] The measurement involves the time difference between the ultrasonic signal emitted by the transmitter, reflected by the user's hand, and reaching the i-th receiver. Calculate the ultrasonic signal transmitted from the transmitter to the receiver. Total propagation distance Select a reference receiver as Calculate other receivers With the reference receiver The time difference between them is The distance difference is obtained as ,in, This represents the speed at which ultrasound travels through the air. For the launch time, It is the i-th receiver The moment the reflected signal is received;

[0026] The coordinates of the hand's desired position in space are represented as... According to the transmitter coordinates and receiver coordinates Establish the system of hyperbolic equations: ,in, Indicates hand to receiver The distance;

[0027] The hyperbolic equations are linearized, and the least squares method is used to iteratively solve for the three-dimensional coordinates of the hand in space by minimizing the equation residuals. ;

[0028] A receiver array is formed by using multiple closely spaced receivers, and the phase difference between the reflected signals arriving at different receivers is measured. Calculate the azimuth angle of the ultrasonic wave reaching the receiving array. To assist in positioning, among which, d is the ultrasonic wavelength, d is the distance between adjacent receivers, and N represents the number of receivers;

[0029] The three-dimensional location point sequence obtained from continuous timestamps The real-time motion trajectory of the hand is constructed, and the trajectory data is differentiated by first and second order to extract the instantaneous velocity vector sequence of the hand. and acceleration vector sequence .

[0030] Preferably, as a preferred embodiment of the TV quick shopping interactive system based on ultrasonic recognition described in this invention, it includes a product positioning module connected to a video stream acquisition unit in the data acquisition module, used to analyze the acquired real-time video stream data and identify purchasable visual elements in the picture and their pixel-level position information on the screen, specifically including the following:

[0031] The received compressed digital video frame data is decoded to restore the original pixel data, each video frame is acquired, and the video frames are preprocessed, including noise suppression, contrast adjustment, and image enhancement operations, to optimize image quality.

[0032] Using a pre-trained deep learning model for object detection, the pre-processed video frames are analyzed to output the detected visual elements suitable for shopping. The set; for each identified shopperable visual element, output its bounding box in the current video frame. and corresponding category labels ,in, Indicates the coordinates of the top left corner. It's the width. It is height;

[0033] For each identified shopperable visual element Through a pre-trained image segmentation model Extract its pixel-level mask The pixel-level mask This is a binary image, where a pixel value of 1 indicates that it belongs to the element, and 0 indicates that it does not belong to the element.

[0034] The segmentation process is represented as follows: ;in, For input video frames, For the detected bounding box, For the corresponding category label, This indicates that the image segmentation model classifies pixels within a specified region and generates a binary mask;

[0035] Based on the pixel-level mask The centroid coordinates of the element on the screen are calculated as follows: , ,in, These are the coordinates of the centroid in the image pixel coordinate system;

[0036] The system performs temporal tracking of shoppable visual elements identified in consecutive video frames to ensure the correlation of the same element across different frames. It also associates the identified bounding boxes with pixel-level masks, assigns a unique ID to each shoppable visual element, and outputs a list of all shoppable visual elements identified and tracked at each time stamp, including bounding boxes. Pixel mask Centroid coordinates Category tags And tracking ID.

[0037] Preferably, as a preferred embodiment of the TV quick shopping interaction system based on ultrasonic recognition described in this invention, it includes a shopping intent matching module connected to a gesture recognition module and a product positioning module, respectively, for analyzing the user's gesture features and the visual element information of the product to determine the user's shopping intent and identify the target product of the intent, specifically including the following:

[0038] When the gesture recognition module detects a pointing gesture indicating a user's shopping intention, it calculates the candidate region mapped onto the product location map on the screen based on the hand's three-dimensional position in space and the gesture's direction vector. The gesture direction vector is obtained from the instantaneous velocity vector of the hand, and further includes:

[0039] The recognition result output by the gesture recognition module includes the instantaneous velocity vector of the hand when the pointing gesture is detected. 3D position of the hand and recognized shopping pointing gestures The three-dimensional position of the user's hand when a pointing gesture is detected. With gesture direction vector Form a ray ,in, These are parameters of the ray, representing the distance along the direction of the velocity vector;

[0040] Calculate the intersection point between ray L and the screen plane. If the ray does not intersect the screen plane, it is considered an invalid direction, and subsequent processing stops. If a valid intersection point exists, convert the intersection point from 3D spatial coordinates to 2D pixel coordinates on the screen. A two-dimensional circular candidate region with radius r is generated, centered on the two-dimensional pixel coordinates of the screen. ;

[0041] Spatially match the candidate region with the boundary information of all purchasable items in the screen product location map, and filter out all items that match the candidate region. Goods that have spatial intersection form a candidate goods set. The system prioritizes gestures based on their accuracy, duration, and user history preferences to determine the target product the user intends to access.

[0042] The accuracy of the gesture is defined as the distance between the center of the gesture and the center of mass of the product. By converting it into a precision score This indicates the precision with which the user's gesture points to the target product. It is an adjustable parameter used to control the degree to which distance affects the score;

[0043] The duration is the continuous length of time the user's gesture is directed at a specific item. The user's historical preferences are derived from the user's historical shopping records, resulting in a user historical preference score. For each item in the candidate item set, its intent matching weight is calculated by combining the scores of the above items. The product with the highest intent matching weight is selected as the final target product.

[0044] Preferably, as a preferred embodiment of the TV quick shopping interaction system based on ultrasonic recognition described in this invention, it includes a shopping interaction module connected to a shopping intent matching module, used to receive target product information determined by the shopping intent matching module, and control the screen display and user interaction of the smart TV to complete the quick shopping operation, specifically including the following:

[0045] Based on the received target product information, retrieve detailed information about the product from the remote product database, including product name, image, price, description, inventory, and purchase link. Control the smart TV screen to display the target product's detailed information via a floating window on the current display interface, clearly informing the user that the identified intent is directed to the product.

[0046] It provides multiple user interaction methods for users to confirm their shopping intentions, including voice confirmation, gesture confirmation, and remote control confirmation. Based on the user's confirmation operation, it guides the user to complete the quick shopping process to complete the transaction.

[0047] On the other hand, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements the functional modules of a TV quick shopping interactive system based on ultrasonic recognition as described above.

[0048] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements a TV quick shopping interactive system based on ultrasonic recognition as described above.

[0049] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0050] By using ultrasonic gesture recognition, users can break free from the constraints of traditional remote controls and engage in non-contact gesture interaction with products on the screen. This greatly enhances the immersive experience for viewers and overcomes the limitations of existing TV shopping interaction methods, such as "clumsy interaction methods, inefficient and vague intent input, and the disconnect between product information and video content." It achieves efficient and clear input of shopping intent. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0052] Figure 1 This is a flowchart of a TV-based quick shopping interactive system based on ultrasonic recognition, according to the present invention.

[0053] Table 1 is a data record table of a simulation experiment of a TV quick shopping interactive system based on ultrasonic recognition according to the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0056] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0057] Example 1

[0058] This embodiment provides, for example Figure 1The present invention relates to a TV quick shopping interactive system based on ultrasonic recognition, which specifically includes a data acquisition module, a gesture recognition module, a product positioning module, a shopping intent matching module, and a shopping interaction module.

[0059] The data acquisition module is used to simultaneously acquire user interaction data and screen content data, and to acquire ultrasonic echo signals and video content data through an ultrasonic sensor array and a video stream acquisition unit.

[0060] The gesture recognition module is connected to the ultrasonic sensor array in the data acquisition module. It processes the ultrasonic echo signal, extracts the first trajectory and position features of the hand in three-dimensional space, and uses a pre-trained deep learning model to recognize the user's predefined shopping pointing gesture.

[0061] The product positioning module is connected to the video stream acquisition unit in the data acquisition module, and is used to analyze the acquired real-time video stream data to identify the purchasable visual elements in the picture and their pixel-level position information on the screen.

[0062] The shopping intent matching module is connected to the gesture recognition module and the product positioning module respectively. It is used to analyze the user's gesture features and the visual element information of the product to determine the user's shopping intent and identify the target product to which the intent is directed.

[0063] The shopping interaction module is connected to the shopping intent matching module and is used to receive the target product information determined by the shopping intent matching module, and control the screen display and user interaction of the smart TV to complete the quick shopping operation.

[0064] In this embodiment, the data acquisition module is specifically described. The data acquisition module is used to simultaneously acquire user interaction data and screen content data, and to acquire ultrasonic echo signals and video content data through an ultrasonic sensor array and a video stream acquisition unit.

[0065] The ultrasonic sensing array consists of at least two ultrasonic transmitters and at least two ultrasonic receivers. The ultrasonic transmitters periodically transmit ultrasonic signals of a specific frequency and waveform to the user interaction area. The specific frequency range is 40kHz-200kHz.

[0066] The ultrasonic receiver synchronously listens to and collects the ultrasonic echo signal reflected from the user's hand, converts the sound wave signal into an electrical signal, and performs pre-amplification and filtering analog processing. Through analog-to-digital conversion, it forms a digital ultrasonic echo data stream.

[0067] The transmitter and receiver, as well as multiple transmitters and receivers, are synchronized using a high-precision clock to ensure accurate measurement of signal propagation time.

[0068] The video stream acquisition unit is used to acquire the video content data currently being played on the smart TV in real time, and further includes:

[0069] Compressed digital video frame data is obtained from the video decoder of the smart TV and transmitted at a set frame rate. The data includes pixel information and timestamps for each video frame. The video frame data acquisition unit synchronizes the acquisition of video frame data with the acquisition of ultrasonic signals to ensure that subsequent modules can associate gestures at a specific moment with video images at the same moment during processing.

[0070] In this embodiment, the gesture recognition module is specifically described. This module is connected to the ultrasonic sensor array in the data acquisition module, processes the ultrasonic echo signal, extracts the first trajectory and position features of the hand in three-dimensional space, and uses a pre-trained deep learning model to recognize the user's predefined shopping pointing gestures. Specifically, this includes the following:

[0071] The raw digitized ultrasonic echo data received from the ultrasonic sensor array is preprocessed, including noise reduction and filtering, and echo signal enhancement. Based on the preprocessed ultrasonic echo signal, an ultrasonic positioning algorithm is used to calculate the real-time position of the user's hand in three-dimensional space to extract the three-dimensional position sequence of the hand. Velocity vector sequence and acceleration vector sequence ;

[0072] Using three-dimensional position sequences, velocity vector sequences, and acceleration vector sequences as input to a deep learning model, the model recognizes user-defined shopping pointing gestures, and further includes:

[0073] At each time step Real-time three-dimensional position of the hand Instantaneous velocity vector Combination with instantaneous acceleration vector This forms a high-dimensional feature vector. ;

[0074] continuous time steps eigenvectors Form a sequence of time-series feature vectors As input to a deep learning model: Where M is the length of the time sequence, representing the time window in which the gesture lasts;

[0075] The time series feature vector sequence Input into the pre-trained deep learning model In this process, feature extraction and pattern recognition are performed on the input sequence. Through nonlinear transformation and inter-layer computation, a probability distribution vector representing different predefined gesture categories is finally output. ,in, , representing the predicted probability for C different gesture categories, and , Let be the probability of recognizing it as the j-th gesture;

[0076] According to the probability distribution vector The category with the highest prediction probability is selected as the final gesture recognition result. When the highest prediction probability Exceeding the preset confidence threshold Then it confirms that the user's predefined shopping pointing gesture has been recognized. The gesture information is then output to subsequent modules for processing.

[0077] The ultrasonic positioning algorithm includes the following steps:

[0078] The measurement involves the time difference between the ultrasonic signal emitted by the transmitter, reflected by the user's hand, and reaching the i-th receiver. Calculate the ultrasonic signal transmitted from the transmitter to the receiver. Total propagation distance Select a reference receiver as Calculate other receivers With the reference receiver The time difference between them is The distance difference is obtained as ,in, This represents the speed at which ultrasound travels through the air. For the launch time, It is the i-th receiver The moment the reflected signal is received;

[0079] The coordinates of the hand's desired position in space are represented as... According to the transmitter coordinates and receiver coordinates Establish the system of hyperbolic equations: ,in, Indicates hand to receiver The distance;

[0080] The hyperbolic equations are linearized, and the least squares method is used to iteratively solve for the three-dimensional coordinates of the hand in space by minimizing the equation residuals. ;

[0081] A receiver array is formed by using multiple closely spaced receivers, and the phase difference between the reflected signals arriving at different receivers is measured. Calculate the azimuth angle of the ultrasonic wave reaching the receiving array. To assist in positioning, among which, d is the ultrasonic wavelength, d is the distance between adjacent receivers, and N represents the number of receivers;

[0082] The three-dimensional location point sequence obtained from continuous timestamps The real-time motion trajectory of the hand is constructed, and the trajectory data is differentiated by first and second order to extract the instantaneous velocity vector sequence of the hand. and acceleration vector sequence .

[0083] In this embodiment, the product positioning module is specifically described. This module is connected to the video stream acquisition unit in the data acquisition module and is used to analyze the acquired real-time video stream data to identify purchasable visual elements in the image and their pixel-level position information on the screen. Specifically, it includes the following:

[0084] The received compressed digital video frame data is decoded to restore the original pixel data, each video frame is acquired, and the video frames are preprocessed, including noise suppression, contrast adjustment, and image enhancement operations, to optimize image quality.

[0085] Using a pre-trained deep learning model for object detection, the pre-processed video frames are analyzed to output the detected visual elements suitable for shopping. The set of shopperable visual elements, wherein the shopperable visual elements are product images displayed on a smart TV screen that guide purchasing behavior; for each identified shopperable visual element, output its bounding box in the current video frame. and corresponding category labels ,in, Indicates the coordinates of the top left corner. It's the width. It is height;

[0086] For each identified shopperable visual element Through a pre-trained image segmentation model Extract its pixel-level mask The pixel-level mask It is a binary image that describes the precise outline and pixel-occupied area of ​​a shopperable visual element on the screen, where a pixel value of 1 indicates that it belongs to the element and 0 indicates that it does not.

[0087] The segmentation process is represented as follows: ;in, For input video frames, For the detected bounding box, For the corresponding category label, This indicates that the image segmentation model classifies pixels within a specified region and generates a binary mask;

[0088] Based on the pixel-level mask The centroid coordinates of the element on the screen are calculated as follows: , ,in, These are the coordinates of the centroid in the image pixel coordinate system;

[0089] Temporal tracking is performed on shoppable visual elements identified in consecutive video frames to ensure the correlation of the same element across different frames. A multi-object tracking algorithm is used to associate the identified bounding boxes and pixel-level masks, assigning a unique ID to each shoppable visual element. The output is a list of all shoppable visual elements identified and tracked at each time stamp, including bounding boxes. Pixel mask Centroid coordinates Category tags And tracking ID.

[0090] In this embodiment, the shopping intent matching module is specifically described. This module is connected to both the gesture recognition module and the product location module. It analyzes the user's gesture features and the visual element information of the product to determine the user's shopping intent and identify the target product. Specifically, it includes the following:

[0091] When the gesture recognition module detects a pointing gesture indicating a user's shopping intention, it calculates the candidate region mapped onto the product location map on the screen based on the hand's three-dimensional position in space and the gesture's direction vector. The gesture direction vector is obtained from the instantaneous velocity vector of the hand, and further includes:

[0092] The recognition result output by the gesture recognition module includes the instantaneous velocity vector of the hand when the pointing gesture is detected. 3D position of the hand and recognized shopping pointing gestures The three-dimensional position of the user's hand when a pointing gesture is detected. With gesture direction vector Form a ray ,in, These are parameters of the ray, representing the distance along the direction of the velocity vector;

[0093] Calculate the intersection point between ray L and the screen plane. If the ray does not intersect the screen plane, it is considered an invalid direction, and subsequent processing stops. If a valid intersection point exists, convert the intersection point from 3D spatial coordinates to 2D pixel coordinates on the screen. A two-dimensional circular candidate region with radius r is generated, centered on the two-dimensional pixel coordinates of the screen. ;

[0094] Spatially match the candidate region with the boundary information of all purchasable items in the screen product location map, and filter out all items that match the candidate region. Goods that have spatial intersection form a candidate goods set. The system prioritizes gestures based on their accuracy, duration, and user history preferences to determine the target product the user intends to access.

[0095] The accuracy of the gesture is defined as the distance between the center of the gesture and the center of mass of the product. By converting it into a precision score This indicates the precision with which the user's gesture points to the target product. It is an adjustable parameter used to control the degree to which distance affects the score;

[0096] The duration is the continuous length of time the user's gesture is directed at a specific item. The user's historical preferences are derived from the user's historical shopping records, resulting in a user historical preference score. For each item in the candidate item set, its intent matching weight is calculated by combining the scores of the above items. The product with the highest intent matching weight is selected as the final target product.

[0097] In this embodiment, the shopping interaction module is specifically described. This module is connected to the shopping intent matching module and is used to receive target product information determined by the shopping intent matching module, and control the screen display and user interaction of the smart TV to complete a quick shopping operation. Specifically, it includes the following:

[0098] Based on the received target product information, retrieve detailed information about the product from the remote product database, including product name, image, price, description, inventory, and purchase link. Control the smart TV screen to display the target product's detailed information via a floating window on the current display interface, clearly informing the user that the identified intent is directed to the product.

[0099] It provides multiple user interaction methods for users to confirm their shopping intentions, including voice confirmation, gesture confirmation, and remote control confirmation. Based on the user's confirmation operation, it guides the user to complete the quick shopping process to complete the transaction.

[0100] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0101] This embodiment also provides a storage medium storing a computer program. When executed by a processor, the program implements the functional modules of a TV-based quick shopping interactive system based on ultrasonic recognition as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0102] Example 2

[0103] The following is another embodiment of the present invention, which provides a TV quick shopping interactive system based on ultrasonic recognition. In order to verify the beneficial effects of the present invention, a simulation experiment is conducted for scientific demonstration.

[0104] This experiment aims to verify the effectiveness of a TV-based quick shopping interaction system based on ultrasonic recognition. By simultaneously collecting user interaction data and screen content data, using gesture recognition, product positioning, shopping intent matching, and other technical means, the system enhances the user's quick shopping experience on smart TVs, achieves an intuitive and natural interaction method, and accurately identifies the user's shopping intent.

[0105] The simulation experiment steps are implemented according to the content of the TV quick shopping interactive system based on ultrasonic recognition provided in Example 1, and the specific steps include:

[0106] The ultrasonic echo signal of the user's hand in three-dimensional space is synchronously collected by an ultrasonic sensor array. The collection frequency is set to 100 frames per second, and the specific transmission frequency range is 40kHz-200kHz.

[0107] The video stream acquisition unit synchronously acquires real-time video stream data from the smart TV screen, with the resolution set to 1920x1080 pixels and the frame rate set to 30 frames per second.

[0108] The ultrasonic echo signal is processed, and the three-dimensional motion trajectory of the hand is reconstructed by a time difference of arrival and phase difference fusion algorithm. A pre-trained deep learning model is used to identify the user's predefined shopping pointing gesture.

[0109] The video stream data is analyzed in real time to identify all purchasable visual elements in the scene. The pixel-level mask is extracted by the instance segmentation model, and the bounding box, pixel-level mask, centroid coordinates, category label and tracking ID of each element are output. The screen product location map is updated to construct the location information of all purchasable visual elements.

[0110] The system calculates a two-dimensional circular candidate region that the gesture maps to on the screen product location map. It then performs spatial matching between the candidate region and the boundary information of all purchasable products on the screen product location map, filters out all products that have spatial intersection with the candidate region, forms a candidate product set, and determines the final target product by calculating the intent matching weight of each candidate product.

[0111] Control the smart TV screen to generate a semi-transparent floating shopping panel overlaid on the video screen. Users can select and complete quick shopping operations by confirming with gestures, voice, or remote control.

[0112] The specific data from the above simulation experiment are as follows:

[0113] Time / second Gesture type Three-dimensional position of the hand (mm) Instantaneous velocity vector (mm / s) Target product (ID) on the screen Candidate region (x, y, r) Match product set (ID) Priority sorting (ID, Score) Target Product (ID) Interaction response time (ms) Success rate 0-1 No gestures (N / A) (N / A) N / A N / A N / A N / A N / A N / A N / A 1-2 Pointing gesture (100,50,500) (10,5,-20) T-Shirt_01 (960,540,50) T-Shirt_01 (T-Shirt_01, 0.92) T-Shirt_01 500 100% 2-3 Keep pointing (102,52,495) (5,2,-10) T-Shirt_01 (965,542,45) T-Shirt_01 (T-Shirt_01, 0.95) T-Shirt_01 N / A N / A 3-4 Clenching fist to confirm (105,55,490) (0,0,0) T-Shirt_01 N / A N / A N / A T-Shirt_01 300 100% 4-5 Pointing gesture (80,60,520) (8,3,-15) Mug_02 (300,700,60) Mug_02,Poster_03 (Mug_02, 0.88;Poster_03, 0.70) Mug_02 450 95% 5-6 Deviation from direction (75,62,510) (2,-1,-5) N / A (310,710,70) Mug_02,Poster_03 (Mug_02, 0.65; Poster_03, 0.50) N / A N / A 80% 6-7 Cancel voice command (N / A) (N / A) N / A N / A N / A N / A N / A 200 100% 7-8 Pointing gesture (120,40,480) (12,-3,-25) Book_04 (1500,200,40) Book_04, Pen_05 (Book_04, 0.90; Pen_05, 0.75) Book_04 400 98%

[0114] Table 1

[0115] Experimental Analysis:

[0116] By comparing the system's gesture recognition results with simulated real user intent, the matching results of target products with preset target products, and conducting statistical analysis of user interaction response time and success rate, the accuracy and robustness of the system in recognizing and predicting user shopping intent are verified.

[0117] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A TV-based quick shopping interactive system based on ultrasonic recognition, characterized in that: It includes a data acquisition module, a gesture recognition module, a product location module, a shopping intent matching module, and a shopping interaction module; The data acquisition module is used to simultaneously acquire user interaction data and screen content data, and to acquire ultrasonic echo signals and video content data through an ultrasonic sensor array and a video stream acquisition unit. The gesture recognition module is connected to the ultrasonic sensor array in the data acquisition module. It processes the ultrasonic echo signal, extracts the first trajectory and position features of the hand in three-dimensional space, and uses a pre-trained deep learning model to recognize the user's predefined shopping pointing gesture. The product positioning module is connected to the video stream acquisition unit in the data acquisition module, and is used to analyze the acquired real-time video stream data to identify the purchasable visual elements in the picture and their pixel-level position information on the screen. The shopping intent matching module is connected to the gesture recognition module and the product positioning module respectively. It is used to analyze the user's gesture features and the visual element information of the product to determine the user's shopping intent and identify the target product to which the intent is directed. The shopping interaction module is connected to the shopping intent matching module and is used to receive the target product information determined by the shopping intent matching module, and control the screen display and user interaction of the smart TV to complete the quick shopping operation.

2. The TV quick shopping interactive system based on ultrasonic recognition according to claim 1, characterized in that: The data acquisition module is used to simultaneously acquire user interaction data and screen content data, and to acquire ultrasonic echo signals and video content data through an ultrasonic sensor array and a video stream acquisition unit. The ultrasonic sensing array consists of at least two ultrasonic transmitters and at least two ultrasonic receivers, wherein the ultrasonic transmitters periodically transmit ultrasonic signals of a specific frequency and waveform to the user interaction area. The ultrasonic receiver synchronously listens to and collects the ultrasonic echo signal reflected from the user's hand, converts the sound wave signal into an electrical signal, and performs pre-amplification and filtering analog processing. Through analog-to-digital conversion, it forms a digital ultrasonic echo data stream. The transmitter and receiver, as well as multiple transmitters and receivers, are synchronized using a high-precision clock to ensure accurate measurement of signal propagation time. The video stream acquisition unit is used to acquire the video content data currently being played on the smart TV in real time, and further includes: Compressed digital video frame data is obtained from the video decoder of the smart TV and transmitted at a set frame rate. The data includes pixel information and timestamps for each video frame. The video frame data acquisition unit synchronizes the acquisition of video frame data with the acquisition of ultrasonic signals to ensure that subsequent modules can associate gestures at a specific moment with video images at the same moment during processing.

3. The TV quick shopping interactive system based on ultrasonic recognition according to claim 1, characterized in that: The gesture recognition module is connected to the ultrasonic sensor array in the data acquisition module. It processes the ultrasonic echo signal, extracts the first trajectory and position features of the hand in three-dimensional space, and uses a pre-trained deep learning model to recognize the user's predefined shopping pointing gestures, specifically including the following: The raw digitized ultrasonic echo data received from the ultrasonic sensor array is preprocessed, including noise reduction and filtering, and echo signal enhancement. Based on the preprocessed ultrasonic echo signal, an ultrasonic positioning algorithm is used to calculate the real-time position of the user's hand in three-dimensional space to extract the three-dimensional position sequence of the hand. Velocity vector sequence and acceleration vector sequence ; Using three-dimensional position sequences, velocity vector sequences, and acceleration vector sequences as input to a deep learning model, the model recognizes user-defined shopping pointing gestures, and further includes: At each time step Real-time three-dimensional position of the hand Instantaneous velocity vector Combination with instantaneous acceleration vector This forms a high-dimensional feature vector. ; continuous time steps eigenvectors Form a sequence of time-series feature vectors As input to a deep learning model: Where M is the length of the time sequence, representing the time window in which the gesture lasts; The time series feature vector sequence Input to pre-trained deep learning model In this process, feature extraction and pattern recognition are performed on the input sequence. Through nonlinear transformation and inter-layer computation, a probability distribution vector representing different predefined gesture categories is finally output. ,in, , representing the predicted probability for C different gesture categories, and , Let be the probability of recognizing it as the j-th gesture; According to the probability distribution vector The category with the highest prediction probability is selected as the final gesture recognition result. When the highest prediction probability Exceeding the preset confidence threshold Then it confirms that the user's predefined shopping pointing gesture has been recognized. The gesture information is then output to subsequent modules for processing.

4. The TV quick shopping interactive system based on ultrasonic recognition according to claim 3, characterized in that: The ultrasonic positioning algorithm includes the following steps: The measurement involves the time difference between the ultrasonic signal emitted by the transmitter, reflected by the user's hand, and reaching the i-th receiver. Calculate the ultrasonic signal transmitted from the transmitter to the receiver. Total propagation distance Select a reference receiver as Calculate other receivers With the reference receiver The time difference between them is The distance difference is obtained as ,in, This represents the speed at which ultrasound travels through the air. For the launch time, It is the i-th receiver The moment the reflected signal is received; The coordinates of the hand's desired position in space are represented as... According to the transmitter coordinates and receiver coordinates Establish the system of hyperbolic equations: ,in, Indicates hand to receiver The distance; The hyperbolic equations are linearized, and the least squares method is used to iteratively solve for the three-dimensional coordinates of the hand in space by minimizing the equation residuals. ; A receiver array is formed by using multiple closely spaced receivers, and the phase difference between the reflected signals arriving at different receivers is measured. Calculate the azimuth angle of the ultrasonic wave reaching the receiving array. To assist in positioning, among which, d is the ultrasonic wavelength, d is the distance between adjacent receivers, and N represents the number of receivers; The three-dimensional location point sequence obtained from continuous timestamps The real-time motion trajectory of the hand is constructed, and the trajectory data is differentiated by first and second order to extract the instantaneous velocity vector sequence of the hand. and acceleration vector sequence .

5. The TV quick shopping interactive system based on ultrasonic recognition according to claim 1, characterized in that: The product positioning module is connected to the video stream acquisition unit in the data acquisition module, and is used to analyze the acquired real-time video stream data to identify purchasable visual elements in the image and their pixel-level position information on the screen, specifically including the following: The received compressed digital video frame data is decoded to restore the original pixel data, each video frame is acquired, and the video frames are preprocessed, including noise suppression, contrast adjustment, and image enhancement operations, to optimize image quality. Using a pre-trained deep learning model for object detection, the pre-processed video frames are analyzed to output the detected visual elements suitable for shopping. The set; for each identified shopperable visual element, output its bounding box in the current video frame. and corresponding category labels ,in, Indicates the coordinates of the top left corner. It's the width. It is height; For each identified shopperable visual element Through a pre-trained image segmentation model Extract its pixel-level mask The pixel-level mask This is a binary image, where a pixel value of 1 indicates that it belongs to the element, and 0 indicates that it does not belong to the element. The segmentation process is represented as follows: ;in, For input video frames, For the detected bounding box, For the corresponding category label, This indicates that the image segmentation model classifies pixels within a specified region and generates a binary mask; Based on the pixel-level mask The centroid coordinates of the element on the screen are calculated as follows: , ,in, These are the coordinates of the centroid in the image pixel coordinate system; The system performs temporal tracking of shoppable visual elements identified in consecutive video frames to ensure the correlation of the same element across different frames. It also associates the identified bounding boxes with pixel-level masks, assigns a unique ID to each shoppable visual element, and outputs a list of all shoppable visual elements identified and tracked at each time stamp, including bounding boxes. Pixel mask Centroid coordinates Category tags And tracking ID.

6. The TV quick shopping interactive system based on ultrasonic recognition according to claim 1, characterized in that: The shopping intent matching module is connected to both the gesture recognition module and the product positioning module. It is used to analyze the user's gesture features and the visual element information of the product to determine the user's shopping intent and identify the target product. Specifically, it includes the following: When the gesture recognition module detects a pointing gesture indicating a user's shopping intention, it calculates the candidate region mapped onto the product location map on the screen based on the hand's three-dimensional position in space and the gesture's direction vector. ; Spatially match the candidate region with the boundary information of all purchasable items in the screen product location map, and filter out all items that match the candidate region. Goods that have spatial intersection form a candidate goods set. The system prioritizes gestures based on their accuracy, duration, and user history preferences to determine the target product the user intends to access. The accuracy of the gesture is defined as the distance between the center of the gesture and the center of mass of the product. By converting it into a precision score This indicates the precision with which the user's gesture points to the target product. It is an adjustable parameter used to control the degree to which distance affects the score; The duration is the continuous length of time the user's gesture is directed at a specific item. The user's historical preferences are derived from the user's historical shopping records, resulting in a user historical preference score. For each item in the candidate item set, its intent matching weight is calculated by combining the scores of the above items. The product with the highest intent matching weight is selected as the final target product.

7. A TV-based quick shopping interactive system based on ultrasonic recognition according to claim 6, characterized in that: The gesture direction vector is obtained from the instantaneous velocity vector of the hand, and the candidate region mapped by the gesture on the screen product location map is calculated. Further including: The recognition result output by the gesture recognition module includes the instantaneous velocity vector of the hand when the pointing gesture is detected. 3D position of the hand and recognized shopping pointing gestures The three-dimensional position of the user's hand when a pointing gesture is detected. With gesture direction vector Form a ray ,in, These are parameters of the ray, representing the distance along the direction of the velocity vector; Calculate the intersection point between ray L and the screen plane. If the ray does not intersect the screen plane, it is considered an invalid direction, and subsequent processing stops. If a valid intersection point exists, convert the intersection point from 3D spatial coordinates to 2D pixel coordinates on the screen. A two-dimensional circular candidate region with radius r is generated, centered on the two-dimensional pixel coordinates of the screen. .

8. A TV-based quick shopping interactive system based on ultrasonic recognition according to claim 1, characterized in that: The shopping interaction module is connected to the shopping intent matching module and is used to receive the target product information determined by the shopping intent matching module, and control the screen display and user interaction of the smart TV to complete the quick shopping operation, specifically including the following: Based on the received target product information, retrieve detailed information about the product from the remote product database, including product name, image, price, description, inventory, and purchase link. Control the smart TV screen to display the target product's detailed information via a floating window on the current display interface, clearly informing the user that the identified intent is directed to the product. It provides multiple user interaction methods for users to confirm their shopping intentions, including voice confirmation, gesture confirmation, and remote control confirmation. Based on the user's confirmation operation, it guides the user to complete a quick shopping process to complete the transaction.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the functional modules of the TV quick shopping interactive system based on ultrasonic recognition as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the functional modules of the TV quick shopping interactive system based on ultrasonic recognition as described in any one of claims 1-8.

Images