A digital stall interaction system and method based on dual-modal video communication
The dual-modal video communication system enables accurate positioning and image extraction of goods at the stall, solving the problems of small product occupancy, occlusion, and unclear boundaries in existing technologies, and improving the accuracy of selection and display.
Patent Information
- Application Number
- CN202611358796.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-03
- Publication Date
- 2026-11-17
AI Technical Summary
In existing stall interaction methods, the products occupy a small area in the video frame, are obscured, or have unclear boundaries, which leads to deviations in the client's selection position. Furthermore, fixed areas are difficult to adapt to changes in product position, and the correspondence between video frames is unclear, affecting the accuracy of selection.
The system employs dual-modal video communication, using a first-modal video stream to capture the overall image of the stall and a second-modal video stream to present local details. It performs time synchronization and coordinate mapping to identify the product location area, and reconstructs the product boundary through pixel-by-pixel scanning, extracting and labeling the target product image.
It improves the accuracy and consistency of product selection, reduces the difficulty of selection in densely packed display scenarios, and ensures the accuracy and synchronization of product location and image extraction.
Smart Images

Figure CN123123601A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically, to a digital stall interaction system and method based on dual-modal video communication. Background Technology
[0002] With the development of digital retail, mobile interaction, and video communication technologies, stall operation scenarios are increasingly adopting video capture devices to provide product display information to clients. Current stall interaction methods typically use a single camera to capture the stall's view; the client selects products from the displayed video, and the backend completes product recognition and information display based on pre-configured product areas.
[0003] However, in actual stall environments, goods are typically numerous, densely arranged, and vary significantly in size. When using a large field of view to capture the overall scene of the stall, the proportion of a single product in the image is relatively small, and products are prone to overlapping or having unclear boundaries, leading to positional errors when customers select products. Furthermore, the positions of stall goods may change during placement, while pre-configured product areas are usually fixed and difficult to adapt to changes in the actual display status of the goods.
[0004] On the other hand, product areas in video footage are typically marked and cropped using rectangles or fixed cropping areas. When the product's outline is not a regular rectangle or the spacing between adjacent products is small, the fixed area can easily include both the surrounding background and adjacent product content, resulting in inaccurate product image boundaries. Furthermore, during video communication, there may be issues such as time asynchrony and unclear frame correspondence between different video frames, making it difficult to maintain an accurate correspondence between the product position selected by the client and the actual displayed product image.
[0005] In view of this, the present invention proposes a digital stall interaction system and method based on dual-modal video communication to solve the above problems. Summary of the Invention
[0006] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution: a digital stall interaction system based on dual-modal video communication, comprising: The acquisition and calibration module is used to acquire the first and second modal video streams of the stall and synchronize them in time, identify the product location area, calibrate the mapping relationship between the two modal coordinate systems, and output the first modal image and the second modal image; The interactive positioning module is used to receive the interactive coordinates of the client for the first modal screen, compare them with the product location area, determine the target product selected by the client and the first modal candidate boundary, and obtain the second modal candidate area by combining the mapping relationship. The boundary reconstruction module is used to perform pixel-by-pixel scanning along several preset directions starting from the center of the second modality candidate region, calculate the difference value between adjacent pixels, determine the pixels whose difference value exceeds the difference threshold as boundary crossing points, and connect them sequentially to form the second modality target boundary. The image extraction module is used to extract the target product image from the second modal screen based on the second modal target boundary, and to inversely map the second modal target boundary to obtain the first modal target boundary, thereby forming target interaction information; The synchronous annotation module is used to annotate the target products in the first modal screen and the second modal screen according to the target interaction information, and output the annotation results and the target product images to the client.
[0007] Preferably, the product location identification area includes: A first video acquisition device and a second video acquisition device are configured at the digital stall to collect video information of different modalities; a first modal video stream covering the stall display area and a second modal video stream presenting local details of the stall display area are collected respectively, and the acquisition time of each video frame is recorded; The first and second modal video frames corresponding to the acquisition time are selected according to the preset synchronization time interval. When the acquisition time difference between the two video frames does not exceed the preset acquisition time difference threshold, the two video frames are established to form a time correspondence and a dual-modal synchronization frame sequence is formed. For the first modal video frame in the dual-modal synchronous frame sequence, the product location area is determined based on the image differences between the product and the surrounding area and the regional boundaries between adjacent products, and a product number is associated with each product location area.
[0008] Preferably, the output of the first modal image and the second modal image includes: Several calibration positions are set in the common field of view of the first modal video stream and the second modal video stream. The image coordinates of each calibration position in the first modal video frame and the second modal video frame are determined respectively, and a correspondence is established between the two image coordinates corresponding to the same calibration position. The coordinate mapping relationship between the first mode coordinate system and the second mode coordinate system is determined based on multiple sets of corresponding image coordinates, and the coordinate mapping relationship is saved. Select a first-mode video frame and a second-mode video frame with the same time correspondence from the dual-modal synchronous frame sequence. Use the selected first-mode video frame as the first-mode picture and the selected second-mode video frame as the second-mode picture, and output them after associating them with the same time marker.
[0009] Preferably, obtaining the second modality candidate region includes: Receive the click or touch position of the client in the first modal screen, convert the click or touch position into interactive coordinates in the coordinate system of the first modal screen, and associate it with the time stamp of the first modal screen; The interaction coordinates are sequentially determined to be inside the closed boundary of each product location area. When the interaction coordinates are inside one of the product location areas, the product number corresponding to that product location area is determined as the target product number, and the closed boundary of that product location area is determined as the first modality candidate boundary. According to the coordinate mapping relationship corresponding to the time identifier, the boundary positions on the first modality candidate boundary are transformed point by point to the second modality coordinate system. The second modality candidate region is formed based on the transformed boundary positions, and the second modality candidate region and the target product number are output as target positioning information.
[0010] Preferably, calculating the difference between adjacent pixels includes: Using the center position of the second modality candidate region as the scanning starting point, the circumference angle is divided equally according to the number of preset scanning directions, and scanning is performed pixel by pixel outward along each scanning direction from the scanning starting point to establish a scanning path. The center position is obtained by calculating the average coordinates of each vertex of the second modality candidate region boundary; each scanning direction is set sequentially from zero degrees with the same angular interval, and the angular interval is obtained by dividing the full circle angle by the preset number of scanning directions; Along the scanning path in each scanning direction, the pixel values of adjacent scanning positions are obtained sequentially at preset pixel intervals, and the pixel difference value between adjacent scanning positions is calculated. The pixel difference value is the absolute value of the difference between the pixel values of adjacent scanning positions. Starting from the scanning start point, the pixel difference value of each position is calculated and recorded sequentially in the scanning direction.
[0011] Preferably, forming the second modal target boundary includes: Along each scanning direction, starting from the scanning start point and moving outwards, the pixel difference value corresponding to each scanning position is compared with a preset difference threshold. The scanning position that first exceeds the preset difference threshold in each scanning direction is determined as the corresponding boundary crossing position. Substitute the boundary crossing positions corresponding to each scanning direction into the corresponding scanning path to calculate the pixel coordinates of the boundary crossing positions in the current video frame of the second mode, and determine the pixel coordinates as the corresponding boundary crossing points; connect the boundary crossing points in the circumferential order of each scanning direction, and connect the first boundary crossing point with the last boundary crossing point to form a closed boundary, thus obtaining the target boundary of the second mode.
[0012] Preferably, the extraction of the target product image includes: The closed area enclosed by the second modal target boundary is determined. Pixels in the second modal image that are located within the closed area are identified as target product pixels and retained. Pixels located outside the closed area are removed to obtain the target product image corresponding to the second modal target boundary.
[0013] Preferably, the formation of target interaction information includes: Obtain each boundary point on the second modal target boundary, and according to the arrangement order of each boundary point in the second modal target boundary, perform an inverse transformation through the coordinate mapping relationship between the first modal coordinate system and the second modal coordinate system to obtain the first modal pixel coordinates corresponding to each boundary point; Connect the pixel coordinates of each first modality in the order of arrangement to form the first modality target boundary; associate the first modality target boundary, the second modality target boundary, the target product number and the target product image with the corresponding time markers to form target interactive information.
[0014] Preferably, outputting the annotation results and the target product image to the client includes: Read the first modal target boundary, second modal target boundary, target product number and target product image from the target interaction information, and determine the corresponding first modal screen and second modal screen according to the time identifier corresponding to the target interaction information; According to the arrangement order of each boundary point in the first modal target boundary, each boundary point is connected sequentially in the first modal screen and the first and last boundary points are connected to form the first modal annotation boundary to identify the target product in the first modal screen; According to the arrangement order of each boundary point in the second modal target boundary, each boundary point is connected sequentially in the second modal screen and the first and last boundary points are connected to form the second modal annotation boundary to identify the target product in the second modal screen, and the target product number is displayed at the second modal annotation boundary; The first modal image, the second modal image, and the target product image, after being annotated, are associated with their corresponding time markers to form interactive output data, which is then output to the client.
[0015] A digital stall interaction method based on dual-modal video communication includes: S1. Collect the first and second modal video streams of the stall and synchronize them in time, identify the product location area, mark the mapping relationship between the two modal coordinate systems, and output the first modal image and the second modal image; S2. Receive the interaction coordinates of the client for the first modal screen, compare them with the product location area, determine the target product selected by the client and the first modal candidate boundary, and obtain the second modal candidate area by combining the mapping relationship; S3. Starting from the center of the candidate region of the second modality, perform a pixel-by-pixel scan along several preset directions, calculate the difference value between adjacent pixels, determine the pixels whose difference value exceeds the difference threshold as boundary crossing points, and connect them in sequence to form the target boundary of the second modality. S4. Extract the target product image from the second modal screen based on the second modal target boundary, and inversely map the second modal target boundary to obtain the first modal target boundary to form target interaction information; S5. Based on the target interaction information, label the target products in the first modal screen and the second modal screen, and output the labeling results and the target product images to the client.
[0016] Compared with the prior art, the present invention has the following beneficial effects: The system acquires an overall view of the stall's display area using a first-modal video stream, while presenting detailed product information via a second-modal video stream. Simultaneously, the video frames from both modalities are time-synchronized and a coordinate mapping relationship is established, enabling a correspondence between product positions in different modal views and improving the accuracy of the correspondence between the client's product selection and the specific product image. By comparing the client's interactive coordinates in the first-modal view with the product location area, the target product is determined. The first-modal candidate boundary is then transformed to the second-modal coordinate system to form a second-modal candidate region. This allows the client to obtain the corresponding specific product area without directly selecting products from the detailed images, reducing the difficulty of selection in densely packed product display scenarios.
[0017] This invention uses the center of the second modality candidate region as the scanning starting point, performs pixel scanning along multiple directions, and determines the boundary crossing position based on the scanning position that first exceeds a preset difference threshold in each scanning direction. Then, a second modality target boundary is formed through circumferential connection, allowing the product boundary to be reconstructed based on pixel changes in the current video frame. This reduces the restriction of fixed rectangular areas on the actual contour of the product and improves the accuracy of target product region determination. The target product image is extracted based on the second modality target boundary, and the reconstructed second modality target boundary is inversely mapped to obtain the first modality target boundary. This is then combined with the target product number and time stamp to form target interaction information, ensuring that the same target product maintains a corresponding position in both modal frames. The target product in both the first and second modal frames is labeled using the target interaction information, and the two labeled frames are associated with the target product image using the same time stamp before being output to the client. This allows the client to simultaneously obtain the target product's position, local details, and corresponding product image within the overall stall frame, thereby improving the consistency of product positioning, image extraction, and result display during digital stall video interaction. Attached Figure Description
[0018] Figure 1This is a schematic diagram of the structure of a digital stall interaction system based on dual-modal video communication according to the present invention; Figure 2 This is a schematic diagram of a digital stall interaction method based on dual-modal video communication according to the present invention. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example
[0020] Please see Figure 1 As shown, this embodiment provides a digital stall interaction system based on dual-modal video communication, specifically including the following steps: The acquisition and calibration module is used to acquire the first and second modal video streams of the stall and synchronize them in time, identify the product location area, calibrate the mapping relationship between the two modal coordinate systems, and output the first modal image and the second modal image; The interactive positioning module is used to receive the interactive coordinates of the client for the first modal screen, compare them with the product location area, determine the target product selected by the client and the first modal candidate boundary, and obtain the second modal candidate area by combining the mapping relationship. The boundary reconstruction module is used to perform pixel-by-pixel scanning along several preset directions starting from the center of the second modality candidate region, calculate the difference value between adjacent pixels, determine the pixels whose difference value exceeds the difference threshold as boundary crossing points, and connect them sequentially to form the second modality target boundary. The image extraction module is used to extract the target product image from the second modal screen based on the second modal target boundary, and to inversely map the second modal target boundary to obtain the first modal target boundary, thereby forming target interaction information; The synchronous annotation module is used to annotate the target products in the first and second modal screens based on the target interaction information, and output the annotation results and target product images to the client. The various modules are connected via wired and / or wireless means to achieve data transmission between them.
[0021] Identifying the product location area includes: A first video acquisition device and a second video acquisition device are configured at the digital stall to collect video information of different modalities; a first modal video stream covering the stall display area and a second modal video stream presenting local details of the stall display area are collected respectively, and the acquisition time of each video frame is recorded; In this embodiment, it should be noted that a first video acquisition device and a second video acquisition device are configured at the digital stall. Both the first video acquisition device and the second video acquisition device can be digital cameras, network cameras, or video terminals with video acquisition functions in the prior art. The first video acquisition device is used to acquire a first modal video covering the entire display area of the stall, and the second video acquisition device is used to acquire detailed videos of local products in the display area of the stall.
[0022] The first video capture device is positioned to cover the entire merchandise display area, ensuring that its captured image includes the areas where all merchandise is located within the stall. The second video capture device is positioned to capture localized details of the merchandise, ensuring that its captured image includes a portion of the merchandise area covered by the first video capture device, and that this portion shares a common field of view with the first video capture device's image. A common field of view means that the same merchandise or merchandise area can appear in both the images captured by the first and second video capture devices simultaneously. This configuration allows the first modal video to reflect the positional relationship of the merchandise within the overall stall display area, while the second modal video provides localized image information of the corresponding merchandise area. Both the first and second video capture devices continuously capture video, recording the capture time for each captured frame using a unified time reference.
[0023] For example, for a stall display area with a width of 3m and a height of 2m, the first video capture device is positioned directly in front of the stall, ensuring its view completely encompasses the 3m x 2m product display area; the second video capture device is positioned to the front right of the display area, covering approximately 1m x 1m of the product area on the right side, and this 1m x 1m area is also within the view of the first video capture device. In this case, the first video capture device captures the overall product distribution of the stall, while the second video capture device captures detailed images of specific products within that area.
[0024] The first and second modal video frames corresponding to the acquisition time are selected according to the preset synchronization time interval. When the acquisition time difference between the two video frames does not exceed the preset acquisition time difference threshold, the two video frames are established to form a time correspondence and a dual-modal synchronization frame sequence is formed. It should be noted that the preset synchronization time interval is used to specify the time interval for selecting video frames from the first and second modal video streams to establish a time correspondence. This interval can be set according to the client's requirements for real-time video interaction and the video frame rate. For example, if both the first and second modal video streams have a frame rate of 30 frames per second, corresponding to a single frame time interval of approximately 33.3 ms, the preset synchronization time interval can be set to 100 ms, meaning that video frames from the two modal video streams corresponding to the same time period are selected for time matching every 100 ms.
[0025] The preset acquisition time difference threshold is used to limit the maximum allowable time deviation between the first modality video frame and the second modality video frame. It can be determined based on the frame rates of the first and second video acquisition devices. For example, when both video acquisition devices have a frame rate of 30 frames per second, the time interval between adjacent video frames is approximately 33.33 ms. If the expected positional change of the stall goods between adjacent frames does not exceed the time error allowed by the detection of a single product boundary, the acquisition time difference threshold can be set to 10 ms. In this case, if the acquisition time of the first modality video frame is 10:00:00.100 and the acquisition time of the second modality video frame is 10:00:00.108, the difference is 8 ms, which is less than 10 ms, thus establishing a time correspondence. If the acquisition time of the second modality video frame is 10:00:00.115, the time difference is 15 ms, which exceeds 10 ms, and this correspondence is not established.
[0026] For the first modal video frame in the dual-modal synchronous frame sequence, the product location area is determined based on the image differences between the product and the surrounding area and the regional boundaries between adjacent products, and a product number is associated with each product location area.
[0027] In this embodiment, for the first modal video frame in the dual-modal synchronous frame sequence, the area where the product is located is determined within a preset stall display area. Image information is then compared point-by-point along adjacent positions of the product area and its surrounding areas to calculate the image brightness difference between adjacent positions. When the image brightness difference exceeds a preset regional difference threshold, it indicates a significant image change at the corresponding position, and this position is determined as a candidate external boundary between the product and its surrounding area. Further, for adjacent products that are in contact with each other or spaced less than a preset interval, image information is compared point-by-point along adjacent positions between the adjacent products to calculate the image brightness difference between adjacent positions. When the image brightness difference at multiple consecutive adjacent positions exceeds the preset regional difference threshold, the corresponding consecutive positions are determined as the regional boundary between adjacent products to distinguish the image areas corresponding to adjacent products.
[0028] The candidate external boundaries between the product and its surrounding area, as well as the regional boundaries between adjacent products, are connected according to spatial continuity. When the connected boundaries form a closed contour, the continuous image region inside the closed contour is determined as the product location region, and a corresponding product number is associated with each product location region, thereby establishing a correspondence between product numbers and product location regions.
[0029] For example, in the first modal video frame, if the brightness values of a product and the adjacent positions of the stall background are 185 and 132 respectively, the brightness difference between the adjacent positions is 53. When the preset regional difference threshold is set to 40, since 53 exceeds 40, this position is considered a candidate outer boundary between the product and the stall background. For two adjacent products, if the brightness differences of consecutive positions in their contact area are 45, 51, 48, and 56 respectively, all exceeding 40, then these consecutive positions are determined as the regional boundary between the two products. After connecting the candidate outer boundary with the regional boundary to form a closed contour, the area inside this closed contour is determined as the location area of the corresponding product. For example, the area with coordinates ranging from 120 to 260 (horizontal coordinate) and 80 to 300 (vertical coordinate) is determined as the product location area and associated with the product number SP001.
[0030] It should be noted that the preset regional difference threshold is used to distinguish between significant brightness changes at product boundaries and normal brightness fluctuations within the product. This threshold can be determined by statistically analyzing the brightness differences between adjacent positions in a normal product display. For example, if multiple frames of a normal display are captured and the maximum brightness difference between adjacent positions within a product is found to be 25, the preset regional difference threshold can be set to 40, making it higher than the normal range of internal brightness fluctuations. Subsequently, when the brightness difference between adjacent positions exceeds 40, that position is then considered a candidate boundary. This prevents normal brightness changes within the product from being misjudged as regional boundaries.
[0031] The output of the first modal image and the second modal image includes: Several calibration positions are set in the common field of view of the first modal video stream and the second modal video stream. The image coordinates of each calibration position in the first modal video frame and the second modal video frame are determined respectively, and a correspondence is established between the two image coordinates corresponding to the same calibration position. In this embodiment, it should be noted that the common field of view refers to the stall display area that can be captured by both the first and second video acquisition devices. Since the first video acquisition device is used to cover the entire stall display area, and the second video acquisition device is used to present local product details, the fields of view of the two video acquisition devices partially overlap. This overlapping area is defined as the common field of view. Only positions within the common field of view that can be identified in both modal images are selected as calibration positions. Specifically, several fixed calibration positions are set within the common field of view. The positions of each calibration position in the actual space of the stall remain unchanged, and the calibration positions are dispersed across different locations within the common field of view. First and second modal video frames containing the calibration positions are acquired respectively. The image coordinates of each calibration position are determined in the first modal video frame, and the image coordinates of the same calibration position are determined in the second modal video frame. A one-to-one correspondence is established between the image coordinates of calibration positions with the same spatial location in the two modal images.
[0032] For example, four calibration positions P1 to P4 are set in the common field of view. The corresponding coordinates in the first modality video frame are (100, 80), (500, 80), (500, 400) and (100, 400), respectively. The corresponding coordinates in the second modality video frame are (65, 55), (465, 60), (460, 350) and (70, 345), thus forming four sets of correspondences between the first modality coordinates and the second modality coordinates.
[0033] The coordinate mapping relationship between the first mode coordinate system and the second mode coordinate system is determined based on multiple sets of corresponding image coordinates, and the coordinate mapping relationship is saved. It should be noted that the pre-defined coordinate mapping relationship is used to describe the transformation relationship between the first modal coordinates and the second modal coordinates. When the merchandise display area of a stall is approximately on the same plane, a two-dimensional plane projection transformation can be used as the pre-defined coordinate mapping relationship. This relationship converts the first modal coordinates to the second modal coordinates through mapping parameters. Substituting the aforementioned sets of corresponding coordinates into the coordinate mapping relationship, the mapping parameters are solved to ensure that each first modal calibration coordinate matches its corresponding second modal calibration coordinate after transformation. This yields the mapping relationship from the first modal coordinate system to the second modal coordinate system, and the mapping parameters are saved. For example, after obtaining the mapping parameters using the aforementioned four sets of corresponding coordinates, substituting the target coordinates (300, 200) in the first modality into the mapping relationship yields the corresponding coordinates in the second modality, such as (265, 180). Subsequently, the target position in the first modality can be transformed to the corresponding position in the second modality.
[0034] Select a first-mode video frame and a second-mode video frame with the same time correspondence from the dual-modal synchronous frame sequence. Use the selected first-mode video frame as the first-mode picture and the selected second-mode video frame as the second-mode picture, and output them after associating them with the same time marker.
[0035] It should be noted that after completing the coordinate mapping calibration, a first-mode video frame and a second-mode video frame are selected from the dual-modal synchronization frame sequence according to the same time correspondence. The selected first-mode video frame is used as the first-mode image, and the selected second-mode video frame is used as the second-mode image. Both are then output after being associated with the same time stamp. For example, if a synchronization frame with the time stamp 10:00:00.100 is selected from the dual-modal synchronization frame sequence, then the corresponding first-mode video frame F1 and second-mode video frame F2 are used as the first-mode image and the second-mode image, respectively, and both are associated with the time stamp 10:00:00.100. This ensures that the interaction position in the subsequent first mode and the target area in the second mode belong to the same acquisition time.
[0036] The second modality candidate regions include: Receive the click or touch position of the client in the first modal screen, convert the click or touch position into interactive coordinates in the coordinate system of the first modal screen, and associate it with the time stamp of the first modal screen; In this embodiment, it should be noted that the client displays a first modal screen corresponding to the current time marker. When the user selects an item in the first modal screen via mouse click, touchscreen touch, or other positioning methods, the client obtains the screen coordinates of the user's operation position in the display interface and converts the screen coordinates into first modal video frame coordinates according to the ratio between the display size of the first modal screen and the actual video frame size. For example, if the resolution of the first modal video frame is 1920×1080, the client display area is 960×540, and the user's click position in the display area is (150, 105), then the interactive coordinates in the first modal screen are obtained as (300, 210) according to the 2x coordinate conversion relationship. At the same time, this interactive coordinate is associated with the current time marker of the first modal screen, for example, associated with the time marker "10:00:00.100".
[0037] The interaction coordinates are sequentially determined to be inside the closed boundary of each product location area. When the interaction coordinates are inside one of the product location areas, the product number corresponding to that product location area is determined as the target product number, and the closed boundary of that product location area is determined as the first modality candidate boundary. After obtaining the interaction coordinates, they are sequentially compared with the spatial positions of each determined product location area to determine whether the interaction coordinates are located inside the closed boundary of the corresponding product location area. Specifically, a detection ray can be constructed along the interaction coordinates in a preset direction, and the number of intersections between the ray and the boundary of the product location area is counted. When the number of intersections is odd, the interaction coordinates are determined to be inside the product location area; when the number of intersections is even, the interaction coordinates are determined to be outside the area. If the interaction coordinates fall on the boundary line, they can be directly regarded as a valid interaction position of the product location area. Thus, the product location area to which the interaction coordinates belong is determined, and the product number corresponding to this area is determined as the target product number. At the same time, the closed boundary of the product location area is used as the first modal candidate boundary. For example, the first modal product location area corresponding to product number SP001 is enclosed by four vertices (120, 80), (260, 80), (260, 300), and (120, 300). The interactive coordinates calculated by the client are (185, 210). Since these coordinates are located inside the aforementioned closed area, SP001 is determined to be the target product number, and the closed boundary formed by connecting the aforementioned four boundary points is determined as the first modal candidate boundary.
[0038] According to the coordinate mapping relationship corresponding to the time identifier, the boundary positions on the first modality candidate boundary are transformed point by point to the second modality coordinate system. The second modality candidate region is formed based on the transformed boundary positions, and the second modality candidate region and the target product number are output as target positioning information.
[0039] Specifically, the boundary points on the first modality candidate boundary are sequentially input into the coordinate mapping relationship to obtain the corresponding coordinates of each boundary point in the second modality coordinate system. Then, the boundary points are connected according to their original connection order to form the second modality candidate region. If the mapped boundary points are (65, 55), (205, 58), (203, 245), and (68, 242), the closed region formed by connecting these boundary points is determined as the second modality candidate region, and it is combined with the target product number SP001 and the corresponding time identifier to form the target positioning information.
[0040] Calculating the difference between adjacent pixels includes: Using the center position of the second modality candidate region as the scanning starting point, the circumference angle is divided equally according to the number of preset scanning directions, and scanning is performed pixel by pixel outward along each scanning direction from the scanning starting point to establish a scanning path. The center position is obtained by calculating the average coordinates of each vertex of the second modality candidate region boundary; each scanning direction is set sequentially from zero degrees with the same angular interval, and the angular interval is obtained by dividing the full circle angle by the preset number of scanning directions; In this embodiment, it should be noted that the second modality candidate region is enclosed by multiple boundary points. Therefore, the average abscissa and ordinate of each boundary point are used as the abscissa and ordinate of the center position, respectively, to obtain the scanning starting point. ;by Starting from the second modality, scanning paths are generated along multiple preset directions in the current video frame.
[0041] To ensure that the scanning direction covers different directions around the candidate region, the circumference angle is divided according to the preset number of scanning directions. Evenly divided, the first The angles corresponding to each scanning direction are: ;in, Indicates the first The angle corresponding to each scanning direction; Indicates the total number of preset scan directions; The index representing the scanning direction, with values ranging from 0 to... ; Regarding the first Each scanning direction, with Establish a scan path starting from the origin: ;in, Indicates starting from the center position along the current scanning direction. The distance moved outward is used to determine the current scan position; Indicates the center location of the second modality candidate region; Indicates the start point of the scan Along the first Distance of movement in each scanning direction The coordinates of the scan position reached later are the current scan position for pixel difference calculation; Indicates the first The unit displacement component of each scanning direction in the abscissa direction of the second modal coordinate system; Indicates the first The unit displacement component of the scanning direction in the vertical direction of the second modal coordinate system, together with the scanning direction, determines the scanning direction. Along the scanning path in each scanning direction, the pixel values of adjacent scanning positions are obtained sequentially at preset pixel intervals, and the pixel difference value between adjacent scanning positions is calculated. The pixel difference value is the absolute value of the difference between the pixel values of adjacent scanning positions. Starting from the scanning start point, the pixel difference value of each position is calculated and recorded sequentially in the scanning direction.
[0042] Difference between adjacent scan positions: ;in, Indicates the first The difference between adjacent pixels between the current position and the next scanning position in each scanning direction is used to reflect the degree of pixel change at that position along the scanning direction; This indicates the current video frame in the second modality is at the next scan position. Pixel value at; Indicates the current video frame in the second modality at the scan position. Pixel value at; Indicates the first In each scanning direction, move along the direction from the scanning start point. The coordinates of the next scan position reached later; This represents the preset pixel interval, which is the pixel distance between the current scan position and the next scan position along the current scan direction. When using pixel-by-pixel scanning... Take the distance corresponding to one pixel; This represents absolute value operations, used to eliminate the positive and negative effects of the difference between two pixel values, so that the difference value represents the magnitude of pixel change between the two positions.
[0043] The formation of the second-mode target boundary includes: Along each scanning direction, starting from the scanning start point and moving outwards, the pixel difference value corresponding to each scanning position is compared with a preset difference threshold. The scanning position that first exceeds the preset difference threshold in each scanning direction is determined as the corresponding boundary crossing position. In this embodiment, it should be noted that the preset difference threshold can be pre-set as the maximum allowable pixel difference value between normal adjacent pixels in the second modality video frame. Its specific value can be determined by performing differential statistics on multiple adjacent pixel positions that do not have product boundaries. For example, when performing differential calculations on 100 pre-selected groups of normal adjacent pixels, if their pixel difference values are mainly distributed in the range of 0 to 30, a value of 30 or slightly higher can be set as the preset difference threshold to avoid normal pixel changes being misjudged as product boundaries.
[0044] Substitute the boundary crossing positions corresponding to each scanning direction into the corresponding scanning path to calculate the pixel coordinates of the boundary crossing positions in the current video frame of the second mode, and determine the pixel coordinates as the corresponding boundary crossing points; connect the boundary crossing points in the circumferential order of each scanning direction, and connect the first boundary crossing point with the last boundary crossing point to form a closed boundary, thus obtaining the target boundary of the second mode.
[0045] In this embodiment, it should be noted that after determining the boundary crossing position for each scanning direction, the scanning distance corresponding to the boundary crossing position is substituted into the scanning path of that scanning direction. The pixel coordinates in the current video frame of the second modality are calculated together with the scanning start coordinates, the scanning direction, and the scanning distance corresponding to the boundary crossing position. The calculated pixel coordinates are then determined as the boundary crossing point corresponding to that scanning direction. Since the aforementioned scanning path has already been determined, no additional coordinate transformation is needed for the boundary crossing position, and its position in the second modality video frame can be directly obtained. For example, if the scanning start point for a certain scanning direction is (200, 150), the corresponding scanning direction is 90°, and the boundary crossing position is determined to be 30 pixels away from the scanning start point, then moving 30 pixels along that direction yields the boundary crossing point (200, 180).
[0046] After obtaining the boundary crossing points corresponding to each scanning direction, the boundary crossing points are arranged according to the angular order of each scanning direction, so that the boundary crossing points corresponding to adjacent scanning directions are adjacent in the arrangement result. Adjacent boundary crossing points are connected sequentially according to this arrangement order. After connecting to the last boundary crossing point, the last boundary crossing point is connected to the first boundary crossing point to form a closed boundary profile. This closed boundary is determined as the second modal target boundary. For example, if 8 scanning directions are set and 8 boundary crossing points are obtained respectively, the corresponding boundary crossing points are connected sequentially according to the direction order of 0°, 45°, 90°..., 315°. Finally, the boundary crossing point in the 315° direction is connected to the boundary crossing point in the 0° direction to obtain the closed second modal target boundary.
[0047] Extracting the target product image includes: The closed area enclosed by the second modal target boundary is determined. Pixels in the second modal image that are located within the closed area are identified as target product pixels and retained. Pixels located outside the closed area are removed to obtain the target product image corresponding to the second modal target boundary.
[0048] In this embodiment, it should be noted that after obtaining the second modal target boundary, the closed area enclosed by the second modal target boundary is used as the extraction area of the target product. Specifically, the pixel positions in the second modal image are judged for region assignment, determining whether each pixel position is located inside the closed area enclosed by the second modal target boundary; when the pixel position is inside the closed area, the corresponding pixel is retained as the target product pixel; when the pixel position is outside the closed area, the corresponding pixel is removed, thereby obtaining a target product image containing only the target product area. Pixels located on the boundary line can be assigned to the closed area to ensure that the edges of the target product are completely preserved.
[0049] For example, the second modal target boundary is formed by connecting eight boundary crossing points in sequence to form a closed area. This closed area covers the target product body and its edge area. Then, the pixel position in the second modal image is judged one by one, the product pixels in the closed area are retained, and the stall background and adjacent product pixels outside the closed area are removed to finally obtain the target product image corresponding to the second modal target boundary.
[0050] The target interaction information includes: Obtain each boundary point on the second modal target boundary, and according to the arrangement order of each boundary point in the second modal target boundary, perform an inverse transformation through the coordinate mapping relationship between the first modal coordinate system and the second modal coordinate system to obtain the first modal pixel coordinates corresponding to each boundary point; In this embodiment, it should be noted that after obtaining the second modal target boundary, the boundary crossing points constituting the second modal target boundary are read, and the circumferential arrangement order of the boundary crossing points according to the scanning direction is retained to obtain the pixel coordinates of each boundary crossing point in the second modal coordinate system. Since the coordinate mapping relationship and corresponding mapping parameters from the first modal coordinate system to the second modal coordinate system have been established, a reverse transformation relationship corresponding to the coordinate mapping relationship is further established to convert the second modal coordinates into the first modal coordinates. Specifically, the second modal pixel coordinates of each boundary crossing point are substituted into the reverse transformation relationship in their arrangement order to calculate the first modal pixel coordinates corresponding to each boundary crossing point, thereby establishing a point-to-point coordinate correspondence between the second modal target boundary and the corresponding first modal boundary. For example, the mapping relationship from the first modal coordinate system to the second modal coordinate system is established by using two-dimensional plane projection transformation, and the mapping parameters are obtained based on multiple sets of calibration coordinates. If the second modal target boundary contains 4 boundary points in a circumferential order, and its second modal pixel coordinates are (205, 100), (300, 110), (295, 250) and (200, 240) respectively, then the above coordinates are substituted into the inverse transformation relationship determined by the mapping parameters to obtain the first modal pixel coordinates (410, 200), (600, 220), (590, 500) and (400, 480) respectively.
[0051] Connect the pixel coordinates of each first modality in the order of arrangement to form the first modality target boundary; associate the first modality target boundary, the second modality target boundary, the target product number and the target product image with the corresponding time markers to form target interactive information.
[0052] It should be noted that, following the original arrangement order of the boundary points in the second modality target boundary, the aforementioned first modality pixel coordinates are sequentially connected, and the last first modality pixel coordinate is connected to the first first modality pixel coordinate, so that the first modality pixel coordinates form a closed boundary, which is then defined as the first modality target boundary. For example, by sequentially connecting four first modality pixel coordinates in the order (410, 200) → (600, 220) → (590, 500) → (400, 480), and connecting (400, 480) and (410, 200), the first modality target boundary can be formed.
[0053] After obtaining the first modal target boundary, the first modal target boundary, the second modal target boundary, the target product number, and the target product image extracted from the second modal frame are associated. This information is then associated with the same time marker corresponding to the current video frame in both modalities, ensuring that the target product location, target product image, and product number for the same client interaction remain consistent, thus forming target interaction information. For example, if the target product number is "SP001" and the corresponding time marker is "10:00:01.100", then this time marker, SP001, the aforementioned first modal target boundary, second modal target boundary, and extracted target product image are associated to form target interaction information corresponding to SP001.
[0054] Outputting the annotation results and target product image to the client includes: Read the first modal target boundary, second modal target boundary, target product number and target product image from the target interaction information, and determine the corresponding first modal screen and second modal screen according to the time identifier corresponding to the target interaction information; In this embodiment, it should be noted that after receiving the target interaction information, the synchronization annotation module reads the first modal target boundary, the second modal target boundary, the target product number, the target product image, and the time stamp. The time stamp is used to determine the video frame corresponding to the current client interaction in both modal frames. Based on the time stamp, the synchronization annotation module searches for first and second modal video frames with the same time stamp in the first and second modal video streams, respectively, and identifies them as the corresponding first and second modal frames.
[0055] According to the arrangement order of each boundary point in the first modal target boundary, each boundary point is connected sequentially in the first modal screen and the first and last boundary points are connected to form the first modal annotation boundary to identify the target product in the first modal screen; After determining the corresponding screen, read each boundary point in the first modal target boundary, and connect adjacent boundary points in sequence according to the arrangement order of each boundary point in the first modal target boundary. Then connect the last boundary point with the first boundary point so that each boundary point forms a closed contour. This closed contour is determined as the first modal annotation boundary, thereby delineating the area where the target product is located in the first modal screen.
[0056] According to the arrangement order of each boundary point in the second modal target boundary, each boundary point is connected sequentially in the second modal screen and the first and last boundary points are connected to form the second modal annotation boundary to identify the target product in the second modal screen, and the target product number is displayed at the second modal annotation boundary; The boundary points in the target boundary of the second modality are read in the same way. Adjacent boundary points are connected sequentially in the corresponding positions of the second modality screen according to their arrangement order. The first and last boundary points are connected to form a closed contour, which serves as the second modality annotation boundary to delineate the target product area in the second modality screen. At the same time, the target product number is displayed at the preset display position of the second modality annotation boundary, so that the client can determine the specific product corresponding to the annotated area.
[0057] The first modal image, the second modal image, and the target product image, after being annotated, are associated with their corresponding time markers to form interactive output data, which is then output to the client.
[0058] In this embodiment, it should be noted that after the annotation of the first modal screen and the second modal screen is completed, the two annotated screens and the target product image are read respectively, and the corresponding time stamp is obtained from the target interaction information. To ensure that the three belong to the same client interaction, the time stamp is used as a unified association identifier and written into the data records corresponding to the first modal annotation screen, the second modal annotation screen, and the target product image respectively. At the same time, the target product number is used as the common product identifier of the three, thereby establishing a correspondence between the first modal annotation screen, the second modal annotation screen, and the target product image. The three, after establishing the correspondence, are combined according to a preset data encapsulation order to form interactive output data, which is sent to the client through the communication link. After receiving the interactive output data, the client identifies the data belonging to the same interaction based on the time stamp and the target product number, and displays the two annotated screens and the target product image respectively.
[0059] For example, assuming the target product ID is "SP001" and the time stamp is "10:00:01.100", the annotation process results in: First modal annotation screen: F1_10:00:01.100; Second modal annotation screen: F2_10:00:01.100; Target product image: IMG_SP001_10:00:01.100. All three are written into the time stamp "10:00:01.100" and the product ID "SP001", and then combined in the order of First modal annotation screen + Second modal annotation screen + Target product image to form the interaction output data corresponding to this interaction with SP001, which is then sent to the client. Example
[0060] Please see Figure 2 As shown, parts not described in detail in this embodiment are described in Embodiment 1. A digital stall interaction method based on dual-modal video communication is provided, including: S1. Collect the first and second modal video streams of the stall and synchronize them in time, identify the product location area, mark the mapping relationship between the two modal coordinate systems, and output the first modal image and the second modal image; S2. Receive the interaction coordinates of the client for the first modal screen, compare them with the product location area, determine the target product selected by the client and the first modal candidate boundary, and obtain the second modal candidate area by combining the mapping relationship; S3. Starting from the center of the candidate region of the second modality, perform a pixel-by-pixel scan along several preset directions, calculate the difference value between adjacent pixels, determine the pixels whose difference value exceeds the difference threshold as boundary crossing points, and connect them in sequence to form the target boundary of the second modality. S4. Extract the target product image from the second modal screen based on the second modal target boundary, and inversely map the second modal target boundary to obtain the first modal target boundary to form target interaction information; S5. Based on the target interaction information, label the target products in the first modal screen and the second modal screen, and output the labeling results and the target product images to the client. Example
[0061] This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the operation mode of the digital stall interaction system based on dual-modal video communication provided above.
[0062] Since the electronic device described in this embodiment is the electronic device used to implement the digital stall interaction system and method based on dual-modal video communication described in this application embodiment, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this embodiment based on the digital stall interaction system and method based on dual-modal video communication described in this application embodiment. Therefore, how the electronic device implements the method in this application embodiment will not be described in detail here. Any electronic device used by those skilled in the art to implement the digital stall interaction system and method based on dual-modal video communication described in this application embodiment falls within the protection scope of this application.
[0063] It should be noted that all formulas in this manual are calculated by removing dimensions and taking their numerical values. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.
[0064] The above description is merely a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for users of ordinary technical skills, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.
Claims
1. A digital stall interaction system based on dual-modal video communication, characterized in that, include: The acquisition and calibration module is used to acquire the first and second modal video streams of the stall and synchronize them in time, identify the product location area, calibrate the mapping relationship between the two modal coordinate systems, and output the first modal image and the second modal image; The interactive positioning module is used to receive the interactive coordinates of the client for the first modal screen, compare them with the product location area, determine the target product selected by the client and the first modal candidate boundary, and obtain the second modal candidate area by combining the mapping relationship. The boundary reconstruction module is used to perform pixel-by-pixel scanning along several preset directions starting from the center of the second modality candidate region, calculate the difference value between adjacent pixels, determine the pixels whose difference value exceeds the difference threshold as boundary crossing points, and connect them sequentially to form the second modality target boundary. The image extraction module is used to extract the target product image from the second modal screen based on the second modal target boundary, and to inversely map the second modal target boundary to obtain the first modal target boundary, thereby forming target interaction information; The synchronous annotation module is used to annotate the target products in the first modal screen and the second modal screen according to the target interaction information, and output the annotation results and the target product images to the client.
2. The digital stall interaction system based on dual-modal video communication according to claim 1, characterized in that, The product location identification area includes: A first video acquisition device and a second video acquisition device are configured at the digital stall to collect video information of different modalities; a first modal video stream covering the stall display area and a second modal video stream presenting local details of the stall display area are collected respectively, and the acquisition time of each video frame is recorded; The first and second modal video frames corresponding to the acquisition time are selected according to the preset synchronization time interval. When the acquisition time difference between the two video frames does not exceed the preset acquisition time difference threshold, the two video frames are established to form a time correspondence and a dual-modal synchronization frame sequence is formed. For the first modal video frame in the dual-modal synchronous frame sequence, the product location area is determined based on the image differences between the product and the surrounding area and the regional boundaries between adjacent products, and a product number is associated with each product location area.
3. The digital stall interaction system based on dual-modal video communication according to claim 2, characterized in that, The output of the first modal image and the second modal image includes: Several calibration positions are set in the common field of view of the first modal video stream and the second modal video stream. The image coordinates of each calibration position in the first modal video frame and the second modal video frame are determined respectively, and a correspondence is established between the two image coordinates corresponding to the same calibration position. The coordinate mapping relationship between the first mode coordinate system and the second mode coordinate system is determined based on multiple sets of corresponding image coordinates, and the coordinate mapping relationship is saved. Select a first-mode video frame and a second-mode video frame with the same time correspondence from the dual-modal synchronous frame sequence. Use the selected first-mode video frame as the first-mode picture and the selected second-mode video frame as the second-mode picture, and output them after associating them with the same time marker.
4. The digital stall interaction system based on dual-modal video communication according to claim 3, characterized in that, The obtained second modality candidate region includes: Receive the click or touch position of the client in the first modal screen, convert the click or touch position into interactive coordinates in the coordinate system of the first modal screen, and associate it with the time stamp of the first modal screen; The interaction coordinates are sequentially determined to be inside the closed boundary of each product location area. When the interaction coordinates are inside one of the product location areas, the product number corresponding to that product location area is determined as the target product number, and the closed boundary of that product location area is determined as the first modality candidate boundary. According to the coordinate mapping relationship corresponding to the time identifier, the boundary positions on the first modality candidate boundary are transformed point by point to the second modality coordinate system. The second modality candidate region is formed based on the transformed boundary positions, and the second modality candidate region and the target product number are output as target positioning information.
5. A digital stall interaction system based on dual-modal video communication according to claim 4, characterized in that, The calculation of the difference between adjacent pixels includes: Using the center position of the second modality candidate region as the scanning starting point, the circumference angle is divided equally according to the number of preset scanning directions, and scanning is performed pixel by pixel outward along each scanning direction from the scanning starting point to establish a scanning path. The center position is obtained by calculating the average coordinates of each vertex of the second modality candidate region boundary; each scanning direction is set sequentially from zero degrees with the same angular interval, and the angular interval is obtained by dividing the full circle angle by the preset number of scanning directions; Along the scanning path of each scanning direction, the pixel values of adjacent scanning positions are obtained sequentially at preset pixel intervals, and the pixel difference value between adjacent scanning positions is calculated. The pixel difference value is the absolute value of the difference between the pixel values of adjacent scanning positions. Starting from the scanning start point, the pixel difference value of each position is calculated and recorded sequentially along the scanning direction.
6. A digital stall interaction system based on dual-modal video communication according to claim 5, characterized in that, The formation of the second modal target boundary includes: Along each scanning direction, starting from the scanning start point and moving outwards, the pixel difference value corresponding to each scanning position is compared with a preset difference threshold. The scanning position that first exceeds the preset difference threshold in each scanning direction is determined as the corresponding boundary crossing position. Substitute the boundary crossing positions corresponding to each scanning direction into the corresponding scanning path to calculate the pixel coordinates of the boundary crossing positions in the current video frame of the second mode, and determine the pixel coordinates as the corresponding boundary crossing points; connect the boundary crossing points in the circumferential order of each scanning direction, and connect the first boundary crossing point with the last boundary crossing point to form a closed boundary, thus obtaining the target boundary of the second mode.
7. A digital stall interaction system based on dual-modal video communication according to claim 6, characterized in that, The extraction of the target product image includes: The closed area enclosed by the second modal target boundary is determined. Pixels in the second modal image that are located within the closed area are identified as target product pixels and retained. Pixels located outside the closed area are removed to obtain the target product image corresponding to the second modal target boundary.
8. A digital stall interaction system based on dual-modal video communication according to claim 7, characterized in that, The target interaction information includes: Obtain each boundary point on the second modal target boundary, and according to the arrangement order of each boundary point in the second modal target boundary, perform an inverse transformation through the coordinate mapping relationship between the first modal coordinate system and the second modal coordinate system to obtain the first modal pixel coordinates corresponding to each boundary point; Connect the pixel coordinates of each first modality in the order of arrangement to form the first modality target boundary; associate the first modality target boundary, the second modality target boundary, the target product number and the target product image with the corresponding time markers to form target interactive information.
9. A digital stall interaction system based on dual-modal video communication according to claim 8, characterized in that, The step of outputting the annotation results and target product image to the client includes: Read the first modal target boundary, second modal target boundary, target product number and target product image from the target interaction information, and determine the corresponding first modal screen and second modal screen according to the time identifier corresponding to the target interaction information; According to the arrangement order of each boundary point in the first modal target boundary, each boundary point is connected sequentially in the first modal screen and the first and last boundary points are connected to form the first modal annotation boundary to identify the target product in the first modal screen; According to the arrangement order of each boundary point in the second modal target boundary, each boundary point is connected sequentially in the second modal screen and the first and last boundary points are connected to form the second modal annotation boundary to identify the target product in the second modal screen, and the target product number is displayed at the second modal annotation boundary; The first modal image, the second modal image, and the target product image, after being annotated, are associated with their corresponding time markers to form interactive output data, which is then output to the client.
10. A digital stall interaction method based on dual-modal video communication, implemented by any one of claims 1 to 9, characterized in that, include: S1. Collect the first and second modal video streams of the stall and synchronize them in time, identify the product location area, mark the mapping relationship between the two modal coordinate systems, and output the first modal image and the second modal image; S2. Receive the interaction coordinates of the client for the first modal screen, compare them with the product location area, determine the target product selected by the client and the first modal candidate boundary, and obtain the second modal candidate area by combining the mapping relationship; S3. Starting from the center of the candidate region of the second modality, perform a pixel-by-pixel scan along several preset directions, calculate the difference value between adjacent pixels, determine the pixels whose difference value exceeds the difference threshold as boundary crossing points, and connect them in sequence to form the target boundary of the second modality. S4. Extract the target product image from the second modal screen based on the second modal target boundary, and inversely map the second modal target boundary to obtain the first modal target boundary, forming target interaction information; S5. Based on the target interaction information, label the target products in the first modal screen and the second modal screen, and output the labeling results and the target product images to the client.
