Method of providing commodity search result information and electronic device

Abstract

Embodiments of the present application disclose a method for providing commodity search result information and an electronic device. The method comprises: in the process of collecting a real-time image stream of a shooting object, performing subject recognition from the real-time image stream, and acquiring the position of the subject in multiple image frames and the change of the state of the subject, the state being related to the position of the subject in the image frame and / or the area proportion of the subject image in the screen; acquiring a commodity information search result related to the recognized subject; generating an information card to be displayed according to the commodity information search result and the state of the subject, wherein different subject states correspond to different information card types; and displaying the information card in the real-time image stream at the position where the subject is located. Through the embodiments of the present application, the search efficiency can be improved, and the user participation can be improved.

Description

Technical Field

[0001] This application relates to the field of image-based information search technology, and in particular to a method and electronic device for providing product search results information. Background Technology

[0002] In traditional product information service systems, product search is based on keywords. Users input product keywords as search criteria, and the system then provides a page of search results matching those criteria. With the development of mobile devices and other technologies, some product information service systems also offer image-based product search functionality. Users can take photos of items they find interesting in real life, or save images they find online if they are interested in the items depicted. These saved photos or images can then be uploaded to the server. Alternatively, users can take photos directly through the photo-taking interface on the client-side page and upload them to the server. Upon receiving the photos or images, the server can identify and compare the subjects in the images to determine and return relevant product search results. Users can then search for the products they need from the search results list and even make online purchases.

[0003] Image-based product search can help users obtain search results more conveniently in various scenarios. However, how to further improve search efficiency for users remains a goal that those skilled in the art are constantly pursuing. Summary of the Invention

[0004] This application provides a method and electronic device for providing product search results information, which can improve search efficiency and enhance user engagement.

[0005] This application provides the following solution:

[0006] A method for providing product search result information includes:

[0007] During the real-time image stream acquisition of the subject, subject recognition is performed from the real-time image stream, and the changes in the position and state of the subject in multiple image frames are obtained. The state is related to the position of the subject in the image frame and / or the area ratio of the subject image on the screen.

[0008] Obtain search results for product information related to the identified entity;

[0009] Information cards to be displayed are generated based on the product information search results and the status of the subject, wherein different subject statuses correspond to different information card types;

[0010] The information card is mapped to the location of the corresponding subject in the real-time image stream for display.

[0011] The step of obtaining search results for product information related to the identified subject includes:

[0012] The image size and clarity of the identified main body are judged. If the image size and clarity of the main body meet the first condition, search results for product information related to the identified main body are obtained.

[0013] This also includes:

[0014] As the image size and clarity of the identified main body gradually improve, if the image size and clarity of the main body meet the second condition, then the search results for product information related to the identified main body are re-acquired, and the content displayed in the information card is updated according to the search results for product information obtained from the re-request.

[0015] This also includes:

[0016] The similarity between the product images in the obtained product information search results and the identified subject images is compared, and the product information search results are filtered based on the comparison results.

[0017] This also includes:

[0018] As the information card moves along with the changing position of the subject, the latest position of the identified subject is read at preset time intervals; wherein, the time interval is related to the persistence of vision of the human eye and is greater than the time interval between each frame in the image stream;

[0019] After the latest position is read, an animation is generated for the currently displayed information card to smoothly move from the current position to the latest position, and the position of the information card is kept unchanged at the latest position until a new latest position is read in the next cycle, at which time a new animation is generated.

[0020] The relative positional relationship between the information card and the anchor point of the main body can be varied; the position of the anchor point is determined based on the center point of the main body.

[0021] The method further includes:

[0022] As the information card moves along with the change in the position of the main body, it is determined whether the information card will collide with the edge of the screen in the current relative position relationship. If so, the relative position relationship is switched to another.

[0023] This also includes:

[0024] Before switching to other relative position relationships, it is also determined whether the information card will collide with another screen edge after switching to other relative position relationships. If not, then switch to other relative position relationships; if so, then maintain the current relative position relationship.

[0025] In the real-time image stream, anchor point elements are also displayed at the location of the anchor point of the main body;

[0026] The method further includes:

[0027] During the process of switching relative positions, the anchor element is hidden;

[0028] After switching the relative position relationship, the anchor element is switched to a visible state.

[0029] If multiple entities are identified in the real-time image stream, their respective product information search results are obtained, and information cards of the corresponding type are generated according to their respective states.

[0030] The method further includes:

[0031] In the process of mapping multiple information cards into the real-time image stream in real time and moving according to the position changes of their respective subjects, it is determined whether collisions will occur between the multiple information cards.

[0032] If so, the display position of the information card corresponding to the subject with the lower priority will be moved according to the different subject status priorities of the information card that collides.

[0033] The information cards corresponding to the different subject states are displayed in different layers, and the layers are arranged in descending order of priority of each subject state.

[0034] This also includes:

[0035] During the process of mapping the information card to the location of the corresponding subject in the real-time image stream for display, if the subject's state changes, the type of information card mapped to the real-time image stream will be switched.

[0036] There is a buffer at the boundary of the judgment parameters for different subject states, so that when the judgment parameters change within the buffer, the current subject state and the type of the corresponding information card remain unchanged.

[0037] In the process of switching card types, the switching process is divided into two stages. The first stage is used to gradually reduce the transparency of the first type of information card currently projected in the real-time image stream. The second stage is used to project the second type of information card into the real-time image stream and gradually increase the transparency of the second type of information card.

[0038] The main body's state includes a display state, which includes a first display state and a second display state; in the display state, the information card displays content related to the product search results; the size and content of the information card corresponding to the first display state and the second display state are different.

[0039] The subject's state also includes a discovery state, and the information card in the discovery state is used to display the identification result of the category to which the subject belongs.

[0040] The state of the subject also includes the interactive state;

[0041] The method further includes:

[0042] After the subject enters the interactive state and maintains it for a threshold time, it will be redirected to the product search results list page or the details page of the product included in the currently displayed information card.

[0043] This also includes:

[0044] After the subject enters the interactive state, the 3D model of the product is projected onto the real-time image stream for display.

[0045] This also includes:

[0046] The information card provides options for adding the displayed products to a bulk checkout product collection.

[0047] In the case of real-time image stream acquisition of the subject using augmented reality (AR) glasses, the method further includes:

[0048] When the operation of grabbing the information card and moving it in the target direction is detected from the real-time image stream, the displayed product is added to the batch checkout product set.

[0049] A method for providing product information includes:

[0050] During the process of acquiring a real-time image stream of the subject through augmented reality (AR) glasses, subject recognition is performed from the real-time image stream.

[0051] Obtain search results for product information related to the identified entity;

[0052] Information cards to be displayed are generated based on the product information search results and mapped to the corresponding location of the subject in the real-time image stream for display.

[0053] When the operation of grabbing the information card and moving it in the target direction is detected from the real-time image stream, the displayed product is added to the batch checkout product set.

[0054] An apparatus for providing product search results information includes:

[0055] The subject recognition unit is used to perform subject recognition from the real-time image stream during the real-time image stream acquisition of the subject, and to obtain the changes in the position and state of the subject in multiple image frames. The state is related to the position of the subject in the image frame and / or the area ratio of the subject image on the screen.

[0056] The product search result acquisition unit is used to acquire product information search results related to the identified subject;

[0057] The information card generation unit is used to generate information cards to be displayed based on the product information search results and the status of the subject, wherein different subject statuses correspond to different information card types;

[0058] The information card display unit is used to map the information card to the location of the corresponding subject in the real-time image stream for display.

[0059] An apparatus for providing product information, comprising:

[0060] The subject recognition unit is used to perform subject recognition from the real-time image stream during the process of acquiring a real-time image stream of a photographed object through augmented reality (AR) glasses.

[0061] The product search result acquisition unit is used to acquire product information search results related to the identified subject;

[0062] The information card display unit is used to generate information cards to be displayed based on the product information search results, and map them to the location of the corresponding subject in the real-time image stream for display.

[0063] The product addition unit is used to add the products displayed in the real-time image stream to the batch settlement product set when the user performs an operation of grabbing the information card and moving it in the target direction.

[0064] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the preceding methods.

[0065] An electronic device, comprising:

[0066] One or more processors; and

[0067] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of any of the preceding methods.

[0068] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0069] Through the embodiments of this application, subject recognition can be performed from the real-time image stream during the acquisition of the subject. Furthermore, the position and state changes of the subject across multiple image frames can be obtained, where the specific state may be related to the clarity of the subject portion of the image and / or its proportion within the image frame. During this real-time image stream acquisition process, search results for product information related to the identified subject can also be obtained. Based on the search results and the subject's state, an information card to be displayed is generated and mapped to the corresponding position of the subject in the real-time image stream. This allows users to not only view the content actually captured by the camera in the image stream but also to view these information cards. Moreover, the type of information card can change with the subject's state. This allows for the display of product search results on a "lens page" via AR, improving search efficiency. Simultaneously, the changing information card type allows users to gradually obtain richer and more detailed information about the searched products, making the information cards more engaging and enhancing user participation.

[0070] In addressing the issue of discontinuous motion trajectories encountered by the information card while following the main body during the implementation of the aforementioned solution, a preferred embodiment of this application can resolve this problem through segmented fitting. Specifically, the main body position information identified by the algorithm can be read periodically. Within the same period, the information card's position remains unchanged until the next period arrives and the latest main body position is read, at which point the information card "starts moving." Specifically, a translational animation of a target duration (equal to the length of the reading period) can be generated for the information card. In this animation, the information card can exhibit continuous motion, thus resolving the problem of discontinuous motion trajectories and reducing the consumption of terminal computing and display resources.

[0071] To address the issue of collisions between information cards and screen edges, two relative positional relationships between the information card and the center point of the main body can be preset. When such collisions occur, the problem can be resolved by switching the relative positional relationships.

[0072] To address the collision issue between information cards, the priority relationship between different subject states can be used to determine the priority of different types of information cards. This allows lower-priority information cards to be identified and moved, preventing them from obscuring each other. Alternatively, multiple layers can be created, displaying information cards corresponding to different subject states on different layers. The hierarchical relationship between these layers can also be fixed according to the priority relationship between subject states; for example, interactive information cards can be permanently displayed on the top layer. This way, even if a certain information card cannot be moved, higher-priority information cards will not be obscured by lower-priority ones.

[0073] To address the potential "jitter" issue that may occur when switching information card types for the same entity during a change in state, a buffer is provided at the boundary of the judgment parameters for different entity states. This ensures that when the judgment parameters change within the buffer, the current entity state and the corresponding information card type remain unchanged.

[0074] In addition, the specific subject state can also include an interactive state, in which interactive information can be provided to the user. For example, if the state is maintained for a certain period of time, the user can be automatically redirected to the search results list page or the details page of a specific product. Alternatively, a 3D model of a specific product can be mapped onto a real-time image for display, and so on.

[0075] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 This is a schematic diagram of the system architecture provided in the embodiments of this application;

[0078] Figure 2 This is a flowchart of the first method provided in the embodiments of this application;

[0079] Figure 3 This is a schematic diagram of the interface provided in an embodiment of this application;

[0080] Figure 4 This is a schematic diagram of the timeline provided in an embodiment of this application;

[0081] Figure 5 This is a flowchart of the second method provided in the embodiments of this application;

[0082] Figure 6 This is a schematic diagram of the first device provided in the embodiments of this application;

[0083] Figure 7 This is a schematic diagram of the second device provided in the embodiments of this application;

[0084] Figure 8 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0085] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0086] In this embodiment, to improve user search efficiency, search results can be prioritized. Specifically, after a user initiates a search request through an image-based product search entry point provided on the client's homepage, if the user selects shooting mode (instead of uploading a local image), the camera or other shooting components on the terminal device can be activated to capture the image. Typically, once in the shooting state, image capture is possible, but it's still in image stream mode; the corresponding page can be called the "lens page." Only after the user clicks the specific "take a photo" button on the "lens page" will the image be saved as a photo or video. In existing technologies, photos or videos are only uploaded to the server for subject recognition and search results after the actual photo or video is generated. However, in this embodiment, subject recognition can be performed on the client side while the "lens page" displays a real-time image stream. If the subject image is clear enough, the client can request relevant product search results information from the server. This product search results information can then be organized into information cards and mapped onto the real-time image stream for display, creating an AR (Augmented Reality) effect.

[0087] Furthermore, since the real-time image stream is dynamically changing, the position and state (including the size of the subject image) of the identified subject also change in real time. Therefore, in this embodiment, the position and state of the subject can be recorded in real time. Moreover, various types of information cards can be predefined, and different types of information cards can be displayed under different subject states. Specifically, the subject state can be determined based on the subject's position in the image, the area ratio between the subject image and the screen, etc., with the specific area ratio varying according to the subject's distance from the camera. For example, the subject state can be divided into discovery state, display state, and interactive state. The display state can be further subdivided into a first display state, a second display state, and so on.

[0088] In this scenario, assuming a user initiates an image search and activates their camera, they can point the lens at a subject (without pressing a shutter button). Through the mobile device, the subject gradually increases in size on the screen, becoming clearer. During this process, information cards containing product search results are mapped onto the image stream for display. This allows the user to see not only the actual captured image but also the information card displayed via AR technology. (In current technology, product search results are not provided during this process; the photo is only submitted to the server to request search results after the user takes the picture.) Furthermore, the information card is dynamic. For example, when the subject is relatively far from the lens, it might only identify the category of the item being photographed, without matching it to any available products in the system. In this case, the subject is in a "discovery state," and the specific information card might only display the category information of the photographed object. Subsequently, as the camera gradually zooms in on the subject, the area occupied by the main image on the screen gradually increases, and the image becomes clearer. When it reaches a point where a product search based on this main image is possible, a search can be initiated to the server. Accordingly, the subject enters the first display state, where the information card can present specific product search results (e.g., a summary of a product in the search results, or a switching option to display summaries of other products in the search results, etc.). Later, when the area of ​​the main image further increases and the image becomes clearer, a new search request can be initiated to the server (or not, depending on actual needs). At this point, the subject can enter the second display state, where the content in the information card can be updated, and a larger information card can be generated to present richer information. If the area of ​​the main image further increases and remains so for a certain period, the subject can enter an interactive state. At this point, it can automatically jump to the details page of the currently displayed product in the information card, or it can map the product's 3D model into the image stream, etc.

[0089] In the process of developing this application, the inventors also discovered that when mapping information cards to an image stream, and when multiple different types of information cards corresponding to different subject states are involved, the display of information cards needs to be processed on the client side, which may encounter some problems. For example, if the information card needs to move with the subject, how can the movement trajectory of the information card be made smoother and more fluid, provided that the performance on the client side is sufficient? Or, since the information card needs to occupy a certain area, it may collide with the screen edge during its movement with the subject. When multiple subjects are identified in the image stream, multiple information cards need to be mapped to the image stream, and collisions may also occur between the multiple information cards. When such collisions occur, the content in the information card may be obscured by the screen edge or other information cards, and how to solve this problem also needs to be considered. In addition, since it is necessary to switch between different types of information cards when the subject state changes, and the change in subject state is usually caused by the change in distance between the subject and the lens, if this distance is exactly at the boundary between two subject states, jitter may occur. That is, the information card may repeatedly and rapidly switch back and forth between two types, and so on. This application provides corresponding solutions to each of the above-mentioned problems in real time.

[0090] In addition to controlling the movement and switching of the information card as described above, in practical implementation, some "action points" can be pre-emptively implemented based on this information card. For example, a specific "action point" could include adding a product to the "shopping cart." That is, users can directly add products to their cart through this information card without needing to navigate to the product search results page or product details page. Specifically, if implemented on mobile devices such as smartphones, the information card can provide specific options for adding the displayed products to the "shopping cart," allowing users to complete the addition by clicking on these options.

[0091] Alternatively, when using AR glasses, users can add items displayed on information cards to their cart through richer interactive methods. For example, users can use the AR glasses' handles or make gestures to grab the information card and move it in a specific direction (e.g., to their chest), thereby adding the displayed items to their cart.

[0092] From a system architecture perspective, see Figure 1This application's embodiments may involve client-side and server-side components in a product information service system. The client-side component may include an App (application) installed on a mobile terminal such as a smartphone, or an App installed on AR glasses devices, etc. Specific algorithms related to subject recognition and whether a search request can be initiated can be deployed on the client-side. Furthermore, the display control of information cards during movement or switching, including image stream processing, card rendering processing, subject following processing, card anti-collision processing, and card switching processing, can also be performed on the client-side. The server-side primarily provides specific product search results, upon which the client can generate specific information cards.

[0093] The specific implementation schemes provided in the embodiments of this application will be described in detail below.

[0094] Example 1

[0095] First, this first embodiment provides a method for providing product search result information from the perspective of the aforementioned client, see [link to previous document]. Figure 2 The method may include:

[0096] S201: During the real-time image stream acquisition of the subject, subject recognition is performed from the real-time image stream, and the position and state changes of the subject in multiple image frames are obtained. The state is related to the position of the subject in the image frame and / or the area ratio of the subject image on the screen.

[0097] In practical implementation, the client-side search entry point can provide an option to initiate product searches based on images. If the user selects this option, they will enter the main interface for image-based product searches. This main interface offers options such as "Shoot" and "Select from Local Album." If the user selects the "Shoot" option, the terminal device's camera will be activated, entering viewfinder mode. At this point, the interface will display the captured image, which will dynamically change. Additionally, a "Real-time Image Stream Mode" switch can be provided on this interface. By default, this switch is off; the user presses the "Take Photo" button after obtaining a satisfactory viewfinder result to obtain the specific photo before initiating a search request. If the user turns on the "Real-time Image Stream Mode" switch, they will enter "Real-time Image Stream Mode." In this mode, the client-side algorithm can perform subject recognition on each frame of the real-time image stream.

[0098] Subject recognition primarily involves identifying foreground objects from an image stream. For example, if a product is being photographed in a room, the product can be identified as the subject, while the background of the room, although captured in the image stream, will not be recognized as the subject. Of course, multiple subjects may be identified within the same image stream. Specific subject recognition results can include the subject's ID, coordinates, dimensions, category, and whether a search request can be sent to the server, etc. The client can record these recognition results for subsequent use in generating and displaying information cards, as well as in motion / switching control.

[0099] The subject number is primarily used to label the specifically identified subjects. This label is provided by the algorithm, and the same subject has the same number in each frame's recognition result. Thus, when multiple subjects are identified in the image stream, they can be distinguished by their subject numbers, enabling subsequent information card tracking and other functions.

[0100] The subject's coordinates primarily refer to the coordinates of its center point on the screen, while its length and width refer to the maximum width and maximum length of the subject image on the screen. During dynamic framing, these coordinates and dimensions constantly change as the distance between the subject and the lens changes, and can be identified using the aforementioned client-side algorithm. Correspondingly, these can be recorded by the client.

[0101] The main category refers to the product category to which the subject image likely belongs; that is, identifying the specific item the subject is, such as a cup, headphones, etc. This information can be used to generate a "catch-all" information card. In other words, when a specific product search result cannot be obtained, information such as the category name can be displayed in the information card based on the product category to which the subject belongs.

[0102] Regarding whether to initiate a search request to the server, this refers to the fact that during dynamic framing, the size and clarity of the subject image are constantly changing. Initially, due to focusing issues, the subject image may not be clear enough or may be small. In this case, the server may struggle to provide accurate product search results based on such a subject image, so it's unnecessary to submit a search request. Later, if the subject image becomes large and clear enough, it can be determined whether to initiate a product search request to the server. Furthermore, as the subject image becomes clearer, the server may be able to provide more accurate search results, so a new product search request can be initiated, and so on. All of these decisions can be made by the aforementioned client-side algorithm during the subject recognition process.

[0103] After the algorithm identifies the above information, recording can be performed on the client side. The position information of the subject can be determined based on information such as the coordinates of the subject's center point. For the determination of the subject state, it can mainly be based on the center point coordinates of the subject image and / or the area ratio of the subject image in the screen.

[0104] For example, in one mode, the subject state can be determined simply based on the area ratio of the subject image in the screen. Specifically, when the subject state is divided into a discovery state, a first display state, a second display state and an interactive state as mentioned above, when the shooting object is still relatively far from the lens, the area ratio of the subject image in the screen (represented by letter M for assumption) is very small, for example, a<M<b. At this time, it may only be possible to identify what category of item the user is shooting, but it is impossible to determine which actually sellable commodities in the system it is related to, therefore, the subject can be determined to be in the "discovery state". Then, as the lens gradually approaches the shooting object, the area occupied by the subject image in the screen gradually increases, for example, b≤M<c, and the image also gradually becomes clear. When the subject image reaches the level where commodity search can be performed based on it, a search can be initiated to the server, and accordingly, the subject enters the first display state. Then, when the area of the subject image further increases, for example, c≤M<d, and the image becomes clearer, a new search request can also be initiated to the server (or, no new request is needed, which depends on actual requirements), at this time, the subject can enter the second display state. Then, if the area of the subject image further increases, for example, d≤M<e, and this condition lasts for a certain period of time (for example, 3S), the subject enters the interactive state, and so on.

[0105] Alternatively, in another case, when determining the subject state, in addition to considering the factor of the area ratio occupied by the subject image, the coordinate factor of the subject image can also be considered. For example, with regard to various predefined subject states, in addition to defining the area ratio of the subject image corresponding to each subject state, the position information corresponding to each subject state can also be defined. For example, for the discovery state, the first display state, the second display state and the interactive state, a "discovery frame", a "display frame", an "interactive frame" and the like can be predefined. In this way, only when the area ratio of the subject image satisfies a<M<b and the center point of the subject is located inside the "discovery frame" and outside the "display frame", can the subject be determined to be in the discovery state. Similarly, only when the area ratio of the subject image satisfies b≤M<c or c≤M<d and the center point of the subject is located inside the "display frame" and outside the "interactive frame", can the subject be determined to be in the display state. If the area ratio of the subject image satisfies d≤M<e and the center point of the subject is located inside the "interactive frame", the subject can be determined to be in the interactive state, and so on.

[0106] It should be noted that the determination of the subject's position and state can be performed synchronously with the subsequent information card generation and display process. That is, the client continuously determines the changes in the subject's position and state, and based on these changes, determines which type of information card to display, when to switch information card types, and how to track the motion of the information cards, etc. Of course, the determination of the subject's position and state does not have to be performed every frame; for example, it can be performed every few frames (e.g., four frames, five frames, etc.), etc., which will be discussed in detail later.

[0107] S202: Obtain search results for product information related to the identified subject.

[0108] During the process of detecting a subject from a real-time image stream, search results for product information related to the identified subject can also be obtained. Specifically, this can be achieved by requesting specific product information search results from the server, or, if this is not the first time a search has been performed on the current subject, by retrieving product information search results from the local cache of the client's terminal device, and so on. The specific product information search results can be multiple products matching the currently identified subject image.

[0109] In practical implementation, as mentioned earlier, the client-side algorithm can assess the identified subject to determine if it meets the first condition. If it does, specific product search results are retrieved. For example, the image size and clarity of the identified subject can be assessed. If the image size and clarity of the subject meet the first condition, product search results related to the identified subject can be retrieved. Furthermore, as the image size and clarity of the identified subject gradually improve, if the image size and clarity of the subject meet the second condition, product search results related to the identified subject can be retrieved again. This allows for updating the content displayed in the information card based on the re-obtained product search results, and so on.

[0110] In specific implementation, if the client-side algorithm determines at a certain moment that a subject image meets the first or second condition, it can extract the subject image from the corresponding image frame in the real-time image stream and submit it to the server. Furthermore, since the client-side algorithm can also identify the subject's ID, category, etc., when submitting the subject image, the identification results regarding the subject's ID, category, etc., can also be submitted to the server. The server can then use this information to search for product results related to that subject image. The returned product search results can also include specific subject ID information.

[0111] Furthermore, after receiving the product search results returned by the server, the client-side algorithm can compare the similarity between the product images in the server-returned search results and the identified subject image, and filter the search results based on the comparison results. In other words, the client-side algorithm obtains the most direct information about the subject image. This information may be partially lost during transmission and encoding / decoding, which could lead to inaccuracies in the product search results provided by the server. Therefore, after the server returns the product search results, the client-side algorithm can perform secondary verification or filtering to improve the quality of the information presented in the information card.

[0112] S203: Generate an information card to be displayed based on the product information search results and the status of the subject, wherein different subject statuses correspond to different information card types.

[0113] After obtaining product information search results, specific information cards can be generated based on these results and the subject's status. Regarding the subject's status, as mentioned earlier, it can be determined based on the area proportion of the subject image, or by combining the coordinates of the subject's center point, etc. Specifically, in the display state, information cards to be displayed can be generated based on the product information search results. The display state can be further divided into a first display state, a second display state, etc., each corresponding to different types of information cards. Therefore, information cards to be displayed can be generated in real time based on the product information search results and the subject's status. The types of information cards can be mainly categorized by card size and content richness. For example, when the subject first enters the display state, the information card can be relatively small, displaying relatively little information. Later, as the subject image increases in size, a larger information card can be used, displaying more information.

[0114] Regarding the content of the information card, in its display state, it can specifically come from the product search results returned by the server. The product search results may include multiple products, but the area of ​​the information card is limited. Therefore, only one product can be selected for display; for example, it could be the information about the product with the highest matching degree to the main image, and so on. Alternatively, in the specific implementation, product switching options can be provided within the information card, allowing users to switch between displaying information about other products, and so on.

[0115] In short, information cards can come in various types. As the area occupied by the main image increases, the information card can vary in size, and the information displayed can become more comprehensive. For example, ... Figure 3As shown in (B), assuming the subject of the photo is a pair of headphones, when the main image occupies 40% (or another value), a smaller information card can be displayed. This smaller information card can include only the product name, reference price, and representative user reviews, etc. Figure 3 As shown in (C), when the main image is enlarged, for example, when the image occupies 60% (or other values), a larger information card can be displayed. This card can include not only the information shown in the smaller card but also richer information such as product performance details. Furthermore, when displaying information about a specific product in the search results within a specific information card, in addition to the product description information mentioned above, marketing information can also be included. For example, if a merchant has configured a coupon for a product, this information card can be presented to the user, allowing them to learn about the specific product's discount information before entering the search results page, thus improving click-through rates, conversion rates, and other metrics. Alternatively, since the subject of the photograph is a physical product, possibly one that the user has already purchased, merchants can also configure user benefits such as "buy one get one free" for certain special products (e.g., beverages). In this case, information about these user benefits can be provided through an information card, allowing users to access these benefits by clicking on the card, thereby enhancing interactivity, and so on.

[0116] Of course, as mentioned earlier, in practical implementation, the identified subject can also include the discovery state. That is, at this point, the area of ​​the subject image is very small, insufficient to accurately provide a precise product search result, but the category of the object can be roughly identified. Therefore, a corresponding information card can also be generated based on this category information. For example, ... Figure 3 As shown in (A), recognition begins when the subject image occupies 20% of the image. However, at this point, the displayed information card may only show the object category information, etc. Of course, in practical applications, when the subject is in the detection state, the initially identified object category may be incorrect. Subsequently, as the subject image gradually increases in size, the category recognition result can be corrected, and the content displayed in the information card can be updated.

[0117] In addition, the identified subject can also include the interactive state. In this case, the subject image occupies a large area, the image is clear enough, and an information card for the interactive state can be displayed. Alternatively, the information display format can be changed. For example, instead of displaying information cards, a 3D model of the product can be mapped onto a real-time information stream. There are various specific interaction methods. For instance, one approach is to redirect to a search results list page after entering and maintaining an interactive state for a certain period (e.g., 3 seconds), or even redirect to the details page of the currently displayed product, and so on.

[0118] S204: Map the information card to the location of the corresponding subject in the real-time image stream for display.

[0119] After generating specific information cards, they can be mapped to the location of the subject in the real-time image stream for display. Since the subject's position is constantly changing, the information cards need to follow the subject's movement. During this movement, the information cards may collide with screen edges or other information cards. Furthermore, when the subject's state changes, it involves switching the information card type, and so on. This application provides solutions to these issues in its embodiments.

[0120] In the process of moving the information card along with the subject, the algorithm can perform subject recognition for each frame in the image stream. The subject's position may change in each frame, and the identified subject positions are multiple discrete points. Therefore, if the information card's position is directly updated based on the identified subject positions in each frame, the information card's movement trajectory may not be smooth enough. Furthermore, this poses a significant challenge to client performance.

[0121] Therefore, a segmented fitting implementation scheme is adopted in this embodiment. Specifically, the client can save the algorithm recognition results. During the process of the information card moving along with the change in the subject's position, the latest position of the recognized subject can be read at preset time intervals. This time interval can be related to the persistence of vision in the human eye and is greater than the time interval between each frame in the image stream. For example, in a specific implementation, this time interval can be 0.35 seconds, that is, the latest position of the subject is read every 0.35 seconds. After reading the latest position, an animation can be generated for the currently displayed information card to smoothly move from its current position to the latest position. Then, the position of the information card can be kept at this latest position until a new latest position is read in the next cycle, at which point a new animation is generated.

[0122] For example, such as Figure 4The timeline shown illustrates an information card that begins displaying at time A. Initially, based on the main subject's position at time A, the information card is displayed at that position without animation. Then, an animation update is performed every n1 seconds. For example, the time interval between time A and time B is n1 seconds. During this period, although the algorithm updates the main subject's position multiple times, and the information recorded on the client side is also continuously updated, the animation remains unchanged; the information card remains at the position it was at time A. When time B arrives, the most recently updated main subject position is used to initiate the information card's movement. Specifically, this movement could involve a translational animation lasting n1 seconds. That is, the information card smoothly moves from its position at time A to its position at time B along a straight line. Afterward, the information card remains at its position at time B until time C arrives n1 seconds later, at which point it moves again using a translational animation lasting n1 seconds, and so on. In other words, in this method, the movement trajectory of the information card is divided into multiple segments of linear motion. These segments are connected to fit the main movement trajectory, making the movement of the information card smoother and reducing stuttering. Furthermore, the multi-segment translational animation is also more performance-friendly for the client.

[0123] Regarding the issue of information cards colliding with the screen edge, since information cards are usually rectangular, they may collide with the screen edge while following the movement of the main image, resulting in part of the information card's content being obscured and unable to be displayed.

[0124] To address this issue, in this embodiment, the relative positional relationship between the information card and the main anchor point (which can be determined based on the position of the main center point; for example, the main center point can be used directly as the anchor point, or the anchor point and the main center point can be on the same horizontal or vertical straight line, etc.) can be defined in advance as various types. For example, two different relative positional relationship types can be defined: a first relative positional relationship and a second relative positional relationship. Specifically, they can be lower right and upper left, respectively. That is, when displaying the information card, the information card can only appear in the lower right or upper left position of the anchor point. Thus, as the information card moves with the change in the position of the main body, it can be determined whether the information card will collide with the edge of the screen in the current relative positional relationship state. If so, it can be switched to another relative positional relationship. For example, the information card was originally in the lower right position of the main anchor point, but as the main body moves, the information card may collide with the right edge of the screen, so that part of the information in the information card may be obscured. At this time, the information card can be switched to the upper left position. In determining whether a collision will occur with the screen edge, assuming the current information card is displayed to the lower right of the main anchor point, the main determination is whether the information card will collide with the right edge of the screen. Specifically, the X-axis coordinate of the main center point can be added to the length of the card, and the result can be determined whether it exceeds the right edge of the screen, etc.

[0125] Of course, in actual implementation, if it is found that the information card will exceed the right edge of the screen in the current relative position relationship, it may also exceed the left edge of the screen after switching to other relative positions of the main anchor point. Therefore, before switching, it is also possible to determine whether the information card will exceed the left edge of the screen after switching to other relative positions of the main anchor point. If so, there is no need to switch; otherwise, switch.

[0126] In the aforementioned implementation method of segmented smooth fitting of the motion trajectory of the information card, the latest position of the subject can be read every n1s, and after moving the information card, it can be determined whether the information card will exceed the edge of the screen.

[0127] In addition, such as Figure 3 As shown at point 31 in (B), during the display of information cards, specific anchor elements can also be displayed on the main image. When switching relative positions, the anchor elements can be hidden first, and then made visible again after the relative position switch is complete. This makes the switching process more aesthetically pleasing.

[0128] The above describes solutions for collisions between information cards and screen edges. In practical applications, multiple subjects may be identified in the same image stream. In this case, if the corresponding product information search results are obtained separately and information cards of the corresponding type are generated according to their respective states, collisions may occur between the multiple information cards during the real-time mapping of the multiple information cards onto the real-time image stream and their movement following the position changes of their respective subjects. That is, one information card may obscure the content of another information card.

[0129] To address this situation, this embodiment of the application can further determine whether a collision will occur between multiple information cards. If so, the obstruction can be eliminated by moving one or more information cards. Regarding which specific information card to move, this embodiment can adjust the display position of the information card corresponding to the lower-priority subject based on the different priority levels of the colliding information cards. In other words, since this embodiment defines multiple different subject states, and these states are affected by the subject's proximity to the lens, multiple subjects in the same image stream may be in different states. Furthermore, subjects closer to the lens (with a larger proportion of their image area) are more likely to be the objects the user most wants to search for, and therefore, their corresponding information cards can be displayed with higher priority. This allows for the sorting of different subject states according to priority, and when a collision is detected between two information cards, the information card corresponding to the lower-priority subject can be moved.

[0130] It's important to note that if a card is moved, it might collide with the screen edge, causing the edge to obscure the card and preventing further movement. Therefore, to mitigate the impact of collisions between different cards (primarily obscuring content), multiple layers can be created. Each layer displays cards corresponding to a specific theme state, and these layers can be prioritized according to the theme state. For example, the first layer could hold cards for interactive states, the second for display states (large cards), the third for discovery states (small cards), and so on. This way, when a theme is in a particular state, the corresponding card is displayed in the layer for that state. Layer switching can also be performed simultaneously when displaying different types of cards for the same theme. For example, if a subject is previously in the first display state and the information card type is a "small card" for the display state, then the information card can be displayed in the aforementioned third layer; at some point, the subject's state changes to the second display state and the information card type switches to a "large card" for the display state. At this time, the "large card" can be mapped to the second layer in the image stream, and so on.

[0131] In this way, information cards corresponding to different subject states are displayed on different layers, with higher-priority subject states appearing on the uppermost layer. This ensures that if information cards corresponding to two different subject states collide, the higher-priority card remains on the upper layer. Furthermore, even if collisions occur and lower-priority cards cannot be moved, it prevents higher-priority cards from being obscured by lower-priority cards.

[0132] Furthermore, since this application embodiment defines multiple different subject states, different subject states can correspond to the generation of different types of information cards. Moreover, the subject state of the same subject in the image stream usually changes. Therefore, the specific display of information cards also involves the issue of switching information card types. That is, during the process of mapping the information card to the location of the corresponding subject in the real-time image stream for display, if the subject state changes, the type of information card mapped to the real-time image stream will be switched. For example, if at a certain moment a smaller-sized information card is displayed for an entity, and later, the entity's image area increases and the image becomes clearer, then a larger-sized information card can be switched. For example, as... Figure 3 As shown in (B) and (C).

[0133] Wherein, the switching of information card types is triggered by a change event of the subject state, and the change of the subject state is related to the change of the area proportion of the subject image. For example, as described above, the area proportion is divided into four intervals, each interval corresponds to one subject state, and if a certain definite value is used as the boundary between various subject states, the "jitter" of the information card may occur at the boundary between two subject types. For example, assuming that when the area proportion of the subject image satisfies b≤M<c, it is a first display state, corresponding to an information card with a smaller size (referred to as a "small card" for convenience of description); when the area proportion of the subject image satisfies c≤M<d, it is a second display state, corresponding to an information card with a larger size (referred to as a "large card" for convenience of description). If the area proportion of the subject image repeatedly changes near the value c (this may be caused by hand jitter when the user holds a mobile terminal), for example, assuming c=0.5, M may change back and forth between 0.49 and 0.51. In this case, the phenomenon that the "small card" and the "large card" are rapidly and repeatedly switched multiple times in the image stream may occur, and this phenomenon can be referred to as the "jitter" phenomenon of the information card.

[0134] In order to solve the "jitter" phenomenon occurring during the switching of information card types, the embodiment of the present application adopts a method of adding a buffer at the boundary of each subject state. For example, under normal circumstances, the division method of each subject state can be as follows:

[0135] a < M < b is the discovery state;

[0136] b ≤ M < c is the "small card" display state;

[0137] c ≤ M < d is the "large card" display state;

[0138] d ≤ M < e is the interactive state.

[0139] However, in order to solve the above "jitter" phenomenon during the switching of information card types, a buffer n can be added at the boundary, and in this case, the division method of each subject state can be as follows:

[0140] a < M < b is the discovery state;

[0141] b+n ≤ M < c is the small card display state;

[0142] c+n ≤ M < d is the large card display state;

[0143] d+n ≤ M < e is the interactive state.

[0144] This allows for a buffer of size "n" at the boundary between two different subject states, instead of using a simple numerical value as the boundary. In this way, if a user experiences "hand tremor" at the boundary between two states of a subject during image stream acquisition, as long as the change in the subject image area caused by the tremor is within "n", the information card type switch will not be triggered, thus solving the information card "jitter" problem at subject type boundaries. Of course, in practical implementation, other judgment parameters can be used to determine the subject state, such as the size and clarity of the subject image.

[0145] Furthermore, during the specific process of switching information card types, since it involves switching from a first-type information card to a second-type information card—two different information cards—it's essentially replacing the first-type information card with the second-type. Therefore, the switching process can be controlled. For example, in one implementation, the switching process can be divided into two stages: the first stage gradually reduces the transparency of the first-type information card currently projected in the real-time image stream (e.g., the transparency can be gradually reduced from 1 to 0, meaning the first-type information card gradually disappears); the second stage projects the second-type information card into the real-time image stream and gradually increases its transparency (e.g., the transparency can be gradually increased from 0 to 1, meaning the second-type information card gradually appears), and so on.

[0146] It's worth noting that while displaying product information through information cards, these cards can also facilitate actions such as adding items to a bulk checkout cart. Specifically, on mobile devices like smartphones, the information cards can offer an option to add displayed items to the cart. This allows users interested in a product to add it directly to their cart without having to navigate to the search results page and then the product details page, thus shortening the user's workflow.

[0147] Alternatively, the solution provided in this application can be used not only in mobile terminals such as smartphones but also in AR glasses devices. That is, the functions provided in this application can be implemented in related applications on the AR glasses device. In this way, when a user is browsing offline physical stores such as shopping malls while wearing AR glasses, they can enter the "real-time image stream mode" through the entry point provided by the application. At this time, they can use the camera in the AR glasses to capture the scene. During the acquisition of the dynamic image stream, subject recognition and switching display of information cards in different subject states are possible, with the information cards following the movement of the subject, etc. Furthermore, in this scenario, one advantage of AR glasses over mobile terminals such as smartphones is that AR glasses do not require handheld use. During the acquisition of the image stream, the user's hands are freed. Simultaneously, the hands can perform some control operations on the screen content through the handle associated with the AR glasses, or by making some gestures, etc. Therefore, in this AR glasses scenario, the operation of adding items to the "shopping cart" does not need to provide a corresponding add-to-cart button on the information card. Instead, it can be achieved by detecting the user's grasping action on the information card and moving it in a target direction (e.g., towards the user's chest, etc.). In other words, while "viewing" an object in a location such as a product through AR glasses, search results for related products can be displayed as information cards, which can be viewed through the AR glasses. If a user is interested in a product displayed on the information card, they can use a controller or gesture to grab the card and move it towards their chest, triggering an add-to-cart action for that product. This interactive method of "reaching out to grab" the information card further enhances the fun of the interaction and user engagement.

[0148] It should be noted that in the aforementioned AR glasses scenario, the specific coordinates of the identified subject can be three-dimensional coordinates, meaning they can include depth information. During the mapping of the information card to the image stream, the display depth of the specific information card can also change following the subject image. Furthermore, when the user performs "grabbing" and moving operations on the information card using gestures, the hand image can be recognized from the image stream. Subsequently, the state of the "grabbed" information card can be switched to follow the movement of that hand image, and so on.

[0149] In summary, through the embodiments of this application, subject recognition can be performed from the real-time image stream during the acquisition of the subject. Furthermore, the position and state changes of the subject across multiple image frames can be obtained, where the specific state may be related to the clarity of the subject portion of the image and / or its proportion within the image frame. During this real-time image stream acquisition process, search results for product information related to the identified subject can also be obtained. Based on the search results and the subject's state, information cards to be displayed are generated and mapped to the corresponding position of the subject in the real-time image stream. This allows users to not only view the content actually captured by the camera in the image stream but also to view these information cards. Moreover, the type of information card can change with the subject's state. This allows for the display of product search results on a "lens page" via AR, improving search efficiency. Simultaneously, the changing information card types allow users to gradually obtain richer and more detailed information about the searched products, making the information cards more engaging and enhancing user participation.

[0150] In addressing the issue of discontinuous motion trajectories encountered by the information card while following the main body during the implementation of the aforementioned solution, a preferred embodiment of this application can resolve this problem through segmented fitting. Specifically, the main body position information identified by the algorithm can be read periodically. Within the same period, the information card's position remains unchanged until the next period arrives and the latest main body position is read, at which point the information card "starts moving." Specifically, a translational animation of a target duration (equal to the length of the reading period) can be generated for the information card. In this animation, the information card can exhibit continuous motion, thus resolving the problem of discontinuous motion trajectories and reducing the consumption of terminal computing and display resources.

[0151] To address the issue of collisions between information cards and screen edges, two relative positional relationships between the information card and the center point of the main body can be preset. When such collisions occur, the problem can be resolved by switching the relative positional relationships.

[0152] To address the collision issue between information cards, the priority relationship between different subject states can be used to determine the priority of different types of information cards. This allows lower-priority information cards to be identified and moved, preventing them from obscuring each other. Alternatively, multiple layers can be created, displaying information cards corresponding to different subject states on different layers. The hierarchical relationship between these layers can also be fixed according to the priority relationship between subject states; for example, interactive information cards can be permanently displayed on the top layer. This way, even if a certain information card cannot be moved, higher-priority information cards will not be obscured by lower-priority ones.

[0153] To address the potential "jitter" issue that may occur when switching information card types for the same entity during a change in state, a buffer is provided at the boundary of the judgment parameters for different entity states. This ensures that when the judgment parameters change within the buffer, the current entity state and the corresponding information card type remain unchanged.

[0154] In addition, the specific subject state can also include an interactive state, in which interactive information can be provided to the user. For example, if the state is maintained for a certain period of time, the user can be automatically redirected to the search results list page or the details page of a specific product. Alternatively, a 3D model of a specific product can be mapped onto a real-time image for display, and so on.

[0155] Furthermore, in 3D glasses scenarios, the interactive method of adding products to the cart by having users "reach out and grab" information cards can be combined to further enhance the fun of the interaction and user participation.

[0156] Example 2

[0157] In this second embodiment, primarily targeting the aforementioned 3D glasses scenario, a method for providing product information is provided. The execution entity for each step of this method can be an application client on the 3D glasses. For details, see [link to documentation]. Figure 5 The method may include:

[0158] S501: During the process of acquiring a real-time image stream of the subject through augmented reality (AR) glasses, subject recognition is performed from the real-time image stream;

[0159] S502: Obtain search results for product information related to the identified subject;

[0160] S503: Generate an information card to be displayed based on the product information search results, and map it to the location of the corresponding subject in the real-time image stream for display;

[0161] S504: When the operation of grabbing the information card and moving it in the target direction is detected from the real-time image stream, the product displayed therein is added to the batch settlement product set.

[0162] For the parts of this embodiment that are not described in detail, please refer to the description in embodiment one and other parts of this application specification, which will not be repeated here.

[0163] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0164] Corresponding to Embodiment 1, this application also provides an apparatus for providing product search result information, see [link to embodiment]. Figure 6 The device may include:

[0165] The subject recognition unit 601 is used to perform subject recognition from the real-time image stream during the real-time image stream acquisition of the subject, and to obtain the changes in the position and state of the subject in multiple image frames, wherein the state is related to the position of the subject in the image frame and / or the area ratio of the subject image on the screen.

[0166] The product search result acquisition unit 602 is used to acquire product information search results related to the identified subject;

[0167] The information card generation unit 603 is used to generate information cards to be displayed based on the product information search results and the status of the subject, wherein different subject statuses correspond to different information card types;

[0168] The information card display unit 604 is used to map the information card to the location of the corresponding subject in the real-time image stream for display.

[0169] Specifically, the request unit can be used for:

[0170] The image size and clarity of the identified main body are judged. If the image size and clarity of the main body meet the first condition, search results for product information related to the identified main body are obtained.

[0171] In addition, the request unit can also be used for:

[0172] As the image size and clarity of the identified main body gradually improve, if the image size and clarity of the main body meet the second condition, then the search results for product information related to the identified main body are re-acquired, and the content displayed in the information card is updated according to the search results for product information obtained from the re-request.

[0173] In addition, the device may also include:

[0174] The filtering unit is used to compare the similarity between the product images in the obtained product information search results and the identified subject images, and to filter the product information search results based on the comparison results.

[0175] Specifically, the device may also include:

[0176] The reading unit is used to read the latest position of the identified subject at preset time intervals as the information card moves along with the position of the subject; wherein the time interval is related to the persistence of vision of the human eye and is greater than the time interval between each frame in the image stream.

[0177] An animation generation unit is used to generate an animation for the currently displayed information card to smoothly move from the current position to the latest position after reading the latest position, and to keep the position of the information card unchanged at the latest position until a new latest position is read in the next cycle, at which time a new animation is generated.

[0178] In addition, the relative positional relationship between the information card and the anchor point of the main body can be varied; the position of the anchor point is determined based on the center point of the main body.

[0179] At this point, the device may also include:

[0180] The relative position relationship switching unit is used to determine whether the information card will collide with the edge of the screen in the current relative position relationship state during the process of the information card moving with the change of the main body position. If so, it switches to other relative position relationships.

[0181] The device may further include:

[0182] The judgment unit is used to determine whether the information card will collide with another screen edge after switching to another relative position relationship before switching to another relative position relationship. If not, the information card will be switched to another relative position relationship; if so, the information card will remain in the current relative position relationship.

[0183] In the real-time image stream, anchor point elements are also displayed at the location of the main anchor point;

[0184] At this point, the device may also include:

[0185] An anchor element display processing unit is used to hide the anchor element during the process of switching relative position relationships, and to switch the anchor element to a visible state after the switch of relative position relationships is completed.

[0186] If multiple entities are identified in the real-time image stream, their respective product information search results are obtained, and information cards of the corresponding type are generated according to their respective states.

[0187] At this point, the device may also include:

[0188] The collision determination unit is used to determine whether a collision will occur between the multiple information cards during the process of mapping multiple information cards into the real-time image stream in real time and moving with the position changes of their respective corresponding subjects.

[0189] The card position moving unit is used to move the display position of the information card corresponding to the lower priority subject according to the different subject status priorities of the information card that collides.

[0190] The information cards corresponding to the different subject states can be displayed in different layers, and the layers are arranged in descending order of priority of each subject state.

[0191] In addition, the device may also include:

[0192] The card switching display unit is used to switch the type of information card mapped to the real-time image stream if the state of the subject changes during the process of real-time mapping of the information card to the location of the corresponding subject in the real-time image stream.

[0193] There is a buffer at the boundary of the judgment parameters for different subject states, so that when the judgment parameters change within the buffer, the current subject state and the type of the corresponding information card remain unchanged.

[0194] Specifically,

[0195] During the card type switching process, the switching process can be divided into two stages. The first stage is used to gradually reduce the transparency of the first type of information card currently projected in the real-time image stream. The second stage is used to project the second type of information card into the real-time image stream and gradually increase the transparency of the second type of information card.

[0196] The subject state is determined based on the subject's position and / or the area ratio of the subject image on the screen.

[0197] The state of the subject includes a display state, which includes a first display state and a second display state; wherein, in the display state, the information card is used to display content related to the product search results; the size and content of the information card corresponding to the first display state and the second display state are different.

[0198] The subject's state also includes a discovery state, and the information card in the discovery state is used to display the identification result of the category to which the subject belongs.

[0199] The state of the subject also includes the interaction state;

[0200] The device may further include:

[0201] The interactive triggering unit is used to redirect the user to the product search results list page or the product details page included in the currently displayed information card after the user enters the interactive state and maintains it for a threshold time.

[0202] Alternatively, after the subject enters the interactive state, the 3D model of the product is projected onto the real-time image stream for display.

[0203] In a specific implementation, the device may further include:

[0204] The add-to-cart option providing unit is used to provide operation options in the information card for adding the displayed products to the bulk checkout product collection.

[0205] Alternatively, the device may also include:

[0206] An operation detection unit is used to add the displayed products to a batch checkout product set when it detects an operation performed by a user in the real-time image stream of the subject captured by augmented reality (AR) glasses, in which the user grabs the information card and moves it in the target direction.

[0207] Corresponding to Embodiment 2, this application also provides a device for providing product information, see [link to embodiment]. Figure 7 The device may include:

[0208] The subject recognition unit 701 is used to perform subject recognition from the real-time image stream during the process of acquiring a real-time image stream of a photographed object through augmented reality (AR) glasses.

[0209] The product search result acquisition unit 702 is used to acquire product information search results related to the identified subject;

[0210] The information card display unit 703 is used to generate an information card to be displayed based on the product information search results, and map it to the location of the corresponding subject in the real-time image stream for display.

[0211] The product adding unit 704 is used to add the products displayed therein to the batch settlement product set when it detects from the real-time image stream that the user has performed an operation of grabbing the information card and moving it in the target direction.

[0212] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0213] And an electronic device, comprising:

[0214] One or more processors; and

[0215] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0216] in, Figure 8 The architecture of an electronic device is illustrated by example. For example, device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, aircraft, etc.

[0217] Reference Figure 8 The device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0218] Processing component 802 typically controls the overall operation of device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods provided in this disclosure. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0219] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0220] Power supply component 806 provides power to various components of device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 800.

[0221] Multimedia component 808 includes a screen that provides an output interface between device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0222] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0223] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0224] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of device 800, changes in the position of device 800 or a component of device 800, the presence or absence of user contact with device 800, the orientation or acceleration / deceleration of device 800, and temperature changes of device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0225] Communication component 816 is configured to facilitate wired or wireless communication between device 800 and other devices. Device 800 can access wireless networks based on communication standards, such as WiFi, or mobile communication networks such as 2G, 3G, 4G / LTE, and 5G. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0226] In an exemplary embodiment, device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0227] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of device 800 to perform the method provided by the present disclosure. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0228] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0229] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0230] The method and electronic device for providing product search results information provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for providing product search result information, characterized in that, include: During the process of acquiring real-time image streams of the subject through augmented reality (AR) glasses, subject recognition is performed from the real-time image stream, and the changes in the position and state of the subject in multiple image frames are obtained. The state includes discovery state, display state, or interactive state. The changes in the state are related to the position of the subject in the image frame and / or the area ratio of the subject image on the screen. Obtain search results for product information related to the identified entity; Information cards to be displayed are generated based on the product information search results and the subject's status. Different subject statuses correspond to different information card types. Different information card types have different sizes and content. In the discovery state, the information card is used to display the identification result of the category to which the subject belongs. In the display state, the information card is used to display content related to the product search results. In the interactive state, the information card is used to display content related to product details. The information card is mapped to the location of the corresponding subject in the real-time image stream for display. When the operation of grabbing the information card and moving it in the target direction is detected from the real-time image stream, the displayed product is added to the batch checkout product set.

2. The method according to claim 1, characterized in that, Also includes: As the information card moves along with the changing position of the subject, the latest position of the identified subject is read at preset time intervals; wherein, the time interval is related to the persistence of vision of the human eye and is greater than the time interval between each frame in the image stream; After the latest position is read, an animation is generated for the currently displayed information card to smoothly move from the current position to the latest position, and the position of the information card is kept unchanged at the latest position until a new latest position is read in the next cycle, at which time a new animation is generated.

3. The method according to claim 1, characterized in that, The relative positional relationship between the information card and the anchor point of the main body can be varied; the position of the anchor point is determined based on the center point of the main body. The method further includes: As the information card moves along with the change in the position of the main body, it is determined whether the information card will collide with the edge of the screen in the current relative position relationship. If so, the relative position relationship is switched to another.

4. The method according to claim 1, characterized in that, If multiple subjects are identified in the real-time image stream, their respective product information search results are obtained, and information cards of the corresponding type are generated according to their respective states. The method further includes: In the process of mapping multiple information cards into the real-time image stream in real time and moving according to the position changes of their respective subjects, it is determined whether collisions will occur between the multiple information cards. If so, the display position of the information card corresponding to the subject with the lower priority will be moved according to the different subject status priorities of the information card that collides.

5. The method according to claim 4, characterized in that, The information cards corresponding to the different subject states are displayed in different layers, and the layers are arranged in descending order of priority of each subject state.

6. The method according to claim 1, characterized in that, Also includes: During the process of mapping the information card to the location of the corresponding subject in the real-time image stream for display, if the subject's state changes, the type of information card mapped to the real-time image stream will be switched.

7. The method according to claim 6, characterized in that, There is a buffer at the boundary of the judgment parameters for different subject states, so that when the judgment parameters change within the buffer, the current subject state and the type of the corresponding information card remain unchanged.

8. The method according to claim 1, characterized in that, The display states include a first display state and a second display state; wherein the size and content of the information cards corresponding to the first display state and the second display state are different.

9. The method according to claim 8, characterized in that, The method further includes: After the subject enters the interactive state and remains in it for a preset time, it will be redirected to the product search results list page.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 9.

11. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 9.

Images