Information display, processing method, terminal equipment, server and storage medium

Abstract

The embodiments of the present application provide an information display and processing method, a terminal device, a server and a storage medium. In the information display method, an image of at least a portion of the item packaging body is displayed in the image display interface, and the code scanning guidance information is superimposed and displayed on the image display interface, so that the user can view the code scanning guidance information while scanning the item traceability code on the item packaging body through the image display interface. Based on this implementation, while the user is performing the code scanning operation, the code scanning guidance information is pushed to the user in real time, which improves the timeliness of the code scanning guidance method, and is conducive to the user to quickly and accurately determine the location of the item traceability code from the item packaging body, and accurately complete the code scanning tracing operation.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to an information display and processing method, a terminal device, a server and a storage medium. Background Art

[0002] Drug traceability refers to the process of monitoring the production and circulation of drugs through an electronic monitoring system. To facilitate traceability, drug manufacturers can print traceability codes on drug packaging boxes. Users can scan the traceability codes of drugs to obtain drug traceability information.

[0003] Usually, the drug traceability code is printed in a small local area on the outer packaging of the drug, and its style is similar to that of the product code, which makes it difficult for users to find the traceability code quickly and accurately. The existing method of guiding users to scan the traceability code is relatively simple. Therefore, a new solution needs to be proposed. Summary of the invention

[0004] Multiple aspects of the present application provide an information display and processing method, a terminal device, a server and a storage medium to flexibly guide users to trace drugs.

[0005] An embodiment of the present application provides an information display method, including: displaying an image of at least a portion of an article packaging body in an image display interface; and superimposing and displaying code scanning guidance information on the image display interface to guide a user to scan an article traceability code on the article packaging body through the image display interface.

[0006] An embodiment of the present application provides an information processing method, including: acquiring an image of at least a portion of an article packaging body sent by a terminal device; identifying the image to determine the article corresponding to the article packaging body; acquiring code scanning guidance information of the article from code scanning guidance information corresponding to a plurality of different articles; and sending the code scanning guidance information of the article to the terminal device for display, so as to guide a user to scan the article traceability code on the article packaging body.

[0007] An embodiment of the present application also provides a terminal device, including: a display component; the display component is used to: display an image of at least a portion of an item packaging body on an image display interface; and superimpose code scanning guidance information on the image display interface to guide a user to scan the item traceability code on the item packaging body through the image display interface.

[0008] An embodiment of the present application also provides a server, including: a processing component and a communication component; the communication component is used to: obtain an image of at least a portion of an article packaging body sent by a terminal device; the processing component is used to: identify the image to determine the article corresponding to the article packaging body; obtain the scanning guidance information of the article from the scanning guidance information corresponding to a plurality of different articles; and send the scanning guidance information of the article to the terminal device for display through the communication component to guide the user to scan the article traceability code on the article packaging body.

[0009] The embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed, can implement the information display method performed by the terminal device or the information processing method performed by the server provided in the embodiment of the present application.

[0010] In the information display method provided in the embodiment of the present application, an image of at least a portion of the item packaging is displayed in the image display interface, and code scanning guidance information is superimposed and displayed on the image display interface, so that the user can view the code scanning guidance information while scanning the item traceability code on the item packaging through the image display interface. Based on this implementation, while the user is performing the code scanning operation, the code scanning guidance information is pushed to the user in real time, which improves the timeliness of the code scanning guidance method, and is conducive to the user to quickly and accurately determine the location of the item traceability code from the item packaging, and accurately complete the code scanning tracing operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0012] Figure 1 A schematic diagram of the structure of an information display system provided for an exemplary embodiment of the present application;

[0013] Figure 2 This is a schematic diagram of obtaining the drug traceability code in the image display interface;

[0014] Figure 3a as well as Figure 3b A schematic diagram of an image of at least a portion of a drug package body captured in an image display interface;

[0015] Figure 4 A schematic diagram showing the image display interface and code scanning guidance information on the same screen;

[0016] Figure 5 A schematic diagram of scanning a drug traceability code according to the code scanning guidance information;

[0017] Figure 6 This is a schematic diagram of the drug traceability information display page;

[0018] Figure 7 A flowchart of an information display method provided for an exemplary embodiment of the present application;

[0019] Figure 8 A flowchart of an information processing method provided for an exemplary embodiment of the present application;

[0020] Fig. 9 A schematic diagram of the structure of a terminal device provided for an exemplary embodiment of the present application;

[0021] Fig.10 A schematic diagram of the structure of a server provided for another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0023] In order to facilitate the supervision of drug production and circulation through the electronic supervision system, drug manufacturers can usually print drug traceability codes on drug packaging boxes when producing drugs. Users can obtain drug traceability information through the traceability code. The drug traceability code can be implemented as an information code such as a QR code, barcode or 3D code, or can also be implemented as other forms of electronic certificates.

[0024] The drug traceability code is usually printed on one side of the medicine box. The area of ​​the drug traceability code is small and the style is similar to the product code. Therefore, most users find it difficult to find the location of the drug traceability code on the medicine box without guidance, which makes it difficult for users to scan the code for drug traceability and also creates certain obstacles to the promotion of the scan code traceability function.

[0025] At present, the method of guiding users to scan the code is usually to provide users with a page for guiding them to scan the code online for them to learn. However, on the one hand, the page is not easy to be noticed by users, and on the other hand, the guidance content provided on the page usually requires users to learn and memorize in advance, which is not conducive to users quickly adjusting the scanning position of the medicine box according to the guidance information.

[0026] In view of the above technical problems, a solution is provided in some embodiments of the present application. The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0027] Figure 1 A structural diagram of an information display system provided by an exemplary embodiment of the present application is shown in FIG. Figure 1 As shown, the information display system 100 includes a terminal device 101 and a server 102 .

[0028] The terminal device 101 refers to a device that can interact with a user and has a communication function. In different application scenarios, the implementation form of the terminal device 101 will be different. For example, in some scenarios, the terminal device 101 can be implemented as a mobile phone, a tablet computer, a computer device, a PDA (Personal Digital Assistant), etc. on the user side.

[0029] The server 102 includes a device that can provide data storage, support data calculation, and has a communication function. In some embodiments, the server 102 can be implemented as a conventional server, a cloud server, a cloud host, a virtual center, and other servers, and this embodiment does not limit this.

[0030] Based on the interaction between the terminal device 101 and the server 102, the information display system 100 can provide a drug traceability function. When the user scans the drug traceability code on the drug package with the terminal device 101, the information display system 100 can display the drug traceability information to the user; when the user does not accurately scan the drug traceability code with the terminal device 101, the information display system 100 can guide the user to scan the code correctly, and when the user accurately scans the drug traceability code, the information display system 100 responds to the user and displays the drug traceability information.

[0031] The terminal device 101 is mainly used to display an image of at least a portion of a drug package on an image display interface. The drug package may include at least one of an outer packaging box, an outer packaging bag, an outer packaging box, a close-fitting inner packaging bottle, an inner packaging bag, and an inner packaging box. When the drug is a capsule drug, the drug package may also include a capsule plate of the drug. Usually, a drug traceability code is printed on the drug package.

[0032] Among them, the image of at least a portion of the drug packaging body can be obtained by capturing the terminal device 101. The terminal device 101 can interact with the user to detect the code scanning operation initiated by the user. In response to the code scanning operation, the terminal device 101 can call up the camera to shoot, and display the image display interface on the camera shooting page. The image display interface is used to display the image captured by the camera. When the camera captures the drug packaging body, the terminal device 101 can display the captured image of at least a portion of the drug packaging body on the image display interface.

[0033] The image display interface may be implemented as a portion of the display area on the shooting page, or the entire display area on the shooting page, and this embodiment does not limit this. When the image display interface is implemented as a portion of the display area on the shooting page, the image display interface may be framed by a viewfinder to guide the user to aim at the object being photographed so that the object being photographed is within the viewfinder. In some embodiments, the image display interface framed by the viewfinder may be described as a scanning window on the shooting page, such as Figure 2 A scanning window 103 is shown.

[0034] Among them, the user who initiates the code scanning operation can be a user who has a demand for drug traceability, such as a drug purchasing user, a transport user in the drug transportation link, a drug regulatory user, etc.

[0035] In some cases, the terminal device 101 may display a page and display a code scanning control on the page, and the user may trigger the code scanning control to initiate the code scanning operation. In other cases, the terminal device 101 may monitor the user's voice command, and the user may initiate the code scanning operation by issuing a code scanning voice command, which is not limited in this embodiment.

[0036] In some cases, when a user needs to trace a drug and points the camera of the terminal device 101 at the drug traceability code, the terminal device 101 can capture the drug traceability code and display the drug traceability code in the image display interface, such as Figure 2 At this time, the terminal device 101 can decode the drug traceability code, send the decoded drug information to the server 102, and receive the drug traceability information returned by the server 102.

[0037] In other cases, when the user needs to trace the drug but does not aim the camera of the terminal device 101 at the drug traceability code, the terminal device 101 can capture an image of at least a portion of the drug package and display the image of at least a portion of the drug package in the image display interface.

[0038] For example, when the camera of the terminal device 101 is aimed at a side of the drug outer packaging box that does not contain the drug traceability code, the image display interface can display the image of the side of the drug packaging box, such as Figure 3a When the camera of the terminal device 101 is not aimed at a certain side of the medicine outer packaging box or the camera is close to the medicine outer packaging box, the terminal device 101 can capture a partial area of ​​a certain side of the medicine outer packaging box, and the image display interface can display a partial image of the side, such as Figure 3b shown.

[0039] In this case, the terminal device 101 can overlay and display code scanning guidance information on the image display interface to guide the user to scan the item traceability code on the item packaging through the image display interface.

[0040] Among them, superimposed display refers to displaying the code scanning guidance information and the image display interface on the same screen, so that the code scanning guidance information is displayed while displaying the image display interface.

[0041] Among them, the code scanning guidance information refers to the information used to guide users to scan the drug traceability code.

[0042] In some embodiments, the code scanning guidance information is provided by the server 102. In this embodiment, the terminal device 101 can obtain an image of at least a portion of the drug package in the image display interface, and send the image to the server 102.

[0043] After receiving the image of at least a portion of the drug package sent by the terminal device 101, the server 102 can recognize the image to determine the drug corresponding to the drug package. The operation of the server 102 recognizing the corresponding drug based on the image of at least a portion of the drug package can be implemented based on deep learning, which will be described in detail in subsequent embodiments and will not be repeated here.

[0044] The server 102 stores a plurality of different code scanning guidance information corresponding to different drugs. Based on this, after determining the drug corresponding to the drug package, the server 102 can obtain the code scanning guidance information of the drug from the code scanning guidance information corresponding to the various different drugs, and send the code scanning guidance information of the drug to the terminal device 101.

[0045] After receiving the scanning guidance information of the drug traceability code returned by the server 102, the terminal device 101 can overlay and display the scanning guidance information on the image display interface. Furthermore, the terminal device 101 can guide the user to scan the drug traceability code on the drug package through the image display interface through the scanning guidance information.

[0046] In the above process, the terminal device 101 obtains an image of at least a portion of the drug package, and the process of identifying the corresponding drug based on the image can be implemented silently, without the user's participation, and the drug identification operation can be completed without the user's perception. In addition, drug identification can be achieved based on at least a portion of the image of the drug package, and the user does not need to completely aim the camera at the drug package, which reduces the requirements for user operation.

[0047] Based on the above, the code scanning guidance information and the image display interface for scanning the drug traceability code can be displayed on the same screen. The user can watch the code scanning guidance information while adjusting the placement angle of the drug packaging body or the scanning angle of the terminal device 101 according to the content scanned by the image display interface, so that the terminal device 101 can scan the drug traceability code.

[0048] In the information display system 100, the above data interaction process between the terminal device 101 and the server 102 can be implemented based on the communication connection relationship established between the terminal device 101 and the server 102. The specific communication connection method can be determined according to the actual application scenario.

[0049] In some exemplary embodiments, the terminal device 101 and the server 102 may communicate with each other by wired communication or wireless communication. The wireless communication includes short-distance communication such as Bluetooth, ZigBee, infrared, WiFi (WIreless-Fidelity), long-distance wireless communication such as LORA, and wireless communication based on mobile network. When connected through mobile network communication, the network standard of the mobile network can be any one of 3d (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), 5G, WiMax, etc.

[0050] In this embodiment, based on the interaction between the terminal device 101 and the server 102, when the user initiates a code scanning operation, an image of at least a portion of the drug packaging body in the image display interface is obtained, and the scanned drug is identified based on the obtained image. After the drug is identified, the code scanning guidance information of the drug is superimposed and displayed on the image display interface, and then the user can perform the code scanning operation while watching the code scanning guidance information. Based on this implementation, while the user is performing the code scanning operation, the code scanning guidance information is pushed to the user in real time, which improves the timeliness of the code scanning guidance method, and is conducive to the user to quickly and accurately determine the location of the drug traceability code from the drug packaging body, and accurately complete the code scanning traceability operation.

[0051] The above embodiment records that the server 102 can recognize at least a portion of the image of the drug package to determine the drug corresponding to the drug package. The following will exemplify the specific optional implementation of the server 102 identifying the drug based on the drug package.

[0052] In some exemplary embodiments, the server 102 may implement drug identification based on a deep learning method.

[0053] Optionally, the deep model may include: a subject region segmentation module and a subject recognition module. Subject segmentation refers to identifying the region in the image that may contain an object as the subject region, and segmenting the subject region from the background region in the image to extract the subject region. Image subject recognition refers to identifying the category to which the subject belongs based on the image features of the subject region.

[0054] Optionally, in order to enable the deep learning model to learn to perform subject area segmentation and subject recognition on the input image, the deep learning model can be trained in advance based on a large amount of training data.

[0055] The training data may include: image samples corresponding to a plurality of different drug packages. Before training, the training data may be labeled, that is, the type of drug corresponding to the image sample, the background area, and the main body area corresponding to the drug package are labeled on each image sample.

[0056] Next, the labeled training data is input into the main region segmentation module of the deep learning model. After the training data is input into the main region segmentation module, the main region segmentation module can extract the image features in each image sample, and predict the main region and background region of each image sample based on the current model parameters of the main region segmentation module. After obtaining the prediction results of the main region segmentation module for the image sample, the main region and background region labeled on the image sample can be used as supervision signals to calculate the prediction loss of the main region segmentation module. Based on the prediction loss, the model parameters of the main region segmentation module can be adjusted. Repeat the above process to iteratively train the main region segmentation module until the prediction loss of the main region segmentation module is less than the set threshold.

[0057] After completing the training of the main body region segmentation module, next, the main body recognition module may be trained based on the output result of the main body region segmentation module.

[0058] Optionally, the main body area and background area of ​​the image sample predicted by the main body area segmentation module can be obtained, and the predicted main body area can be input into the main body recognition module. In the main body recognition module, a set feature extraction algorithm can be used to extract the image features of the main body area to obtain the image features corresponding to each main body area. Since the image sample is marked with the corresponding classification of the drug, the main body recognition module can establish the corresponding relationship between the classification of the drug and the image features after obtaining the image features corresponding to the main body area in each image sample, and save the corresponding relationship in the server 102.

[0059] Among them, the image feature extraction operation involved in the deep learning model can be implemented based on the deep feature extraction operation, or can be implemented based on the local feature extraction operation, or based on the combination of deep features and local features. Among them, the deep feature extraction operation can be implemented based on Convolutional Neural Networks (CNN), which will not be repeated here.

[0060] Based on the above, after training the main body region segmentation module and the main body recognition module, the server 102 can save the deep learning model locally for subsequent use. When the server 102 receives an image of at least a portion of the drug package sent by the terminal device 101, the image can be input into the pre-trained deep learning model. The deep learning model can extract features from the input image, and based on the pre-trained model parameters, predict the drug corresponding to the image, and output the prediction result.

[0061] Based on the above, when the server 102 receives an image of at least a portion of the drug packaging body sent by the terminal device 101, the image can be input into a pre-trained deep learning model. In the deep learning model, the main body area in the image is identified based on the main body area segmentation module, and the extracted main body area is extracted based on the main body recognition module to obtain the image features to be matched. Next, the server 102 can perform feature matching on the extracted image features to be matched with the pre-stored image features of different drugs, and determine the drug corresponding to the image features to be matched based on the results of the feature matching.

[0062] For example, the server 102 may compare the image feature to be matched with the image feature corresponding to each pre-stored drug to obtain the similarity between the image feature to be matched and the image feature corresponding to each pre-stored drug. Next, the drug whose similarity meets the set requirements is selected as the drug corresponding to the image feature to be matched.

[0063] Based on the above embodiment, the server 102 can accurately identify the drug corresponding to the drug package based on deep learning, which is conducive to providing users with accurate drug traceability code scanning guidance services.

[0064] In the above and following embodiments of the present application, the server 102 may pre-store code scanning guidance information corresponding to a plurality of drugs, and the code scanning guidance information may be stored corresponding to the drug identification. The drug identification may include at least one of: the drug name, the drug number, and the drug SPU (Standard Product Unit) information, which is not limited in this embodiment.

[0065] After the corresponding drug is identified based on the image of at least a portion of the drug packaging, the server 102 may determine the code scanning guidance information corresponding to the identified drug based on the correspondence between the drug identification and the code scanning guidance information.

[0066] Optionally, the code scanning guidance information may be at least one of a code scanning guidance video, a code scanning prompt guidance audio, a code scanning guidance step diagram, and a code scanning guidance dynamic image. Among them, the code scanning guidance step diagram may include multiple images.

[0067] Among them, in the preprocessing stage, a code scanning instruction video corresponding to each drug can be filmed, and the code scanning instruction video and the drug identification can be saved in the server 102 accordingly; each code scanning instruction video can be converted into a code scanning instruction dynamic image, and the code scanning instruction dynamic image and the drug identification can be saved in the server 102 accordingly; the packaging body of each drug can be filmed to obtain multiple images, and text annotations are performed on the multiple images to obtain step diagrams of code scanning tracing, and the step diagrams of code scanning tracing and the drug identification can be saved in the server 102 accordingly.

[0068] Among them, the code scanning guidance video and the code scanning guidance dynamic image can use continuous image frames to dynamically display at least one of the target surface containing the drug traceability code on the drug packaging, the relative position relationship between the target surface and the non-target surface of the drug packaging that does not contain the drug traceability code (such as the front side printed with the drug name), the position of the drug traceability code on the target surface, and the position of the non-drug traceability code (such as the product code) on the drug packaging. In addition, the placement angle of the drug packaging body, the adjustment method of the placement direction, etc. can also be dynamically displayed. For the code scanning guidance video, audio guidance information can also be added to the code scanning guidance video to guide the user's code scanning operation, which will not be repeated.

[0069] Among them, the code scanning prompt guidance audio can be obtained by manual pre-recording, or it can be obtained by voice conversion based on the pre-established code scanning guidance copy. The code scanning prompt guidance audio can describe at least one of the main features of the target surface containing the drug traceability code on the drug packaging body, the relative position relationship between the target surface and the non-target surface on the drug packaging body that does not contain the drug traceability code, and the position of the drug traceability code on the target surface by voice. Among them, the code scanning guidance step diagram information can use a discrete image sequence to display at least one of the target surface containing the drug traceability code on the drug packaging body, the relative position relationship between the target surface and the non-target surface on the drug packaging body that does not contain the drug traceability code, the position of the drug traceability code on the target surface, and the position of the non-drug traceability code (such as the product code) on the drug packaging body. In each step diagram, the position of the correct traceability code and / or the position of the incorrect traceability code can be marked with text, which will not be repeated.

[0070] In this embodiment, corresponding code scanning guidance information is prepared in advance for each drug, so that a highly targeted code scanning guidance operation can be achieved.

[0071] After the terminal device 101 receives the code scanning guidance information of the medicine returned by the server 102, the code scanning guidance information can be superimposed and displayed on the image display interface.

[0072] Optionally, the terminal device 101 may display a floating window on the page where the image display interface is located, and display code scanning guidance information in the floating window.

[0073] Optionally, when the terminal device 101 displays a floating window, the floating window may be displayed in other areas outside the image display interface to avoid blocking the viewing area. Figure 4 as well as Figure 5 As shown, the floating window can be displayed below the image display interface. Optionally, if the transparency of the floating window is high, the floating window and the image display interface can be appropriately superimposed, which is not limited in this embodiment.

[0074] Optionally, when displaying the code scanning guidance information, the terminal device 101 may display the target surface containing the drug traceability code on the drug package, display the relative position relationship between the target surface and the non-target surface of the drug package that does not contain the drug traceability code, display the position of the drug traceability code on the target surface, and display the position of the non-drug traceability code on the drug package. The above-mentioned code scanning guidance information can be displayed through at least one of a code scanning guidance video, a code scanning guidance step diagram, and a code scanning guidance dynamic image, which is not limited in this embodiment.

[0075] A typical way to display the code scanning guidance information is as follows Figure 4 as well as Figure 5 As shown. Figure 4 as well as Figure 5 In the illustration, a floating window is displayed below the scanning window. In the floating window, a code scanning instruction video is displayed. The code scanning instruction video can be played continuously to display the drug packaging box from multiple angles, and to display the side of the drug packaging box with the drug traceability code printed on it and mark the position of the drug traceability code. At the same time, a code scanning instruction step diagram can be displayed below the code scanning instruction video. In the code scanning instruction step diagram, multiple images are used to respectively display the front of the medicine box, the relative position relationship between the front of the medicine box and the side with the drug traceability code printed on it, a close-up of the side with the drug traceability code printed on it, as well as text annotations and the side with an incorrectly printed code.

[0076] exist Figure 4 In the illustration, the user has not yet accurately found the drug traceability code. By viewing the code scanning guidance information below the scanning window, the user can accurately find the location of the drug traceability code and adjust the angle of the drug packaging box. Then, the terminal device 101 can capture the drug traceability code with a camera, such as Figure 5 shown.

[0077] In this embodiment, based on the code scanning guidance video, code scanning guidance step diagram and code scanning guidance dynamic image, the operation method of determining the drug traceability code from the drug packaging can be demonstrated to the user in a visual way, making it easier for users to flexibly master the code scanning operation skills.

[0078] In addition, based on the method of displaying the scanning window and scanning guidance information on the same screen, on the one hand, the terminal device does not need to use other methods (such as setting up a special scanning learning page) to display the style of the drug traceability code and the position of the drug traceability code on the drug packaging; on the other hand, the user does not need to learn and memorize the style of the drug traceability code and the position of the drug traceability code on the drug packaging in advance, and only needs to directly scan the drug packaging to complete the scanning traceability operation, which is very convenient.

[0079] In addition, the scanning guidance information provided in this embodiment is highly targeted. The scanning guidance information displayed by the terminal device 101 corresponds to the medicine that the user is currently scanning. Therefore, the user can obtain the scanning guidance information more clearly, directly and accurately, thereby improving the efficiency of scanning traceability.

[0080] Optionally, when the user adjusts the position of the drug package so that the terminal device 101 can scan the drug traceability code, the terminal device 101 can obtain the drug traceability code in the image display interface, and based on the drug traceability code, obtain the corresponding drug traceability information and display the drug traceability information.

[0081] In some embodiments, the terminal device 101 locally stores traceability information of multiple drugs, and the traceability information of each drug is stored corresponding to the drug identification. After the terminal device 101 obtains the drug identification through drug traceability code analysis, it can obtain the traceability information corresponding to the drug traceability code locally and display it.

[0082] In other embodiments, after the terminal device 101 obtains the identification of the drug through parsing the drug traceability code, it can send a traceability information acquisition request to the server 102 according to the identification of the drug. After receiving the traceability information acquisition request including the identification of the drug, the server 102 can send the traceability information corresponding to the identification of the drug to the terminal device 101.

[0083] A typical way to display drug traceability information is as follows: Figure 6 As shown. The drug traceability information may include basic information such as the drug name, manufacturer, specifications, and drug details, and may also include drug verification information such as the number of people who scanned the drug traceability code and the scanning record. Based on drug traceability information, users can understand information on drugs in multiple dimensions and improve drug safety.

[0084] It is worth noting that the implementation method of identifying a drug based on an image of at least a portion of the drug packaging body and obtaining the code scanning guidance information corresponding to the drug can be implemented as a "remote computing" method based on the server 102 as described in the above embodiment. When the server is implemented as a cloud server, the above computing method can be described as a "cloud computing" method.

[0085] In other optional embodiments, the above implementation method of identifying a drug based on an image of at least a portion of the drug packaging body and obtaining the code scanning guidance information corresponding to the drug can be implemented as an "edge computing" method based on the terminal device 101. Based on this method, proximal services can be provided to users.

[0086] In the implementation of "edge computing", the deep learning model described in the above embodiment can be run on the terminal device 101. After the terminal device 101 obtains an image of at least a portion of the drug packaging body, the image can be input into the locally running deep learning model to obtain the recognition result of the image. Optionally, when the terminal device 101 has a large storage space, the code scanning guidance information corresponding to the various drugs described in the above embodiment can be stored locally on the terminal device 101.

[0087] After the terminal device 101 obtains the medicine corresponding to the image, it can determine the scanning guidance information of the medicine corresponding to the image from the scanning guidance information corresponding to the multiple medicines, and overlay and display the scanning guidance information on the page where the image display interface is located. In this way, the terminal device 101 can break through network restrictions and provide users with faster, safer, and more reliable scanning guidance services.

[0088] Optionally, in this embodiment, the server 102 can provide an update service for the deep learning model. When the deep learning model is optimized, the server 102 can push the optimized deep learning model to the terminal device 101. Optionally, the server 102 can also provide an update service for the code scanning guidance information corresponding to the drug. For example, when the code scanning guidance information of some drugs is added, the server 102 can send the newly added drugs and the corresponding code scanning guidance information to the terminal device 101 for the terminal device 101 to use.

[0089] In addition to providing scanning guidance services for tracing medicines, the information display system 100 provided by the aforementioned embodiments can also provide scanning guidance services for tracing other items, such as: scanning guidance services for tracing food, scanning guidance services for tracing chemical materials, scanning guidance services for tracing hazardous waste, scanning guidance services for tracing cosmetics, etc.

[0090] When providing scanning guidance for different items, a deep learning model capable of identifying different items based on an image of at least a portion of the item packaging body may be trained, and scanning guidance information for different items may be prepared in advance. In one embodiment, the deep learning model and the corresponding relationship between the scanning guidance information and the identification of the item may be stored on the terminal device 101; in another embodiment, the deep learning model and the corresponding relationship between the scanning guidance information and the identification of the item may be stored on the server 102.

[0091] Based on this, in one embodiment, when the user scans the code to trace the item based on the terminal device 101, if the item traceability code on the item packaging is not scanned, the terminal device 101 can identify the item corresponding to the image based on the image of at least a portion of the item packaging in the image display interface and the deep learning model stored locally, and obtain the scanning guidance information corresponding to the item from the scanning guidance information of multiple different items stored locally.

[0092] In another embodiment, the terminal device 101 may obtain an image of at least a portion of the item packaging body in the image display interface, and send the image to the server 102. The server 102 identifies the item corresponding to the image based on a pre-trained deep learning model, obtains the scanning guidance information corresponding to the item from the scanning guidance information of a plurality of different items, and returns the scanning guidance information of the item traceability code to the terminal device 101.

[0093] After the terminal device 101 obtains the code scanning guidance information corresponding to the item, it can overlay and display the code scanning guidance information on the image display interface to guide the user to scan the item traceability code on the item packaging through the image display interface.

[0094] Based on this implementation, users only need to scan the item to obtain the scanning guidance information corresponding to the item, and can quickly and accurately scan the item traceability code according to the scanning guidance information to achieve item traceability. No further details will be given.

[0095] In addition to the information display system described in the above embodiments, the present application embodiment provides an information display method, such as Figure 7 As shown, when the information display method is executed on the terminal device side, it may include the following steps:

[0096] Step 701: Display an image of at least a portion of an article package in an image display interface.

[0097] Step 701: On the image display interface, code scanning guidance information is superimposed and displayed to guide the user to scan the item traceability code on the item packaging through the image display interface.

[0098] In some exemplary embodiments, before superimposing and displaying the code scanning guidance information, the method further includes: sending the image to a server, so that the server returns the code scanning guidance information of the corresponding item traceability code according to the image.

[0099] In some exemplary embodiments, a method of superimposing and displaying code scanning guidance information on the image display interface includes: displaying a floating window on the page where the image display interface is located; and displaying the code scanning guidance information in the floating window.

[0100] In some exemplary embodiments, displaying the code scanning guidance information includes at least one of the following: displaying a target surface on the item packaging that contains the item traceability code; displaying the relative position relationship between the target surface and a non-target surface on the item packaging that does not contain the item traceability code; displaying the position of the item traceability code on the target surface; displaying the position of the non-item traceability code on the item packaging.

[0101] In some exemplary embodiments, the code scanning guidance information includes at least one of: a code scanning guidance video, a code scanning prompt guidance audio, a code scanning guidance step diagram, and a code scanning guidance dynamic image.

[0102] In some exemplary embodiments, after displaying the code scanning guidance information, it also includes: obtaining the item traceability code in the image display interface; obtaining the traceability information of the corresponding item according to the item traceability code, and displaying the traceability information of the item.

[0103] In some exemplary embodiments, the article packaging body includes at least one of an outer packaging box, an outer packaging bag, an inner packaging bottle, an inner packaging bag and an inner packaging box of the article.

[0104] In some exemplary embodiments, the items include: medicines.

[0105] In this embodiment, an image of at least a portion of the article packaging is displayed in the image display interface, and the code scanning guidance information is superimposed and displayed on the page where the image display interface is located, so that the user can view the code scanning guidance information while scanning the article traceability code on the article packaging through the image display interface. Based on this implementation, while the user is performing the code scanning operation, the code scanning guidance information is pushed to the user in real time, which improves the timeliness of the code scanning guidance method, and is conducive to the user to quickly and accurately determine the location of the article traceability code on the article packaging, and accurately complete the code scanning tracing operation.

[0106] In addition to the information display system described in the above embodiments, the present application embodiment provides an information display method, such as Figure 8 As shown, when the information display method is executed on the server side, it may include the following steps:

[0107] Step 801: Acquire an image of at least a portion of an article package sent by a terminal device.

[0108] Step 802: Identify the image to determine the article corresponding to the article package.

[0109] Step 803: Obtain the code scanning guidance information of the item from the code scanning guidance information corresponding to the multiple different items.

[0110] Step 804: Send the scanning guidance information of the item to the terminal device for display to guide the user to scan the item traceability code on the item packaging.

[0111] In some exemplary embodiments, a method for identifying the image to determine the item corresponding to the item packaging body includes: inputting the image into a deep learning model; in the deep learning model, identifying the main area in the image; performing feature extraction on the main area to obtain image features to be matched; performing feature matching on the image features to be matched with pre-stored image features of different items, and determining the item corresponding to the image features to be matched based on the results of the feature matching.

[0112] In some exemplary embodiments, the scanning guidance information of the item includes: at least one of: a scanning guidance video, a scanning guidance step diagram, and a scanning guidance dynamic image.

[0113] In some exemplary embodiments, the article packaging body includes at least one of an outer packaging box, an outer packaging bag, an inner packaging bottle, an inner packaging bag and an inner packaging box of the article.

[0114] In some exemplary embodiments, the items include: medicines.

[0115] In this embodiment, the scanned item can be identified based on the image of at least a portion of the item packaging. After the item is identified, the scanning guidance information of the item can be sent to the terminal device. Furthermore, the user can perform the scanning operation while viewing the scanning guidance information. Based on this implementation, while the user is performing the scanning operation, the scanning guidance information is pushed to the user in real time, which improves the timeliness of the scanning guidance method, and is conducive to the user to quickly and accurately determine the location of the item traceability code from the item packaging, and accurately complete the scanning traceability operation.

[0116] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 801 to 803 can be device A; for another example, the execution subject of steps 801 and 802 can be device A, and the execution subject of step 803 can be device B; and so on.

[0117] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel, and the sequence numbers of the operations, such as 801, 802, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.

[0118] It should be noted that the descriptions such as “first” and “second” in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit “first” and “second” to different types.

[0119] Fig. 9 is a schematic diagram of the structure of a terminal device provided by an exemplary embodiment of the present application, and the terminal device is applicable to the information display system provided by the aforementioned embodiment. Fig. 9 As shown, the terminal device includes:

[0120] Display component 901 , communication component 902 , image acquisition component 903 , memory 904 and processing component 905 .

[0121] Among them, the display component 901 is used to: display an image of at least a portion of the article packaging body in the image display interface; and superimpose code scanning guidance information on the image display interface to guide the user to scan the article traceability code on the article packaging body through the image display interface.

[0122] The image acquisition component 903 is used to acquire an image of at least a portion of the article packaging body for display by the display component 901 .

[0123] The memory 904 is used to store computer programs and can be configured to store various other data to support operations on the terminal device. Examples of such data include instructions for any application or method used to operate on the terminal device, contact data, phone book data, messages, pictures, videos, etc.

[0124] The processing component 905 is coupled to the memory 904 and is used to execute the computer program in the memory 904 and call other components to implement the functions recorded in this embodiment.

[0125] Further optionally, the communication component 902 is used to: before the display component 901 overlays and displays the code scanning guidance information, send the image to the server, so that the server returns the corresponding code scanning guidance information of the item traceability code according to the image.

[0126] Further optionally, when the display component 901 displays the code scanning guidance information on the image display interface, it is specifically used to: display a floating window on the page where the image display interface is located; and display the code scanning guidance information in the floating window.

[0127] Further optionally, when displaying the code scanning guidance information, the display component 901 is specifically used to perform at least one of the following operations: displaying the target surface on the item packaging that contains the item traceability code; displaying the relative position relationship between the target surface and the non-target surface on the item packaging that does not contain the item traceability code; displaying the position of the item traceability code on the target surface; displaying the position of the non-item traceability code on the item packaging.

[0128] Further optionally, the code scanning guidance information includes: at least one of: a code scanning guidance video, a code scanning prompt guidance audio, a code scanning guidance step diagram, and a code scanning guidance dynamic image.

[0129] Further optionally, after the display component 901 displays the code scanning guidance information, the image acquisition component 903 is also used to: obtain the item traceability code in the image display interface; the processing component 905 is used to: obtain the traceability information of the corresponding item according to the item traceability code, and display the traceability information of the item through the display component 901.

[0130] Further optionally, the article packaging body includes: at least one of an outer packaging box, an outer packaging bag, an inner packaging bottle, an inner packaging bag and an inner packaging box of the article.

[0131] Further optionally, the items include: medicines.

[0132] Further, if Fig. 9 As shown, the terminal device also includes: a power supply component 906, an audio component 907 and other components. Fig. 9 Only some components are shown schematically, which does not mean that the terminal equipment only includes Fig. 9 Components shown.

[0133] Among them, the display component 901 includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0134] Among them, the image acquisition component 903 may include a lens and an image sensor, and the image sensor may be implemented as a CCD (Charge-coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0135] In this embodiment, when the user initiates a code scanning operation, the code scanning guidance information of the item is displayed on the page where the image display interface is located, and then the user can perform the code scanning operation while viewing the code scanning guidance information. Based on this implementation method, while the user is performing the code scanning operation, the code scanning guidance information is pushed to the user in real time, which improves the timeliness of the code scanning guidance method, and is conducive to the user to quickly and accurately determine the location of the item traceability code from the item packaging body, and accurately complete the code scanning traceability operation.

[0136] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by the terminal device in the above method embodiment.

[0137] Fig.10 FIG. 1 is a schematic diagram showing the structure of a server provided by an exemplary embodiment of the present application, and the server is applicable to the information display system provided by the aforementioned embodiment. Fig.10 As shown, the server includes: a memory 1001 , a processing component 1002 , and a communication component 1003 .

[0138] The memory 1001 is used to store computer programs and can be configured to store various other data to support operations on the server. Examples of such data include instructions for any application or method operating on the server, contact data, phone book data, messages, pictures, videos, etc.

[0139] The communication component 1003 is used to obtain an image of at least a portion of the article packaging body sent by the terminal device.

[0140] The processing component 1002 is coupled to the memory 1001 and is used to execute the computer program in the memory 1001, so as to: recognize the image to determine the item corresponding to the item package; obtain the scanning guidance information of the item from the scanning guidance information corresponding to a plurality of different items, and send the scanning guidance information of the item to the terminal device for display through the communication component 1003, so as to guide the user to scan the item traceability code on the item package.

[0141] Further optionally, when the processing component 1002 recognizes the image to determine the article corresponding to the article packaging body, it is specifically used to: input the image into a deep learning model; in the deep learning model, identify the main area in the image; perform feature extraction on the main area to obtain image features to be matched; perform feature matching on the image features to be matched with pre-stored image features of different articles, and determine the article corresponding to the image features to be matched based on the results of the feature matching.

[0142] Further optionally, the scanning guidance information of the item includes: at least one of: a scanning guidance video, a scanning prompt guidance audio, a scanning guidance step diagram, and a scanning guidance dynamic image.

[0143] Further optionally, the article packaging body includes: at least one of an outer packaging box, an outer packaging bag, an inner packaging bottle, an inner packaging bag and an inner packaging box of the article.

[0144] Further optionally, the items include: medicines.

[0145] Further, if Fig.10 As shown, the server also includes: a power supply component 1004 and other components. Fig.10 Only some components are shown schematically, which does not mean that the server only includes Fig.10 Components shown.

[0146] In this embodiment, the scanned item can be identified based on the image of at least a portion of the item packaging. After the item is identified, the scanning guidance information of the item can be sent to the terminal device. Furthermore, the user can perform the scanning operation while viewing the scanning guidance information. Based on this implementation, while the user is performing the scanning operation, the scanning guidance information is pushed to the user in real time, which improves the timeliness of the scanning guidance method, and is conducive to the user to quickly and accurately determine the location of the item traceability code from the item packaging, and accurately complete the scanning traceability operation.

[0147] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by the server in the above method embodiment.

[0148] Above Fig. 9 and Fig.10 The memory in can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0149] Above Fig. 9 and Fig.10 The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 3D, 3G, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0150] Above Fig. 9 and Fig.10 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0151] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0153] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0155] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0156] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0157] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0158] It should also be noted that the terms "include", "comprising" or any other variation thereof are intended to cover non-exclusive inclusion, so that a process, method, drug or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, drug or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, drug or device comprising the element.

[0159] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An information display method, characterized in that: include: Displaying an image of at least a portion of the article package in an image display interface; On the image display interface, code scanning guidance information is superimposed and displayed to guide the user to scan the item traceability code on the item packaging through the image display interface; The code scanning guidance information is used to show the user, in a visual way, how to determine the item traceability code from the item packaging; The code scanning guidance information is obtained by identifying the items corresponding to the item packaging body and from the code scanning guidance information of different items.

2. The method according to claim 1, characterized in that Before the code scanning guidance information is displayed, it also includes: The image is sent to a server, so that the server returns scanning guidance information of the corresponding item traceability code according to the image.

3. The method according to claim 1, characterized in that On the image display interface, code scanning guidance information is superimposed and displayed, including: Displaying a floating window on the page where the image display interface is located; The code scanning guidance information is displayed in the floating window.

4. The method according to claim 1, characterized in that The code scanning guidance information is displayed, including at least one of the following: Displaying the target surface of the article packaging body containing the article traceability code; Displaying the relative positional relationship between the target surface and the non-target surface of the article packaging body that does not contain the article traceability code; Displaying the location of the item traceability code on the target surface; Display the location of the non-item traceability code on the item packaging.

5. The method according to claim 1, characterized in that The code scanning guidance information includes: at least one of a code scanning guidance video, a code scanning prompt guidance audio, a code scanning guidance step diagram, and a code scanning guidance dynamic image.

6. The method according to claim 1, characterized in that After displaying the code scanning guidance information, it also includes: Obtaining the item traceability code in the image display interface; According to the item traceability code, the traceability information of the corresponding item is obtained, and the traceability information of the item is displayed.

7. The method according to any one of claims 1 to 6, characterized in that: The article packaging body includes: at least one of an outer packaging box, an outer packaging bag, an inner packaging bottle, an inner packaging bag and an inner packaging box of the article.

8. The method according to any one of claims 1 to 6, characterized in that: The items include: medicines.

9. An information processing method, characterized in that: include: Acquire an image of at least a portion of the article packaging body sent by the terminal device; Recognize the image to determine the article corresponding to the article package; Obtaining code scanning guidance information of the item from code scanning guidance information corresponding to a plurality of different items; Sending the scanning guidance information of the item to the terminal device for display, so as to guide the user to scan the item traceability code on the item packaging; The code scanning guidance information is used to show the user, in a visual way, how to determine the item traceability code from the item packaging.

10. The method according to claim 9, characterized in that Recognizing the image to determine the article corresponding to the article package includes: Inputting the image into a deep learning model; In the deep learning model, identifying a subject area in the image; Extracting features from the subject area to obtain image features to be matched; The image feature to be matched is matched with pre-stored image features of different objects, and the object corresponding to the image feature to be matched is determined according to the result of the feature matching.

11. The method according to claim 9, characterized in that The scanning guidance information of the item includes: at least one of a scanning guidance video, a scanning guidance step diagram, and a scanning guidance dynamic image.

12. The method according to any one of claims 9 to 11, characterized in that: The items include: medicines.

13. A terminal device, characterized in that: include: Display components; The display component is used to: display an image of at least a portion of the article packaging body on the image display interface; On the image display interface, code scanning guidance information is superimposed and displayed to guide the user to scan the item traceability code on the item packaging through the image display interface; wherein the code scanning guidance information is used to show the user the operation method of determining the item traceability code from the item packaging in a visual way; the code scanning guidance information is obtained by identifying the item corresponding to the item packaging and from the code scanning guidance information of different items.

14. A server, characterized in that: include: processing components and communication components; The communication component is used to: acquire an image of at least a portion of the article packaging body sent by the terminal device; The processing component is used to: recognize the image to determine the article corresponding to the article package; Obtaining the scanning guidance information of the item from the scanning guidance information corresponding to the multiple different items; sending the scanning guidance information of the item to the terminal device for display through the communication component to guide the user to scan the item traceability code on the item packaging; The code scanning guidance information is used to show the user, in a visual way, how to determine the item traceability code from the item packaging.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, it can implement the information display method described in any one of claims 1 to 8 or the information processing method described in any one of claims 9 to 12.

Images