Information query and display method and device

Abstract

The application relates to an information query and display method and device, which comprises the following steps: after a first image collected by a camera of a display terminal device in real time is displayed and a first operation on the first image is received, first query information comprising the first image is sent to a server; the first image is rendered according to a second image returned by the server, and the rendered first image is displayed; second query information comprising a third image is sent to the server according to a second operation of adding new content of an image currently collected by the camera; the third image is rendered according to a fourth image returned by the server, and the rendered third image is displayed. The first query information is determined, time-related images of the first image or the third image are rapidly and efficiently obtained for the user, and the rendered images are displayed for the user, so that the user can understand the development process of the required query content from the perspective of time change through the displayed images.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for information retrieval and display. Background Technology

[0002] In related technologies, users can search for desired images by inputting text or image information. The search results often present a variety of images corresponding to the user's input, resulting in a disorganized display. This is especially problematic when users want to see historical images of a building or street view; the search cost is high and the efficiency is low. Therefore, providing a method to quickly and easily retrieve and display the historical images of buildings or street views that users need is a pressing technical problem that needs to be solved. Summary of the Invention

[0003] In view of this, an information query and display method and device are proposed.

[0004] In a first aspect, embodiments of this application provide an information query and display method, applied to a terminal device, the method comprising:

[0005] The first image captured in real time by the camera of the terminal device is displayed in the first image interface;

[0006] Receive a first operation on the first image, the first operation including a sliding operation or a scrolling operation;

[0007] In response to the first operation, first query information is sent to the server, the first query information including the first image;

[0008] Receive a second image sent by the server in response to the first query information, the second image including an image that is time-related to the first image in the first query information;

[0009] The first part of the first image is rendered according to the second image, and the rendering process of the first part of the first image includes replacing the first part of the first image with the second image;

[0010] The rendered first image is displayed, which includes: a first part that has been rendered and a second part of the unrendered first image other than the first part;

[0011] In response to the received second operation, second query information is sent to the server. The second operation includes adding a portion different from the first image to the third image currently being captured in real time by the camera. The second query information includes the third image.

[0012] Receive a fourth image sent by the server in response to the second query information, the fourth image including an image that is time-related to the third image in the second query information;

[0013] The rendering process of the first part of the third image is performed based on the fourth image, and the rendering process of the first part of the third image includes replacing the first part of the third image with the fourth image;

[0014] The rendered third image is displayed, which includes: a first part that has been rendered and a second part of the unrendered third image other than the first part.

[0015] The information query and display method of this application enables the terminal device to send the first query information, including the first image, determined according to the user's first operation, to the server, and display the rendered first image obtained by rendering the first image based on the second image returned by the server in response to the first query information. This allows the user to directly query information while viewing the first image captured in real time by the camera on the interface, and the rendered first image is displayed on the interface, bringing the user an immersive information query experience. Moreover, the query speed is fast, the accuracy is high, and the time correlation between the displayed rendered first image and the first query information is higher.

[0016] Furthermore, during the display of the rendered first image, a second query message is sent in response to the second operation. This ensures that as long as it is determined that the image currently captured by the camera in real time has changed from the first image to a third image with additional parts different from the first image, the second query message can be sent to the server to re-query the information. This ensures that the query results seen by the user are always consistent with the image captured by the camera in real time, enhancing the viewing experience of immersive information query.

[0017] In one possible implementation, the second operation further includes magnifying the third image currently being captured in real-time by the camera. This ensures that whenever a change is detected in the third image (i.e., magnification or addition of a portion different from the first image), a second query can be sent to the server to re-query the information, guaranteeing that the query results seen by the user are always consistent with the image captured in real-time by the camera.

[0018] In one possible implementation, the method further includes:

[0019] The system receives a third operation and, based on the third operation, enlarges and displays the rendered first image or the rendered third image. The third operation includes the operation of enlarging the image captured by the camera in real time.

[0020] In this way, if it is determined that the user needs to zoom in on the content displayed on the current interface, the rendered first image or the rendered third image can be directly zoomed in and displayed, thereby improving the speed of responding to user operations.

[0021] In one possible implementation, the first graphical interface further displays a first control, and the first operation includes a trigger operation on the first control. The first control allows the user to directly initiate the first operation by manipulating it, simplifying the process and improving the speed of determining the first query information.

[0022] In one possible implementation, the first query information and / or the second query information further include location information of the terminal device. Adding location information can improve the accuracy of the query results returned by the server (i.e., the returned second image and / or fourth image).

[0023] In one possible implementation, the method further includes:

[0024] Before sending the first query information to the server, the first image is edited according to the detected editing operations on the first image; and / or

[0025] Before sending the second query information to the server, the third image is edited according to the detected editing operations on the third image.

[0026] Before issuing the first and / or second query information, the first and / or second query information can be edited according to the user's operation. This can remove information that is not related to the target object that the user wants to query, thereby removing redundant information. This makes the second image retrieved by the server based on the edited first query information and / or the fourth image retrieved based on the edited second query information more accurate and better meet the user's query needs.

[0027] In one possible implementation, the rendered image is displayed in any of the following ways, wherein the rendered image includes the rendered first image and / or the rendered third image:

[0028] One of the rendered images is displayed in full screen on the query results display interface;

[0029] The query results display interface displays part or all of the long image formed by stitching together all the rendered images;

[0030] The query results display interface shows a dynamic image formed from multiple rendered images;

[0031] The query results display interface shows the associated video formed from the multiple rendered images.

[0032] By using various rendering image display methods, the display of rendered images can be diversified, which can meet the different display needs of users.

[0033] In one possible implementation, the query results display interface also displays a prompt message to remind the user to confirm whether the currently displayed rendered image is correct. The method further includes:

[0034] The information determined based on the detected operation in response to the prompt message, and the fifth image captured in real time by the camera, are sent to the server as the third query information; or

[0035] The information determined based on the detected operation in response to the prompt information will be sent to the server as supplementary information to the first query information or the second query information.

[0036] By displaying prompts, users can take further steps to perform a new search or supplement their initial search information if they find issues such as incorrect or inaccurate rendered or second images that do not meet their search requirements.

[0037] In one possible implementation, the query results display interface also shows some related information corresponding to the rendered image.

[0038] The associated information comes from the server and is information related to the second or fourth image corresponding to the rendered image. The associated information includes at least one of the following: time information corresponding to the second or fourth image, event information related to the second or fourth image, character information related to the second or fourth image, and description information of the target object in the first or third image.

[0039] Displaying some associated information corresponding to the rendered image can increase the amount of information in the rendered image, allowing users to further understand the changes in the rendered image by combining the associated information.

[0040] In one possible implementation, the first area of ​​the query results display interface also displays thumbnails corresponding to each of the second or fourth images arranged in chronological order.

[0041] Displaying thumbnails allows users to first get a general idea of ​​how the information they want to search for has changed over time, and then select the images they need to view in more detail as needed.

[0042] In one possible implementation, the first area also displays time information for a second or fourth image corresponding to each thumbnail. This allows users to further understand the time corresponding to the second or fourth image by combining the time information.

[0043] In one possible implementation, the method further includes:

[0044] Receive a fourth operation for the currently displayed rendered image, the long image, or the thumbnail;

[0045] In response to the fourth operation, the currently displayed content is switched to the rendered first image or the rendered third image corresponding to the fourth operation;

[0046] The fourth operation includes at least one of the following operations on the currently displayed rendered image, the corresponding physical key on the terminal device, or the thumbnail: swipe operation, scroll operation, move operation, click operation, double-click operation, and long-press operation.

[0047] In one possible implementation, when a rendered image is displayed in full screen on the query results display interface, the currently displayed content is switched to the rendered first image or the rendered third image corresponding to the fourth operation in response to the fourth operation, including any one of the following switching methods:

[0048] According to the gradient speed determined by the operation speed of the fourth operation, the current rendered image and the next rendered image are switched in a gradient manner in full screen. The gradient manner includes: displaying a first object that exists in the current rendered image but does not exist in the next rendered image in a gradually disappearing manner until the switch is completed, and displaying a second object that does not exist in the current rendered image but exists in the next rendered image in a gradually appearing manner until the switch is completed.

[0049] According to the movement speed determined by the operation speed of the fourth operation, the currently rendered image displayed in full screen is moved out of the query result display interface and the next rendered image is moved into the query result display interface. During the movement, the query result display interface includes a portion of the currently rendered image and a portion of the next rendered image.

[0050] This allows users to switch image displays according to their viewing needs, satisfying their preferences for different images. At the same time, the customized image switching methods provide users with a better visual experience.

[0051] In one possible implementation, the method further includes:

[0052] Receive a fifth operation for the currently displayed rendered image, long image, or thumbnail;

[0053] Display the associated information corresponding to the fifth operation.

[0054] In one possible implementation, the associated information corresponding to the fifth operation is displayed, including any of the following methods:

[0055] The information bar in the query results display interface displays the associated information corresponding to the fifth operation;

[0056] The pop-up window above the query results display interface displays the associated information corresponding to the fifth operation;

[0057] If the associated information corresponding to the fifth operation includes an image, part or all of the long image formed by stitching together the associated information will be displayed in the query results display interface.

[0058] This allows users to not only understand the changes and development of the information they want to query based on the displayed rendered images, long images, or thumbnails, but also to gain a deeper understanding of the events, people, and other information related to the information they want to query by combining related information.

[0059] In one possible implementation, the method further includes:

[0060] During the display of the first image, if the duration during which the first image remains unchanged exceeds a duration threshold, the first query information is sent to the server. This simplifies the user's actions when determining the first query information.

[0061] In one possible implementation, the method further includes: removing irrelevant objects from the rendered image before displaying it. This avoids irrelevant objects from interfering with the user's viewing of the rendered image, ensuring the accuracy of the rendered image displayed to the user.

[0062] Secondly, embodiments of this application provide an information query and display device, applied to a terminal device, the device comprising:

[0063] The first display module displays the first image captured in real time by the camera of the terminal device in the first image interface;

[0064] The first operation receiving module receives a first operation for the first image, the first operation including a sliding operation or a scrolling operation.

[0065] A first sending module, in response to the first operation, sends first query information to the server, the first query information including the first image;

[0066] A first image receiving module receives a second image sent by the server in response to the first query information, the second image including an image that is time-related to the first image in the first query information;

[0067] The first rendering processing module renders a first part of the first image according to the second image. The rendering processing of the first part of the first image includes replacing the first part of the first image with the second image.

[0068] The first rendered image display module displays the rendered first image, which includes: a first part that has been rendered and a second part of the unrendered first image excluding the first part;

[0069] The second sending module, in response to a received second operation, sends second query information to the server. The second operation includes adding a portion different from the first image to the third image currently being captured in real time by the camera. The second query information includes the third image.

[0070] The second image receiving module receives a fourth image sent by the server in response to the second query information, the fourth image including an image that is time-related to the third image in the second query information;

[0071] The second rendering processing module renders the first part of the third image according to the fourth image. The rendering processing of the first part of the third image includes replacing the first part of the third image with the fourth image.

[0072] The second rendered image display module displays a rendered third image, which includes: a first part that has been rendered and a second part of the third image that has not been rendered, excluding the first part.

[0073] In this way, the terminal device can send the first query information, including the first image, determined based on the user's first operation, to the server, and display the rendered first image obtained by rendering the first image based on the second image returned by the server in response to the first query information. This allows the user to directly query information while viewing the first image captured in real time by the camera on the interface, and the rendered first image is displayed on the interface, bringing the user an immersive information query experience. Moreover, the query speed is fast, the accuracy is high, and the time correlation between the displayed rendered first image and the first query information is higher.

[0074] Furthermore, during the display of the rendered first image, a second query message is sent in response to the second operation. This ensures that as long as it is determined that the image currently captured by the camera in real time has changed from the first image to a third image with additional parts different from the first image, the second query message can be sent to the server to re-query the information. This ensures that the query results seen by the user are always consistent with the image captured by the camera in real time, enhancing the viewing experience of immersive information query.

[0075] In one possible implementation, the second operation further includes magnifying the third image currently being captured in real-time by the camera. This ensures that whenever a change is detected in the third image (i.e., magnification or addition of a portion different from the first image), a second query can be sent to the server to re-query the information, guaranteeing that the query results seen by the user are always consistent with the image captured in real-time by the camera.

[0076] In one possible implementation, the device may further include:

[0077] The third rendering image display module receives a third operation and enlarges and displays the rendered first image or the rendered third image according to the third operation. The third operation includes the operation of enlarging the image captured by the camera in real time.

[0078] In this way, if it is determined that the user needs to zoom in on the content displayed on the current interface, the rendered first image or the rendered third image can be directly zoomed in and displayed, thereby improving the speed of responding to user operations.

[0079] In one possible implementation, the first graphical interface further displays a first control, and the first operation further includes a triggering operation on the first control.

[0080] The first control setting allows users to directly issue the first operation by manipulating the first control, simplifying the process of issuing the first operation and improving the speed of determining the first query information.

[0081] In one possible implementation, the first query information and / or the second query information further include location information of the terminal device. Adding location information can improve the accuracy of the query results returned by the server (i.e., the returned second image and / or fourth image).

[0082] In one possible implementation, the device may further include: an information editing module, used for:

[0083] Before sending the first query information to the server, the first image is edited according to the detected editing operations on the first image; and / or

[0084] Before sending the second query information to the server, the third image is edited according to the detected editing operations on the third image.

[0085] Before issuing the first and / or second query information, the first and / or second query information can be edited according to the user's operation. This can remove information that is not related to the target object that the user wants to query, thereby removing redundant information. This makes the second image retrieved by the server based on the edited first query information and / or the fourth image retrieved based on the edited second query information more accurate and better meet the user's query needs.

[0086] In one possible implementation, the rendered image is displayed in any of the following ways, wherein the rendered image includes the rendered first image and / or the rendered third image:

[0087] One of the rendered images is displayed in full screen on the query results display interface;

[0088] The query results display interface displays part or all of the long image formed by stitching together all the rendered images;

[0089] The query results display interface shows a dynamic image formed from multiple rendered images;

[0090] The query results display interface shows the associated video formed from the multiple rendered images.

[0091] By using various rendering image display methods, the display of rendered images can be diversified, which can meet the different display needs of users.

[0092] In one possible implementation, the query results display interface also displays a prompt message, which reminds the user to confirm whether the currently displayed rendered image is correct. The device may further include:

[0093] The third sending module sends the information determined based on the detected operation in response to the prompt information, and the fifth image captured in real time by the camera, as the third query information to the server; or

[0094] The supplementary information sending module sends the information determined based on the detected operation in response to the prompt information to the server as supplementary information to the first query information or the second query information.

[0095] By displaying prompts, users can take further steps to perform a new search or supplement their initial search information if they find issues such as incorrect or inaccurate rendered or second images that do not meet their search requirements.

[0096] In one possible implementation, the query results display interface also shows some related information corresponding to the rendered image.

[0097] The associated information comes from the server and is information related to the second or fourth image corresponding to the rendered image. The associated information includes at least one of the following: time information corresponding to the second or fourth image, event information related to the second or fourth image, character information related to the second or fourth image, and description information of the target object in the first or third image.

[0098] Displaying some associated information corresponding to the rendered image can increase the amount of information in the rendered image, allowing users to further understand the changes in the rendered image by combining the associated information.

[0099] In one possible implementation, the first area of ​​the query results display interface also displays thumbnails corresponding to each of the second or fourth images arranged in chronological order.

[0100] Displaying thumbnails allows users to first get a general idea of ​​how the information they want to search for has changed over time, and then select the images they need to view in more detail as needed.

[0101] In one possible implementation, the first area also displays time information for a second or fourth image corresponding to each thumbnail. This allows users to further understand the time corresponding to the second or fourth image by combining the time information.

[0102] In one possible implementation, the device may further include:

[0103] The fourth operation receiving module receives a fourth operation for the currently displayed rendered image, long image, or thumbnail.

[0104] The image switching module, in response to the fourth operation, switches the currently displayed content to the rendered first image or the rendered third image corresponding to the fourth operation;

[0105] The fourth operation includes at least one of the following operations on the currently displayed rendered image, the corresponding physical key on the terminal device, or the thumbnail: swipe operation, scroll operation, move operation, click operation, double-click operation, and long-press operation.

[0106] In one possible implementation, when a rendered image is displayed in full screen on the query results display interface, the currently displayed content is switched to the rendered first image or the rendered third image corresponding to the fourth operation in response to the fourth operation, including any one of the following switching methods:

[0107] According to the gradient speed determined by the operation speed of the fourth operation, the current rendered image and the next rendered image are switched in a gradient manner in full screen. The gradient manner includes: displaying a first object that exists in the current rendered image but does not exist in the next rendered image in a gradually disappearing manner until the switch is completed, and displaying a second object that does not exist in the current rendered image but exists in the next rendered image in a gradually appearing manner until the switch is completed.

[0108] According to the movement speed determined by the operation speed of the fourth operation, the currently rendered image displayed in full screen is moved out of the query result display interface and the next rendered image is moved into the query result display interface. During the movement, the query result display interface includes a portion of the currently rendered image and a portion of the next rendered image.

[0109] This allows users to switch image displays according to their viewing needs, satisfying their preferences for different images. At the same time, the customized image switching methods provide users with a better visual experience.

[0110] In one possible implementation, the device further includes:

[0111] Receive a fifth operation for the currently displayed rendered image, long image, or thumbnail;

[0112] Display the associated information corresponding to the fifth operation.

[0113] In one possible implementation, the associated information corresponding to the fifth operation is displayed, including any of the following methods:

[0114] The information bar in the query results display interface displays the associated information corresponding to the fifth operation;

[0115] The pop-up window above the query results display interface displays the associated information corresponding to the fifth operation;

[0116] If the associated information corresponding to the fifth operation includes an image, part or all of the long image formed by stitching together the associated information will be displayed in the query results display interface.

[0117] This allows users to not only understand the changes and development of the information they want to query based on the displayed rendered images, long images, or thumbnails, but also to gain a deeper understanding of the events, people, and other information related to the information they want to query by combining related information.

[0118] In one possible implementation, the first sending module is further configured to, during the display of the first image, if it is detected that the duration for which the first image remains unchanged exceeds a duration threshold, then send the first query information to the server. This simplifies the operations performed by the user when determining the first query information.

[0119] In one possible implementation, the device may further include:

[0120] The deletion module removes irrelevant objects from the rendered image before displaying it.

[0121] This avoids irrelevant objects from interfering with the user's view of the rendered image, ensuring the accuracy of the rendered image displayed to the user.

[0122] Thirdly, embodiments of this application provide an information display device, including:

[0123] processor;

[0124] Memory used to store processor-executable instructions;

[0125] The processor is configured to implement the aforementioned information query and display method when executing the instructions.

[0126] Fourthly, embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the aforementioned information query and display method.

[0127] Fourthly, embodiments of this application provide a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in an electronic device, the processor in the electronic device executes the above-described information query and display method.

[0128] These and other aspects of this application will become more apparent in the description of the following embodiments(s). Attached Figure Description

[0129] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.

[0130] Figure 1 This diagram illustrates an application scenario of an information query and display method according to an embodiment of this application.

[0131] Figure 2 A flowchart illustrating an information query and display method according to an embodiment of this application is shown.

[0132] Figures 3A-3E This diagram illustrates the determination of the first query information in an information query and display method according to an embodiment of this application.

[0133] Figures 4A-4H This diagram illustrates the display of street view images in an information query and display method according to an embodiment of this application.

[0134] Figures 5A-5B This diagram illustrates the display of architectural images in an information query and display method according to an embodiment of this application.

[0135] Figures 6A-6B This diagram illustrates the display of a person-type image in an information query and display method according to an embodiment of this application.

[0136] Figures 7A-7B This diagram illustrates the display of animal images in an information query and display method according to an embodiment of this application.

[0137] Figure 8 This diagram illustrates the display of images of cultural relics in an information query and display method according to an embodiment of this application.

[0138] Figure 9 This is a schematic diagram showing the rendering of a third image in an information query and display method according to an embodiment of this application.

[0139] Figure 10 This diagram illustrates the structure of a terminal device 100. Detailed Implementation

[0140] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0141] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0142] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0143] As cities develop and change, urban buildings and streetscapes undergo transformations, sometimes being repaired, sometimes demolished and rebuilt. These buildings and streetscapes often carry people's memories and even represent the rise and fall of a region. However, when users encounter a particular building or scene, they can only rely on vague memories to recall its historical appearance or extensively search for historical images on the internet, lacking a more convenient and comprehensive way to obtain and display historical images.

[0144] In related technologies, users can obtain historical images of a building or street view through web search or map street view. Web search involves users inputting information about the building or street view they want to search for to obtain its corresponding historical images. However, the search cost is usually high, and the displayed images are often cluttered or incomplete, resulting in a poor user experience. Map street view, on the other hand, is based on the street view function of map applications to obtain building and street view image information for a specific location. However, it can only provide users with the latest version of the image corresponding to the current time, and it is difficult to obtain and display corresponding historical images.

[0145] To address the aforementioned problems, this application provides an information query and display method and apparatus. Figure 1 This diagram illustrates an application scenario of an information query and display method according to an embodiment of this application. For example... Figure 1 As shown, the information query and display method and apparatus provided in this application can be applied to terminal devices. Based on the query information of the target object input by the user, it can quickly and efficiently obtain time-related images (i.e., the second image and / or the fourth image in this application) related to the target object for the user, and display the images based on the related images in a preset display rule order, so that the user can understand the development process of the target object through the displayed images, and understand the changes of the target object from the perspective of time change.

[0146] Among them, such as Figure 1As shown, the target object can be an object that undergoes a developmental process of change, and the causes of this change can be human factors, natural factors, etc. The associated images can include historical images of the target object relative to the current moment (actual images corresponding to the target object in history, such as screenshots from photos or video resources), images at the current moment, and images corresponding to the future plans of the target object (such as planning maps, 3D models, etc., images that can show the future changes of the target object). The target object can include humans (e.g., Figures 6A-6B The image displayed is related to a person, and animals (such as...) Figures 7A-7B The exhibition displays related images of animals, plants, microorganisms, and cultural relics (such as...). Figure 8 The images displayed are related to the cultural relics), and buildings (such as...). Figures 5A-5B The images displayed are related to buildings, objects, and street scenes (such as...). Figures 4A-4F The displayed image is a street view image, a natural landscape, a cultural landscape, a vehicle, an appliance, a piece of furniture, a tool, or a terminal device.

[0147] The terminal device involved in this application is a device with a display screen or capable of displaying content to a user through other means. The terminal device may include at least one of the following: mobile phone, foldable electronic device, tablet computer, desktop computer, laptop computer, handheld computer, laptop, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device, or smart city device. This application does not impose any special limitation on the specific type of terminal device.

[0148] Figure 2 A flowchart illustrating an information query and display method according to an embodiment of this application is shown. Figure 2 As shown, in this method, the terminal device is used to execute steps S11-S17, and the server is used to execute steps S21 and S24.

[0149] In step S11, if a query request is detected from a user, the user's input operation for inputting or determining query information can be detected, and the first query information can be determined based on the input operation.

[0150] In this application, the method executed by the terminal device can be implemented through an application or as a function of an application. The terminal device can determine that the user has issued a query request when it detects that the user has launched the application or launched the query function of the application.

[0151] The first query information can be at least one of image, video, audio, and text. The method for determining the first query information in step S11 can be based on different terminal device types, user selection settings, etc., and can include any one of the following query information determination methods one to three. Figures 3A-3E This diagram illustrates the determination of first query information in an information query and display method according to an embodiment of this application. The following is in conjunction with... Figures 3A-3E Methods one through three for determining query information are described illustratively.

[0152] Method 1 for determining information:

[0153] If a query request is detected, it can be done as follows: Figure 3A The information input interface T1 is displayed directly on the terminal device's screen. The information input interface T1 displays input components for the user to input initial query information. For example... Figure 3A As shown, the input component may include at least one of the following: an input box L2 for text and / or voice input; an import control K2 for importing local images, text, audio, and video; a capture control K1 for activating the terminal device's camera to capture video footage in real time; a preview box L1 for displaying the first query information already entered by the user; and a control for processing the first query information displayed in the preview box L1, such as... Figure 3A The L1 section displays a "Delete" control for deleting selected content, an "Edit" control for initiating editing operations on selected content, and so on. Input operations can include user-initiated actions on input components, as well as other user-initiated actions for inputting and / or determining initial query information.

[0154] In showing users Figure 3A During the process of the information input interface T1 shown, if a text input operation (such as clicking on L2) is detected in the input box L2, the detected user input text is determined as the first query information, and this first query information is set as follows: Figure 3B , Figure 3C The image is shown in preview box L1 (e.g.) Figure 3B , Figure 3C The text "xxx Road" is displayed in the preview box L1.

[0155] In showing users Figure 3A During the information input interface T1 shown, if a voice input operation (such as a long press on L2) is detected in the input box L2, the user's voice input is first obtained, and then the voice can be directly used as the first query information and displayed in the preview box L1. Alternatively, the voice can be analyzed to determine the corresponding text information, and this text information can be determined as the first query information and displayed as shown in the preview box L1. Figure 3B , Figure 3C The image is shown in preview box L1 (e.g.) Figure 3B , Figure 3C The text "xx1 bus stop" is displayed in the preview box L1.

[0156] In showing users Figure 3A During the process of inputting information into interface T1 as shown, if a trigger operation on the import control K2 is detected, it can be done as follows: Figure 3B As shown, the information input interface T1 displays an import box L3, which then determines the first query information based on the text (text information in the form of Word files, etc.), video, image, and audio files selected or dragged by the user from the local machine. Specifically, for text and audio files, the recognized text is identified and then used as the first query information. For videos and images, they can be directly used as the first query information. The first query information for text can then be displayed as follows: Figure 3B , Figure 3C Directly displaying text, or providing the first query information for videos or images, can be like... Figure 3B , Figure 3C Displayed using thumbnails or similar methods.

[0157] In showing users Figure 3A During the information input interface T1 process shown, if a trigger operation on the shooting control K1 is detected, the camera of the terminal device can be turned on, and as shown... Figure 3D The first interface T11 directly switches the content displayed on the screen to display the first image (i.e., the viewfinder image) captured in real time by the camera. During the display of the first interface T11, if a user's photo-taking or video-recording operation on the first image in the first interface T11 is detected (e.g., a click on the shooting control K10 can be identified as a photo-taking operation, and a long press on the shooting control K10 can be identified as a video-recording operation), an image or video is captured. After completing the image or video capture, the following can be done... Figure 3DThe system continues to execute the user-instructed operation according to the "Preview 1", "Preview 2", or "Return" rules. The "Preview 1", "Preview 2", and "Return" rules are merely illustrative examples provided in this application. Those skilled in the art can set actual rules based on these examples and their needs, and this application does not impose any limitations on this.

[0158] Under the "Preview 1" rule, after completing image or video capture, you can... Figure 3D The system switches the content displayed on the screen to a first preview interface T12, where a first image or video captured in real-time is shown to the user for previewing and operation on the first image or video in T12. Based on the user's operation, at least one of the following operations can be performed: If the system determines that the user has selected the currently displayed first image or video as the first query information based on the detected operation (such as a trigger operation on the "OK" control in T12), the content displayed on the screen can be switched back to the previous state. Figure 3C The information input interface T1 is shown, and the first image or first video that has been identified as the first query information is added to L1 of T1; when it is determined that the user needs to retake the first image or first video based on the detected operation (such as detecting the trigger operation of the "retake" control in T12), the content displayed is switched to T11, and the detection of whether the user has issued a photo or video recording operation continues; and so on.

[0159] Under the "Preview 2" rule, after completing image or video capture, you can... Figure 3DThe content displayed on the screen is switched to the second preview interface T13. In this preview interface T13, the user is shown a first image or video captured in real-time. At least one of the following operations can be performed: Based on detected operations (such as a trigger operation on the first control K3 in T13), it can be determined that the user is directly querying the information. The first image or video currently displayed in T13, along with other previously determined first query information, can be unified as all the first query information to execute step S12; An editing request for the selected query information is detected (such as a trigger operation on the "Edit" control in T13). The system further displays an interface for editing the first image or video, or adds controls (not shown in the figure) for editing, allowing the user to edit the first image or video in the editing interface or with the help of the editing controls. After confirming that the user has finished editing, the system continues to display the edited first image or video to the user in T13. If it is determined that the user needs to retake the first image or video based on the detected operation (such as detecting a trigger operation on the "retake" control in T13), the displayed content is switched to T11, and the system continues to detect whether the user has issued a photo or video recording operation; etc. Among them, the setting of the first control K3 allows the user to directly issue the first operation by manipulating the first control, simplifying the process of the user issuing the first operation and improving the speed of determining the first query information.

[0160] Under the "Return" rule, you can also do so after completing image or video capture. Figure 3D According to the "return" rule, the first image or second video captured is directly identified as the first query information, and the content displayed on the screen is switched back to the previous state. Figure 3C The information input interface T1 is shown, and the first image or first video that has been identified as the first query information is added to L1 of T1.

[0161] After determining the first query information entered or confirmed by the user, such as Figure 3C As shown in the preview box L1 of the information input interface T1, during the process of displaying the first selected query information to the user, upon detecting a selection operation on the first query information in the preview box L1, the system determines the query information that the user has selected from the first query information and that requires further processing. This can be achieved by highlighting the currently selected first query information to remind the user, for example, by adding a border (e.g., ...). Figure 3C The thumbnail is highlighted using methods such as adding a border, then the selected first query information is the image or video corresponding to that thumbnail, and so on. Subsequently, if a deletion operation targeting the selected first query information is detected (e.g., if a deletion operation is detected...), the selected first query information will be highlighted. Figure 3CIf the "Delete" control in T1 is triggered, the selected first query information is deleted from the first query information. If an edit request for the selected first query information is detected, an interface for editing the selected first query information is further displayed to the user, or controls for editing (not shown in the figure) are added, so that the user can edit the selected first query information in the editing interface or with the help of the editing controls. After confirming that the user has completed the editing, the edited selected first query information continues to be displayed in the preview box L1.

[0162] Method 2 for determining information:

[0163] When a query request is detected, the difference between query information determination method two and the aforementioned query information determination method one is that: The terminal device's camera can be directly activated, and the first interface T11 can be displayed to the user through the terminal device. The first interface T11 includes the first image captured in real-time by the camera (i.e., the viewfinder view), eliminating the need to first display T11 and then display T11 based on user operation. Furthermore, during the display of the first interface T11, if a user's photo or video recording operation is detected on the first interface T11 (e.g., a click on the shooting operation control K10 can be identified as a photo-taking operation, and a long press on the shooting operation control K10 can be identified as a video recording operation), an image or video can be captured. After completing the image or video recording, the process can proceed as follows... Figure 3D In the "Preview 2" rule, continue to execute the user-instructed operation, as described above, and will not be repeated here.

[0164] Method 3 for determining information:

[0165] Upon detecting a query request, the terminal device's camera can be directly activated, and the terminal device can be used to display information to the user, such as... Figure 3EThe first image interface T14 shown contains a first image (i.e., a viewfinder image) captured in real time by a camera. During the display of the first image interface T14, if first query information is determined, step S12 continues. The first query information can be determined in any of the following ways: If the duration for which the first image displayed in T14 remains unchanged exceeds a duration threshold (e.g., 5 seconds), then the first image is determined as the first query information. "The first image remaining unchanged" can mean that the change in the first image is less than or equal to a preset change threshold. Whether the first image has changed can be determined by detecting whether the spatial position of the camera has changed, whether the shooting direction of the camera has changed, etc. If a first operation is detected by the user, then the first image is determined as the first query information. The first operation can be a detected trigger operation on the first control K3, a quick operation such as a sliding or scrolling operation on the first image, a trigger operation on a corresponding physical key set on the terminal device, etc.

[0166] For devices with touchscreens, such as mobile phones, laptops, and terminal devices, the first query information can be determined using any of the three methods mentioned above, based on default settings, or by the user selecting one of these methods. For VR devices, the first query information can be determined directly using either method two or three.

[0167] Before issuing the first query information, the first query information (including the first image, text, etc.) can be edited according to the user's operation. This can remove information irrelevant to the target object the user wants to query, thus eliminating redundant information. This makes the second image determined by the server based on the edited first query information more accurate and better meets the user's query needs. Similarly, before issuing the second or third query information in this application, the information can also be edited according to the user's operation.

[0168] It should be noted that the above-mentioned methods for determining query information (methods one to three) are merely exemplary implementations of determining the first query information provided in this application. Those skilled in the art can set the implementation method for determining the first query information according to actual needs, and this application does not impose any restrictions on this.

[0169] In step S12, the determined first query information is sent to the server.

[0170] Among them, the terminal device displays to the user such as Figure 3A , Figure 3C , Figure 3D , Figure 3EWhen the interface is accessed, if the conditions for sending query information are met, the first query information can be sent to the server. The conditions for sending query information can include any of the following: detecting a trigger operation on the first control K3; detecting a preset shortcut operation such as sliding or scrolling to instruct the terminal device to send the first query information to the server; detecting a trigger operation on the corresponding physical key set on the terminal device; determining that the duration of the first query information exceeds a predetermined duration threshold (e.g., 5s, 10s, etc.); detecting that the duration of the first image displayed in T14 without change exceeds a duration threshold (e.g., 5s); etc.

[0171] In one possible implementation, the first query information, in addition to the information input or determined by the user, may also include location information determined by the terminal device through positioning. Adding location information to the first query information, as well as the second and third query information described below, can improve the accuracy of the query results returned by the server (i.e., the returned second and / or fourth images). If it is detected that the terminal device has enabled positioning and the first query information includes the first image, the first query information sent to the server in step S12 may also include the location information determined by the terminal device through positioning. The first query information sent to the server may include at least one of the following: a first image, a first video, input text, voice, images, and videos, location information determined by positioning, etc.

[0172] In another embodiment, if the user does not input location information, the image captured by the camera is used directly as the first query information to execute subsequent steps. If the user finds that the query result is different from the expected result, they can then input location information through the terminal device to correct it. This saves the user the effort of inputting location information at the beginning.

[0173] like Figure 2 As shown, after receiving the first query information, the server executes steps S21 and S22.

[0174] Step S21: Process and analyze the first query information to determine the object characteristics of the target object indicated by the first query information (step S211); then execute step S212 to obtain multiple candidate images matching the object characteristics from a pre-established image information database (or candidate images can be captured from the network in real time), and obtain multiple second images after filtering and / or processing the multiple candidate images (step S213). Step S22: Send the multiple second images to the terminal device according to preset rules. Alternatively, step S22 can be executed after step S212 according to preset rules, that is, after determining multiple candidate images, the multiple candidate images are directly sent to the terminal device as second images.

[0175] The object characteristics of the target object can include its name, location information, shooting angle, relative distance to the target object, area proportion of the target object in the image or video in the first query information, and other characteristics and shooting-related information that describe the target object itself. For example, if the target object is a building, its object characteristics could be the building's name, geographical location (such as latitude and longitude), shooting angle, relative distance to the building, area proportion of the building in the image, and location. If the target object is a person, its object characteristics could be the person's name, shooting angle, relative distance to the person, and area proportion of the person in the image or video. If the target object is an animal, plant, or object, its object characteristic could be its name.

[0176] The filtering of multiple candidate images performed in step S213 by the server may include operations such as deduplication and image group selection to reduce the number of multiple candidate images; the processing of multiple candidate images may include image processing operations such as association information matching, preprocessing, sorting, compositing, and image increment determination for candidate images determined as the second image.

[0177] Deduplication:

[0178] The server identifies duplicate images among multiple candidate images and deletes the duplicate images.

[0179] If multiple similar images are found to be in the same time interval and have a similarity greater than or equal to the deduplication similarity threshold among multiple candidate images, the best image with the highest matching degree with the first query information is determined among the multiple similar images, and the remaining similar images other than the best image are determined as duplicate images.

[0180] The time interval and / or deduplication similarity threshold for similar images can be fixed, such as using a year as the time interval and a deduplication similarity threshold of 90% to determine similar images for each year among multiple candidate images. Alternatively, the time interval and / or deduplication similarity threshold can be determined in real-time based on information such as the number of candidate images, the time corresponding to each candidate image, and a preset range of associated images to be sent to the terminal device, to provide users with a better viewing experience of associated images. For example, when there are many candidate images, the time interval corresponding to similar images can be expanded and / or the deduplication similarity threshold can be increased to ensure that the number of associated images ultimately sent to the terminal device is within a preset range and to reduce the total number of associated images sent to the terminal device. When there are few candidate images, the time interval corresponding to similar images can be narrowed and / or the deduplication similarity threshold can be lowered to ensure that the number of associated images ultimately sent to the terminal device is within a preset range and to enrich the total number of associated images sent to the terminal device.

[0181] Where the first query information includes images (including first images captured in real-time by a camera and / or images uploaded by the user) and / or videos (including first videos captured in real-time by a camera and / or videos uploaded by the user), the degree of matching between each similar image and the first query information can be determined by the similarity of the area proportion, shooting angle, and / or position of the target object in each similar image within the images and / or videos in the first query information. The image with the highest similarity to the first query information among multiple similar images is then determined as the best image. Where the first query information is text, the degree of matching between each similar image and the first query information can be determined by comparing the features of the similar images with the comparison results of the text-based first query information.

[0182] Image group selection:

[0183] The server can divide multiple candidate images into multiple image groups based on the similarity between each candidate image. The similarity between candidate images in each image group is greater than or equal to the group similarity threshold. Then, the multiple candidate images in the image group with the largest number and / or the number closest to the maximum value of the aforementioned preset number range can be directly determined as multiple associated images. Alternatively, the server can first perform deduplication on the multiple candidate images in each image group according to the above-described "deduplication" operation, and then determine the multiple candidate images in the image group with the largest number and / or the number closest to the maximum value of the aforementioned preset number range as multiple second images.

[0184] The filtering operation can involve deduplicating only multiple candidate images, deduplicating first and then selecting image groups, or selecting image groups first and then deduplicating; this application does not impose any restrictions on this. By filtering multiple candidate objects, the similarity between the multiple second images displayed to the user can be guaranteed, reducing the number of second images and decreasing the display of redundant information, so that the user can view second images related to the target object in the first query information in a shorter or reasonable time.

[0185] Related information matching:

[0186] After identifying multiple second images, the server can retrieve association information for each second image from an image database and / or from the network in real time to enrich the information displayed to the user related to the target object and the corresponding second image. Association information may include: introductory information about the target object in the second image, such as the object's name (or surname), location information, and image capture information (capture time, photographer, etc.); information about events related to the target object in the second image (including event names, occurrence times, and other introductory information), such as groundbreaking ceremonies for buildings, completion ceremonies for schools and research institutes, relocation and inauguration ceremonies, and events related to the birth and achievements of famous figures; and information about people related to the target object in the second image, such as brief biographies. Association information can be in the form of text, video, images, etc. Those skilled in the art can set the content of the association information according to actual needs, and this application does not impose any limitations on this.

[0187] Preprocessing:

[0188] After identifying multiple second images, the server can preprocess them to improve the visual effect of the image display for the user. This preprocessing includes at least one of the following: removing irrelevant objects from the second image that are unrelated to the target object. Irrelevant objects can be people, animals, vehicles, sky, etc., which are different from and unrelated to the target object, thus improving the transmission efficiency of the second image; rendering the second image; image compression and / or enhancement; adjusting the color, contrast, brightness, etc. of the second image; and fusing related information with the corresponding second image (e.g., displaying the corresponding related information in a certain area of ​​the second image, or generating a new second image based on the related information and the second image). Figure 4B (For example, in the diagram, a display panel L4 for showing related information about the second image is added to the original second image, forming a new second image, etc.). Alternatively, the terminal device can perform some or all of the above preprocessing before displaying part or all of the second image.

[0189] Sort:

[0190] After identifying multiple second images, the server can sort them chronologically based on the capture time of each second image and / or the occurrence time of events related to the target object. Alternatively, the terminal device can sort the multiple second images as needed before displaying them.

[0191] synthesis:

[0192] After identifying multiple second images, the server can combine these sorted images to create a dynamic image or associated video, and then send the combined dynamic image or associated video to the terminal device. Alternatively, the terminal device can also perform the combination of dynamic images or associated videos.

[0193] Image increment determination:

[0194] After the second image is determined, if the first query information includes the first image, the second image sent to the terminal device can be a full image (i.e., including all information of the image) obtained after completing at least one of the above filtering and processing steps, or an incremental image generated based on the differences or changes between the full image and the first image. The incremental image can be incremental data of the full image relative to the first image, corresponding to a first part of the first image that differs from the full image. If the image in the first query information is a "user-uploaded image," the second image sent to the terminal device can be a full image (i.e., including all information of the image) obtained after completing at least one of the above filtering and processing steps, or an incremental image generated based on the differences or changes between the full image and the "user-uploaded image," corresponding to a first part of the "user-uploaded image" that differs from the full image. When it is necessary to send the second image to the terminal device, the differences or changes between two adjacent second images can also be sent to the terminal device as information of the second image, based on the order of multiple second images.

[0195] In step S22, the server sends one or more second images to the terminal device.

[0196] The server may send one or more second images in the following ways: sending multiple second images to the terminal device sequentially in order of sorting from front to back; or sending multiple second images to the terminal device together; or sending the target second image with the highest similarity to the first query information, the first image in the sorting order, or the last image in the sorting order to the terminal device first, and then sending the other second images to the terminal device sequentially based on the request order of the terminal device. Those skilled in the art can configure the method of sending the second images according to actual needs, and this application does not impose any restrictions on this.

[0197] In step S13, one or more second images are received from the server; then step S14 is performed to display images to the user based on the second images.

[0198] Among them, depending on the type of the target object, the type of the terminal device, and the way the server sends multiple second objects, the image display method for displaying images to the user based on the second image includes at least one of the following image display methods 1-5. Figures 4A-4H This diagram illustrates the display of street view images in an information query and display method according to an embodiment of this application. Figures 5A-5B This diagram illustrates the display of architectural images in an information query and display method according to an embodiment of this application. Figures 6A-6B This diagram illustrates the display of a person-type image in an information query and display method according to an embodiment of this application. Figures 7A-7B This diagram illustrates the display of animal images in an information query and display method according to an embodiment of this application. Figure 8 This diagram illustrates the display of images of cultural relics in an information query and display method according to an embodiment of this application. The following is in conjunction with... Figures 4A-4F , Figures 5A-5B , Figures 6A-6B , Figures 7A-7B , Figure 8 The image display methods 1-5 are described schematically.

[0199] Image display method 1:

[0200] If all the second images are received, the user can be shown, for example... Figure 4A The query results are displayed in interface T2. That is:

[0201] Example 1: When each second image is a full image, the target second image among multiple second images can be displayed to the user in full screen in the query results display interface T2, and thumbnails of each second image can be displayed to the user in the first area Q1 of T2 in chronological order.

[0202] Example 2: When each second image is a full image and the first query information includes the first image or a user-uploaded image, a rendered image P1 obtained by rendering the target second image can be displayed in full screen in T2. ​​The rendered image can include: a rendered first image obtained by rendering the first image based on the target second image, or a rendered user-uploaded image obtained by rendering the "user-uploaded image" based on the target second image. Thumbnails of each second image are displayed to the user in chronological order in the first area Q1 of T2. If the second image is a full image, the rendering process can be replacing the first part of the first image that differs from the second image with the corresponding part of the second image; or replacing the first part of the user-uploaded image that differs from the second image with the corresponding part of the second image. The rendered first image includes the first part replaced with the corresponding part of the second image and the second part of the original first image that differs from the first part. The rendered user-uploaded image includes the first part replaced with the corresponding part of the second image and the second part of the original user-uploaded image that differs from the first part.

[0203] Example 3: If each second image is a full image and the first query information includes the first image or a user-uploaded image, the rendered image P1 obtained by rendering the target second image can also be displayed in full screen in T2. ​​Furthermore, thumbnails of each rendered image can be displayed to the user in the first region Q1 of T2, sorted chronologically.

[0204] Example 4: If the first query information includes a first image or a user-uploaded image, and the second image is an incremental image, the rendered image P1 obtained by rendering the target second image can be displayed in full screen in T2. ​​Thumbnails of each rendered image are then displayed to the user in chronological order in the first region Q1 of T2. If the second image is an incremental image, the rendering process involves replacing the first part of the first image with the second image, or replacing the first part of the user-uploaded image with the second image. The rendered first image includes the first part replaced by the second image and a second part in the original first image that differs from the first part. The rendered user-uploaded image includes the first part replaced by the second image and a second part in the original user-uploaded image that differs from the first part.

[0205] By displaying thumbnails in Q1, users can first get a general idea of ​​how the information they want to query changes over time, and then select the second image or rendered image that they need to view in detail as needed.

[0206] Furthermore, thumbnails corresponding to the full-screen second image or rendered image (including the rendered first image, the rendered user-uploaded image, or the rendered third image mentioned below) in Q1 can be highlighted by adding borders or other methods (the implementation method is described above and will not be repeated here). This allows users to determine the temporal order of the full-screen second image with other second images based on Q1, or to determine the temporal order of the full-screen rendered image with other rendered images based on Q1. A timeline Z can also be displayed to the user in Q1, with time information (image capture time and / or the occurrence time of related events) added to the timeline Z for each thumbnail corresponding to the second image. This allows users to determine the time information of each second image or rendered image based on the timeline Z.

[0207] In such Figure 4A During the display of T2, based on the detected fourth operation from the user to switch the full-screen display image (i.e., the second image or the rendered image), the currently full-screen display image can be switched to the second image or the rendered image corresponding to the fourth operation. The fourth operation can be a detected operation targeting the area of ​​T2 excluding Q1 (i.e., the rendered image displayed in full screen), a specific thumbnail in Q1, or a corresponding physical key on the terminal device (e.g., a...). Figure 4B The system allows for operations such as swiping, scrolling, moving, clicking, double-tapping, and long-pressing in C1. On the timeline Z of Q1, a display adjustment control K4 corresponding to the currently displayed full-screen image can be displayed, allowing users to adjust and switch the current full-screen display by moving K4. The fourth operation can also be a detected user movement of K4; the terminal device can switch the currently displayed full-screen image to the second image or rendered image corresponding to the fourth operation based on the direction and distance of movement of K4. During the switching of the full-screen image, the image is switched according to either of the preset switching methods one or two.

[0208] Switching to Method 1: (e.g.) Figure 4A As shown, according to the movement speed determined by the operation speed of the fourth operation, the current image displayed in full screen is moved out of the query result display interface T2, and the next image (including the rendered image or the second image) is moved into the query result display interface T2. During the movement, the query result display interface T2 includes a portion of the current image and a portion of the next image. Figure 4A The fourth operation shown is a movement operation for K4. The movement speed of the image can be determined based on the speed at which K4 is moved, so that the current image is gradually moved out of T2 while the next image is gradually moved into T2.

[0209] Switching to Method Two: (e.g.) Figure 4BAs shown, the current image and the next image are switched in a full-screen manner with a gradient speed determined according to the operation speed of the fourth operation. The gradient method includes: gradually fading out a first object H that exists in the current image but not in the next image until the switch is complete, and gradually appearing a second object W that does not exist in the current image but exists in the next image until the switch is complete. This allows users to understand the change process between images based on the gradient display method. The implementation of the gradient includes methods similar to the switching between adjacent pages in related technical demonstration documents, such as "blinds," "fade in," "fade out," and "random lines," etc., and this application does not limit this to these methods.

[0210] in, Figure 4B The fourth operation shown is a scrolling operation for the physical key C1. The gradient speed can be determined based on the user's scrolling speed of C1, and the transition between the current and next images can be completed by scrolling one or a specified number of "tooth" increments or by scrolling a specified angle. Taking the transition between the current and next images as an example of scrolling one "tooth" increment, in the absence of a detected fourth operation, the result would be as follows: Figure 4B As shown in "Before Switching", the current image is displayed in full screen in T2; if the user is detected scrolling C1 and has not completed a single "tooth" of scrolling, then... Figure 4B In the "Switching" state, T2 displays the current image in full screen, gradually fades out the first object H, and gradually reappears the second object W; when user scrolling C1 is detected and a "tooth" scroll has been completed, it is as follows... Figure 4B The next image, including the second object W, is displayed in full screen in T2 after the "switching is complete" state. This second image switching method allows users to intuitively understand the differences and changes between different images based on the changes in objects on the interface, providing a better visual experience and creating an immersive viewing experience.

[0211] Among them, such as Figure 4A As shown, T2 can also display a prompt message L7, which reminds the user to confirm whether the currently displayed rendered image or second image is correct. This allows the user to perform further operations to re-search or supplement the first query information and search again if they find that the rendered image or second image is incorrect, inaccurate, or does not meet the query requirements.

[0212] If it is determined from the detected user operation that the user wants to query again, the information determined from the detected operation in response to the prompt information and the new fifth image captured by the camera can be sent to the server as the third query information (the operations performed by the server and the terminal device are similar to those performed by the two after the terminal device sends the first query information, and will not be described in detail here for the sake of simplicity). In this way, the third query information can be re-entered to perform the query again.

[0213] Alternatively, if the detected user action indicates that the user wants to query again, the information determined based on the detected action in response to the prompt message can be sent to the server as supplementary information to the first or second query message. In other words, the newly determined supplementary information is used as supplementary content to the already sent first or second query message, enabling a second query. The detected information determined based on the action in response to the prompt message may include: user-inputted or determined location information, input text and / or voice information, etc., which can serve as query information (including the first, second, or third query message).

[0214] For example, if a user triggers an operation on the "I want to supplement" control in L7, and the first or second query information corresponding to the currently displayed image does not include location information (determined by the terminal device's location), the user can be prompted to enable the terminal device's location function, and the determined location information can be defined as "information determined based on the detected operation in response to the prompt information." Then, the aforementioned re-query or re-query operation can be performed. In other image display methods of this application, L2 can also be displayed in T2 and the re-query or re-query operation can be performed in response to user operation, but for simplicity, L7 is not shown in other figures of this application.

[0215] Among them, such as Figure 4A , Figure 4BAs shown, T2 may also include one or more display panels L4, each containing partial related information corresponding to the corresponding second image. Displaying related information can increase the amount of information in the query results, allowing users to further understand the changes in the information they are querying by combining the related information. Furthermore, when a fifth operation is detected for a full-screen image or a thumbnail of Q1, the related information corresponding to the fifth operation is displayed to the user. The related information displayed for the fifth operation may be different from the related information displayed in L4 of the currently displayed T2. It may be further detailed information of the related information displayed in the currently displayed second image or rendered image corresponding to the fifth operation, or it may be completely different related information. The related information displayed in response to the fifth operation may be all or part of the related information corresponding to the second image of the fifth operation.

[0216] The fifth operation may include: detecting operations different from the fourth operation, such as swiping, clicking, double-tapping, and scrolling, on an image displayed in full screen. For example, when displaying an image to the user... Figure 4A In the process T2 shown, the fourth operation could be a detected left-right swipe operation on the region of T2 excluding Q1 and on a thumbnail within Q1. Therefore, the fifth operation could be a vertical swipe operation on the full-screen image. The fifth operation could also include a detected trigger operation on the associated information display control K5 shown in T2. ​​K5 could be... Figure 5A , Figure 8 K5 is a separately configured, user-operable control; K5 can also be, for example... Figure 6B The control K5, also known as "yyy Academy," is an object derived from a second image, a rendered image, a thumbnail of the second image, or a thumbnail of the rendered image that can be manipulated by the user; etc. Those skilled in the art can configure the implementation of the fifth operation according to actual needs, and this application does not limit this.

[0217] The associated information display method in response to the fifth operation may include at least one of the following associated information display methods 1-3.

[0218] Related information display method 1:

[0219] When it is determined that the associated information to be displayed in response to the fifth operation is only text, it can be as follows: Figure 6B , Figure 8 As shown, information is further displayed as an information bar L6 in T2, where related information is presented to the user as text. Furthermore, the display of related information can be stopped upon detecting a sixth operation that halts the display of related information.

[0220] The sixth operation can be a detected trigger operation on the close control K6 displayed at other locations on L6 or T2. The sixth operation can also be a detected operation opposite to or relative to the fifth operation that responds to the display of associated information. For example, during the process of displaying the corresponding associated information in response to the fifth operation, a detected trigger operation on K5 in T2 corresponding to the rendered image, second image, or thumbnail of "receiving the corresponding associated information in response to the fifth operation"; during the process of displaying the corresponding associated information in response to the fifth operation, an operation opposite in direction to the fifth operation that is sliding or scrolling; during the process of displaying the corresponding associated information in response to the fifth operation, a detected double-click operation relative to the fifth operation that is a single click; during the process of displaying the corresponding associated information in response to the fifth operation, a detected single-click operation relative to the fifth operation that is a double click; and so on. Those skilled in the art can set the implementation method of the sixth operation according to actual needs, and this application does not limit this.

[0221] Related information display method 2:

[0222] When it is determined that the associated information to be displayed in response to the fifth operation includes images or videos, it can be as follows: Figure 5B As shown, pop-up window L5 is further displayed to the user in T2, where corresponding related information is shown. The implementation method for displaying related information in L5 can refer to the implementation method for displaying the rendered image, the second image, and the related video in T2, and will not be elaborated here.

[0223] Related information display method 3:

[0224] When it is determined that the associated information to be displayed in response to the fifth operation includes an image, it can be as follows: Figure 6A , Figure 7B The corresponding related information is displayed in T2 in the form of a stitched long image, that is... Figure 6A A long infographic showing the development process of YYY Academy, or Figure 7B The curly braces in T2 indicate a long image depicting the "evolutionary process of species." Furthermore, the related information long image displayed in T2 can be moved and its content changed by sliding, moving, and scrolling on the related information long image.

[0225] When the associated information to be displayed in response to the fifth operation includes an image, the display method can be set with reference to image display methods 1-5, and will not be elaborated further to avoid redundancy.

[0226] If it is determined that the server has not performed the aforementioned preprocessing on the second image before displaying it in full screen, the preprocessed second image or the rendered image obtained by rendering the preprocessed second image can be displayed in full screen after preprocessing.

[0227] Image display method 2:

[0228] If all second images are received, the user can also be shown images such as... Figure 4C , Figure 6A , Figure 7B The query results display interface T2 shown here. That is, multiple "second images" or "rendered images" are pre-stitched into a long image according to time sequence, which can be as follows: Figure 4C , Figure 6A , Figure 7B The long image shown, which is stitched together from left to right, can also be... Figure 4C , Figure 6A , Figure 7B The image is a long, vertically stitched image (not shown). Then, in T2, a portion of the long image, including or corresponding to the target second image, and the time axis Z are first displayed. Then, based on the detected user input regarding the long image or a fourth operation such as movement, sliding, or scrolling via physical keys, the long image is moved in the direction and distance to change the currently displayed second image or rendered image in T2. ​​Specifically, the long image is as follows... Figure 4C , Figure 6A , Figure 7B In the case of left and right splicing as shown, the operation direction can be as follows: Figure 4C , Figure 6A , Figure 7B The middle arrow indicates left or right. When the long image is stitched vertically, the direction of operation can be the same as... Figure 4C , Figure 6A , Figure 7B The middle arrow points vertically upwards or downwards.

[0229] Among them, such as Figure 4C , Figure 6A As shown, L4 in T2 also displays partial association information corresponding to each second image. Furthermore, when a fifth operation is detected for a specific second image or rendered image, the association information corresponding to the fifth operation is displayed to the user.

[0230] If it is determined that the server has not performed the aforementioned preprocessing on each of the second images before forming the long image, the second images can be preprocessed before forming the long image.

[0231] Image display method 3:

[0232] In the case of receiving all second images or a single target second image, it is also possible to provide users with, for example Figure 4DThe query result display interface T2 is shown below. Specifically, T2 first displays the target second image or a rendered image P1 corresponding to the target second image in full screen. Then, based on the direction and distance of the user's detected fourth operation (movement, sliding, scrolling, etc.) on the currently displayed image, the image currently displayed in T2 is switched to the next image corresponding to the fourth operation. During the switching of the full-screen image, the image is switched according to switching method one or two as described above.

[0233] Among them, such as Figure 4D As shown, T2 also displays some associated information corresponding to the second image or rendered image displayed in full screen via L4. Furthermore, if a fifth operation is detected targeting the second image or rendered image displayed in full screen, the associated information corresponding to the fifth operation is displayed to the user.

[0234] Image display method 4:

[0235] In the case of receiving all second images or a single target second image, it is also possible to provide users with, for example Figure 4E The query results display interface T2 is shown. The difference between image display method 4 and the image display method 3 mentioned above is that T2 also displays a switching control K11 for switching between a second image or a rendered image to achieve full-screen display.

[0236] The toggle control K11 can be any of the following: such as Figure 4E The control shown can scroll up or down as indicated by the arrow; it can scroll left or right; it can slide up or down; it can slide left or right; it can move up or down; it can move left or right. The toggle control K11 can be as follows: Figure 4E The image shown can be displayed separately in a designated area of ​​T2 without overlapping with the second image or rendered image. Alternatively, K11 can be displayed directly on top of the second image or rendered image displayed in full screen without setting a designated area.

[0237] During the detection of the fourth operation such as scrolling, sliding, or moving for K11, the second image or rendered image displayed in full screen is switched according to the operation direction and distance of the fourth operation and the preset switching method one or two mentioned above.

[0238] Image display method 5:

[0239] In the case of receiving all second images or a single target second image, it is also possible to provide users with, for example Figure 4F The query results display interface T2 is shown. The difference between image display method 5 and the image display method 4 mentioned above is that, according to the operation direction and distance of the fourth operation, the second image or the rendered image is switched to be displayed in full screen according to switching method three.

[0240] Among them, such as Figure 4F As shown, switching method three can be: upon detecting the fourth operation, first switch the second image or rendered image displayed in full screen in T2 to, as shown in the image below. Figure 4F The diagram shows multiple second images or multiple rendered images displayed in a stacked manner in T2. ​​The next second image or next rendered image to be displayed to the user when the fourth operation is stopped is highlighted with a border or similar design. The display of the next second image or next rendered image in the stacked imagery changes according to the direction and distance of the fourth operation. If it is determined that the user has stopped issuing the fourth operation, the next second image or next rendered image from the stacked images at the time the user stopped the fourth operation is displayed in full-screen mode.

[0241] The stacking method for multiple images displayed in a stacked format can be as follows: Figure 4F The stacking shown in the K11 configuration can also be horizontal stacking, or it can be a non-stacked method, such as directly displaying thumbnails of each second image or rendered image in a tiled format (in which case the thumbnail corresponding to the fourth operation in multiple thumbnails is highlighted), and so on. Those skilled in the art can set the implementation of the stacking or non-stacked form in the switching method of the second image or rendered image according to actual needs, and this application does not limit it in this regard.

[0242] Image display method 6:

[0243] Even after receiving all the second images, it is also possible to provide the user with, for example Figure 5A , Figure 5B , Figure 6B , Figure 7A , Figure 8 The query results display interface T2 is shown below. The difference between image display method 6 and the image display method 2 described above is that the entire long image is displayed in T2. ​​If the long image is too long, the individual second images or rendered images within it can be displayed as thumbnails. For example... Figure 5A As shown, when a fourth operation (such as a click, double-click, or long-press) is detected on a thumbnail in T2, the second image or rendered image selected by the user based on the fourth operation is displayed in full screen in T2. ​​The content displayed in T2 can be directly switched from a long image to the selected second image or the selected rendered image. Alternatively, when a fourth operation (such as a click, double-click, or long-press) is detected on a thumbnail in T2, the selected second image or the selected rendered image can also be displayed above the long image displayed in T2 in the form of a floating window.

[0244] It should be noted that the above-mentioned image display methods 1-6 are merely illustrative examples of query result display in this application. Those skilled in the art can also set them according to actual needs, and this application does not impose any restrictions on them.

[0245] In one possible implementation, if the terminal device receives a dynamic image or associated video from the server in step S13, or if it receives all the second images and displays them in a dynamic or video manner, then in step S14, the user is shown a full-screen display of the dynamic image or associated video in T2.

[0246] Among them, it can be like Figure 4G The display only shows dynamic images or associated videos in T2. ​​When displaying associated videos, a pause / start playback control K7 can also be shown to the user in T2. ​​If a trigger operation is detected on K7 during the playback of the associated video, playback of the associated video will stop; if a trigger operation is detected on K7 during the playback of the associated video, playback of the associated video will begin. The display state or method of K7 differs between the playback and paused states of the associated video, allowing the user to distinguish the current state.

[0247] Among them, such as Figure 4H As shown, Q1 can also be displayed in T2 while simultaneously displaying dynamic images or associated videos. Figure 4H The difference between Q1 in method 1 and Q1 in image display method 2 is that, when displaying dynamic images or associated videos, if a selection operation such as clicking, double-clicking, or long-pressing on the thumbnail is detected, or an operation such as sliding, moving, or scrolling on K4 is detected, the display of the dynamic image can be stopped, and the second image or rendered image corresponding to the selected thumbnail, determined based on the selection operation or the current position of K4, can be displayed in T2. ​​Furthermore, when displaying associated videos, the playback progress of the associated video can be adjusted based on the selection operation or the operation on K4, so that it corresponds to the second image or rendered image corresponding to the selected thumbnail.

[0248] In step S15, if the rendered first image is displayed to the user, the camera can be controlled to continue working, giving the user the visual experience that the rendered first image is rendered and overlaid directly on the image captured by the camera in real time. Then, operation detection can continue for the camera or the displayed rendered first image.

[0249] In one possible implementation, step S15 may further include: if a second operation is received, then in response to the second operation, sending second query information to the server. The second query information may include a third image. The second operation may be an operation that adds a portion different from the first image to the third image currently being captured in real time by the camera.

[0250] The second operation may include: detecting a change in the spatial position of the camera, such as a change in spatial position caused by the movement of a user holding or carrying a terminal device, resulting in a new portion of the third image that differs from the first image. This change in the camera's spatial position includes any one of the following movements corresponding to the target object in the first image: moving left, right, backward, upward, or downward; or at least two of the following movements: moving left, right, backward, upward, downward, or forward. The camera's focal length is adjusted (not by increasing). The camera's shooting angle changes. And so on.

[0251] The second query information may include the third image and location information determined by the terminal device. Alternatively, the second query information may include information obtained by replacing the first image in the first query information with the second image. Or, the second query information may be incremental information relative to the first query information. And so on. The method for determining the second query information is similar to the methods for determining the first query information described in steps one to three above, and will not be elaborated upon here.

[0252] In one possible implementation, the second operation may further include magnifying the third image currently being captured in real-time by the camera. The user can zoom in or out by manipulating the touchscreen of the terminal device, for example, by moving two fingers in opposite directions. Alternatively, the user can control physical buttons or the zoom controls displayed on T2 to perform the zoom operation. This ensures that whenever a change is detected in the third image being captured in real-time by the camera (i.e., magnification or the addition of a portion different from the first image), a second query is sent to the server to re-query, guaranteeing that the query results seen by the user are always consistent with the image captured in real-time by the camera, providing the user with an immersive information query experience.

[0253] In one possible implementation, step S15 may further include: receiving a third operation, and magnifying and displaying the rendered first image or the rendered third image according to the third operation. The third operation includes magnifying the image captured by the camera in real time, so the image currently captured by the camera in real time is a magnified version of a portion of the first image or the third image. This allows for direct magnification and display of the rendered first image or the rendered third image when it is determined that the user needs to zoom in on the content displayed on the current interface, thereby improving the speed of responding to user operations.

[0254] In step S16, if the second query information is determined, the second query information is sent to the server. This allows the server to retrieve and determine multiple matching fourth images based on the received second query information (step S23), and then send the multiple fourth images to the terminal device (step S24). The server's execution of steps S23 and S24 is the same as steps S21 and S22 described above, and will not be repeated here.

[0255] In step S17, an image is displayed to the user based on the received fourth image. The implementation process of step S17 is similar to that of step S14, and can be referred to above, so it will not be repeated here.

[0256] By responding to the second and third operations, the terminal device can promptly display a rendered image corresponding to the currently captured image by the camera when the image captured in real time changes.

[0257] Figure 9 This diagram illustrates the rendering of a third image in an information query and display method according to an embodiment of this application. To clearly explain the process by which the terminal device of this application switches the displayed content in T2 in response to a second or third operation, the following is combined with... Figure 9 Please provide an explanation.

[0258] like Figure 9 As shown, if the terminal device receives a second operation, and this second operation switches the image captured in real time by the camera from the first image to the third image 1, where the third image 1 includes an additional region corresponding to area S compared to the first image, then the terminal device sends a second query to the server. Based on the fourth image returned by the server, it further displays the rendered third image 1 to the user in T2, where the rendered third image 1 includes an additional region S' compared to the rendered first image. It should be noted that, to save bandwidth, the terminal device can send either the additional portion of the third image compared to the first image or the complete third image.

[0259] like Figure 9 As shown, if the terminal device receives a third operation, and this third operation causes the image captured by the camera in real time to switch from the first image to the third image 2, where the third image 2 is a magnified image of the portion corresponding to region D of the first image, then the user can be directly shown the magnified image of region D of the rendered first image in T2.

[0260] This application also provides an information query and display device, applied to a terminal device, the device comprising:

[0261] The first display module displays the first image captured in real time by the camera of the terminal device in the first image interface;

[0262] The first operation receiving module receives a first operation for the first image, the first operation including a sliding operation or a scrolling operation.

[0263] A first sending module, in response to the first operation, sends first query information to the server, the first query information including the first image;

[0264] A first image receiving module receives a second image sent by the server in response to the first query information, the second image including an image that is time-related to the first image in the first query information;

[0265] The first rendering processing module renders a first part of the first image according to the second image. The rendering processing of the first part of the first image includes replacing the first part of the first image with the second image.

[0266] The first rendered image display module displays the rendered first image, which includes: a first part that has been rendered and a second part of the unrendered first image excluding the first part;

[0267] The second sending module, in response to a received second operation, sends second query information to the server. The second operation includes adding a portion different from the first image to the third image currently being captured in real time by the camera. The second query information includes the third image.

[0268] The second image receiving module receives a fourth image sent by the server in response to the second query information, the fourth image including an image that is time-related to the third image in the second query information;

[0269] The second rendering processing module renders the first part of the third image according to the fourth image. The rendering processing of the first part of the third image includes replacing the first part of the third image with the fourth image.

[0270] The second rendered image display module displays a rendered third image, which includes: a first part that has been rendered and a second part of the third image that has not been rendered, excluding the first part.

[0271] In this way, the terminal device can send the first query information, including the first image, determined based on the user's first operation, to the server, and display the rendered first image obtained by rendering the first image based on the second image returned by the server in response to the first query information. This allows the user to directly query information while viewing the first image captured in real time by the camera on the interface, and the rendered first image is displayed on the interface, bringing the user an immersive information query experience. Moreover, the query speed is fast, the accuracy is high, and the time correlation between the displayed rendered first image and the first query information is higher.

[0272] Furthermore, during the display of the rendered first image, a second query message is sent in response to the second operation. This ensures that as long as it is determined that the image currently captured by the camera in real time has changed from the first image to a third image with additional parts different from the first image, the second query message can be sent to the server to re-query the information. This ensures that the query results seen by the user are always consistent with the image captured by the camera in real time, enhancing the viewing experience of immersive information query.

[0273] In one possible implementation, the second operation further includes magnifying the third image currently being captured in real-time by the camera. This ensures that whenever a change is detected in the third image (i.e., magnification or addition of a portion different from the first image), a second query can be sent to the server to re-query the information, guaranteeing that the query results seen by the user are always consistent with the image captured in real-time by the camera.

[0274] In one possible implementation, the device may further include:

[0275] The third rendered image display module receives a third operation and, based on the third operation, enlarges and displays either the rendered first image or the rendered third image. The third operation includes enlarging the image captured in real-time by the camera. This allows for direct enlargement and display of either the rendered first image or the rendered third image when it is determined that the user needs to zoom in on the content displayed on the current interface, thus improving the speed of responding to user actions.

[0276] In one possible implementation, the first graphical interface further displays a first control, and the first operation includes a trigger operation on the first control. The first control allows the user to directly initiate the first operation by manipulating it, simplifying the process and improving the speed of determining the first query information.

[0277] In one possible implementation, the first query information and / or the second query information further include location information of the terminal device. Adding location information can improve the accuracy of the query results returned by the server (i.e., the returned second image and / or fourth image).

[0278] In one possible implementation, the device may further include: an information editing module, used for:

[0279] Before sending the first query information to the server, the first image is edited according to the detected editing operations on the first image; and / or

[0280] Before sending the second query information to the server, the third image is edited according to the detected editing operations on the third image.

[0281] Before issuing the first and / or second query information, the first and / or second query information can be edited according to the user's operation. This can remove information that is not related to the target object that the user wants to query, thereby removing redundant information. This makes the second image retrieved by the server based on the edited first query information and / or the fourth image retrieved based on the edited second query information more accurate and better meet the user's query needs.

[0282] In one possible implementation, the rendered image is displayed in any of the following ways, wherein the rendered image includes the rendered first image and / or the rendered third image:

[0283] One of the rendered images is displayed in full screen on the query results display interface;

[0284] The query results display interface displays part or all of the long image formed by stitching together all the rendered images;

[0285] The query results display interface shows a dynamic image formed from multiple rendered images;

[0286] The query results display interface shows the associated video formed from the multiple rendered images.

[0287] By using various rendering image display methods, the display of rendered images can be diversified, which can meet the different display needs of users.

[0288] In one possible implementation, the query results display interface also displays a prompt message, which reminds the user to confirm whether the currently displayed rendered image is correct. The device may further include:

[0289] The third sending module sends the information determined based on the detected operation in response to the prompt information, and the fifth image captured in real time by the camera, as the third query information to the server; or

[0290] The supplementary information sending module sends the information determined based on the detected operation in response to the prompt information to the server as supplementary information to the first query information or the second query information.

[0291] By displaying prompts, users can take further steps to perform a new search or supplement their initial search information if they find issues such as incorrect or inaccurate rendered or second images that do not meet their search requirements.

[0292] In one possible implementation, the query results display interface also shows some related information corresponding to the rendered image.

[0293] The associated information comes from the server and is information related to the second or fourth image corresponding to the rendered image. The associated information includes at least one of the following: time information corresponding to the second or fourth image, event information related to the second or fourth image, character information related to the second or fourth image, and description information of the target object in the first or third image.

[0294] Displaying some associated information corresponding to the rendered image can increase the amount of information in the rendered image, allowing users to further understand the changes in the rendered image by combining the associated information.

[0295] In one possible implementation, the first area of ​​the query results display interface also displays thumbnails corresponding to each of the second or fourth images, arranged sequentially in chronological order. Displaying thumbnails allows users to first get a general understanding of how the information they are querying changes over time, and then select the images they need to view in detail as needed.

[0296] In one possible implementation, the first area also displays time information for a second or fourth image corresponding to each thumbnail. This allows users to further understand the time corresponding to the second or fourth image by combining the time information.

[0297] In one possible implementation, the device may further include:

[0298] The fourth operation receiving module receives a fourth operation for the currently displayed rendered image, long image, or thumbnail.

[0299] The image switching module, in response to the fourth operation, switches the currently displayed content to the rendered first image or the rendered third image corresponding to the fourth operation;

[0300] The fourth operation includes at least one of the following operations on the currently displayed rendered image, the corresponding physical key on the terminal device, or the thumbnail: swipe operation, scroll operation, move operation, click operation, double-click operation, and long-press operation.

[0301] In one possible implementation, when a rendered image is displayed in full screen on the query results display interface, the currently displayed content is switched to the rendered first image or the rendered third image corresponding to the fourth operation in response to the fourth operation, including any one of the following switching methods:

[0302] According to the gradient speed determined by the operation speed of the fourth operation, the current rendered image and the next rendered image are switched in a gradient manner in full screen. The gradient manner includes: displaying a first object that exists in the current rendered image but does not exist in the next rendered image in a gradually disappearing manner until the switch is completed, and displaying a second object that does not exist in the current rendered image but exists in the next rendered image in a gradually appearing manner until the switch is completed.

[0303] According to the movement speed determined by the operation speed of the fourth operation, the currently rendered image displayed in full screen is moved out of the query result display interface and the next rendered image is moved into the query result display interface. During the movement, the query result display interface includes a portion of the currently rendered image and a portion of the next rendered image.

[0304] This allows users to switch image displays according to their viewing needs, satisfying their preferences for different images. At the same time, the customized image switching methods provide users with a better visual experience.

[0305] In one possible implementation, the device further includes:

[0306] Receive a fifth operation for the currently displayed rendered image, long image, or thumbnail;

[0307] Display the associated information corresponding to the fifth operation.

[0308] In one possible implementation, the associated information corresponding to the fifth operation is displayed, including any of the following methods:

[0309] The information bar in the query results display interface displays the associated information corresponding to the fifth operation;

[0310] The pop-up window above the query results display interface displays the associated information corresponding to the fifth operation;

[0311] If the associated information corresponding to the fifth operation includes an image, part or all of the long image formed by stitching together the associated information will be displayed in the query results display interface.

[0312] This allows users to not only understand the changes and development of the information they want to query based on the displayed rendered images, long images, or thumbnails, but also to gain a deeper understanding of the events, people, and other information related to the information they want to query by combining related information.

[0313] In one possible implementation, the first sending module is further configured to, during the display of the first image, if it is detected that the duration for which the first image remains unchanged exceeds a duration threshold, then send the first query information to the server. This simplifies the operations performed by the user when determining the first query information.

[0314] In one possible implementation, the device may further include:

[0315] The deletion module removes irrelevant objects from the rendered image before displaying it.

[0316] This avoids irrelevant objects from interfering with the user's view of the rendered image, ensuring the accuracy of the rendered image displayed to the user.

[0317] The working principles and processes of each module of the information query and display device provided in this application, as well as the technical effects they can achieve, can be found in the description of the response steps in the information query and display method section above, and will not be repeated here.

[0318] Figure 10 This diagram illustrates the structure of a terminal device 100.

[0319] Terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) connector 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0320] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0321] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0322] The processor 110 can generate operation control signals based on the instruction opcode and timing signals to control the instruction fetching and execution. When executing instructions, the processor 110 can perform steps S11-S14, etc., as executed by the terminal device in the aforementioned information query and display method.

[0323] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 may be a cache memory. This memory can store instructions or data that the processor 110 has used or that are used frequently. If the processor 110 needs to use the instruction or data, it can directly retrieve it from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0324] In some embodiments, the processor 110 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. The processor 110 can connect to modules such as touch sensors, audio modules, wireless communication modules, displays, and cameras through at least one of these interfaces.

[0325] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may also adopt different interface connection methods or a combination of multiple interface connection methods as described in the above embodiments.

[0326] The display screen 194 is used to display images, videos, etc. The display screen 194 is used to display content to the user in the aforementioned information query and display method under the control of the processor 110. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include one or more display screens 194.

[0327] Terminal device 100 can realize camera function through camera module, ISP, video codec, GPU, display screen 194, application processor AP, neural network processor NPU, etc.

[0328] The camera module can be used to acquire color image data and depth data of the subject. The Information Signal Processor (ISP) can be used to process the color image data acquired by the camera module. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits this electrical signal to the ISP for processing, converting it into a visible image. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set within the camera module. The camera module is used, under the control of the processor 110, to acquire the viewfinder image and capture a first image, a third image, or a first video.

[0329] In some embodiments, the camera module may consist of a color camera module and a 3D sensing module.

[0330] In some embodiments, the photosensitive element of the camera in the color camera module can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats.

[0331] In some embodiments, the 3D sensing module can be a time-of-flight (TOF) 3D sensing module or a structured light 3D sensing module. Structured light 3D sensing is an active depth sensing technology. The basic components of a structured light 3D sensing module may include an infrared emitter, an IR camera module, etc. The working principle of a structured light 3D sensing module is to first emit a specific pattern of light onto the object being photographed, then receive the light coding on the object's surface, compare it with the original projected light pattern, and calculate the object's three-dimensional coordinates using triangulation principles. These three-dimensional coordinates include the distance between the terminal device 100 and the object being photographed. Similarly, TOF 3D sensing can be an active depth sensing technology. The basic components of a TOF 3D sensing module may include an infrared emitter, an IR camera module, etc. The working principle of a TOF 3D sensing module is to calculate the distance (i.e., depth) between the TOF 3D sensing module and the object being photographed by measuring the infrared reflection time to obtain a 3D depth map.

[0332] Structured light 3D sensing modules can also be applied to fields such as facial recognition, motion-sensing game consoles, and industrial machine vision inspection. Time-of-flight (TOF) 3D sensing modules can also be applied to fields such as game consoles, augmented reality (AR) / virtual reality (VR).

[0333] In other embodiments, the camera module may consist of two or more cameras. These two or more cameras may include a color camera, which can be used to acquire color image data of the object being photographed. These two or more cameras may employ stereo vision technology to acquire depth data of the object being photographed. Stereo vision technology is based on the principle of human parallax. Under natural light, two or more cameras capture images of the same object from different angles, and then triangulation and other calculations are performed to obtain the distance information, i.e., depth information, between the terminal device 100 and the object being photographed.

[0334] In some embodiments, the terminal device 100 may include one or more camera modules. Specifically, the terminal device 100 may include one front-facing camera module and one rear-facing camera module. The front-facing camera module is typically used to capture color image data and depth data of the photographer facing the display screen 194, while the rear-facing camera module is used to capture color image data and depth data of the subject (such as a person, landscape, etc.) being photographed.

[0335] A digital signal processor (DSP) is used to process digital signals, and can also process other digital signals. For example, when the terminal device 100 selects a frequency point, the DSP is used to perform Fourier transforms on the frequency energy, etc.

[0336] Video codecs are used to compress or decompress digital video. Terminal device 100 may support one or more video codecs. Thus, terminal device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0337] Terminal device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0338] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0339] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music through the speaker 170A or output audio signals for hands-free calling.

[0340] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the terminal device 100 answers a phone call or voice message, the receiver 170B can be brought close to the listener's ear to hear the voice.

[0341] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Terminal device 100 may be equipped with at least one microphone 170C. In some embodiments, terminal device 100 may be equipped with two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, terminal device 100 may be equipped with three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc. Microphone 170C can perform sound signal detection during the process of obtaining query information, acquire the user's voice input, and send it to processor 110.

[0342] Touch sensor 180K, also known as a "touch device," can be disposed on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K is used to detect touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be disposed on the surface of terminal device 100, in a different location than display screen 194. Touch sensor 180K is used to detect touch operations issued by the user and send them to processor 110, so that processor 110 can further determine whether it belongs to the operation involved in the above-described information query and display method, and if it is an operation involved in the above-described information query and display method, further determine what kind of operation it is.

[0343] Button 190 may include a power button, volume buttons, etc. Button 190 may be a mechanical button or a touch button. Terminal device 100 may receive button input and generate key signal inputs related to user settings and function control of terminal device 100.

[0344] An embodiment of this application provides an information query and display device, including: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions.

[0345] Embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0346] Embodiments of this application provide a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in the processor of a terminal device, the processor in the terminal device executes the above-described method.

[0347] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital video disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing.

[0348] The computer-readable program instructions or code described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0349] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this application.

[0350] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0351] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0352] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0353] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0354] It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented using hardware (such as circuits or ASICs (Application Specific Integrated Circuits)) that performs the corresponding function or action, or using a combination of hardware and software, such as firmware.

[0355] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0356] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An information query and display method, characterized in that, Applied to a terminal device, the method includes: The first image captured in real time by the camera of the terminal device is displayed in the first image interface; Receive a first operation on the first image, the first operation including a sliding operation or a scrolling operation; In response to the first operation, first query information is sent to the server, the first query information including the first image; Receive a second image sent by the server in response to the first query information, the second image including an image that is time-related to the first image in the first query information; The first part of the first image is rendered according to the second image, and the rendering process of the first part of the first image includes replacing the first part of the first image with the second image; The rendered first image is displayed, which includes: a first part that has been rendered and a second part of the unrendered first image other than the first part; In response to the received second operation, second query information is sent to the server. The second operation includes adding a portion different from the first image to the third image currently being captured in real time by the camera. The second query information includes the third image. Receive a fourth image sent by the server in response to the second query information, the fourth image including an image that is time-related to the third image in the second query information; The rendering process of the first part of the third image is performed based on the fourth image, and the rendering process of the first part of the third image includes replacing the first part of the third image with the fourth image; The rendered third image is displayed, which includes: a first part that has been rendered and a second part of the unrendered third image other than the first part.

2. The method according to claim 1, characterized in that, The second operation also includes the operation of magnifying the third image currently being captured in real time by the camera.

3. The method according to claim 1, characterized in that, The method further includes: The system receives a third operation and, based on the third operation, enlarges and displays the rendered first image or the rendered third image. The third operation includes the operation of enlarging the image captured by the camera in real time.

4. The method according to any one of claims 1-3, characterized in that, The first graphical interface also displays a first control, and the first operation further includes a triggering operation on the first control.

5. The method according to any one of claims 1-3, characterized in that, The first query information and / or the second query information also include location information of the terminal device.

6. The method according to any one of claims 1-3, characterized in that, The method further includes: Before sending the first query information to the server, the first image is edited according to the detected editing operations on the first image; and / or Before sending the second query information to the server, the third image is edited according to the detected editing operations on the third image.

7. The method according to claim 1, characterized in that, The rendered image can be displayed in any of the following ways, wherein the rendered image includes the rendered first image and / or the rendered third image: One of the rendered images is displayed in full screen on the query results display interface; The query results display interface displays part or all of the long image formed by stitching together all the rendered images; The query results display interface shows a dynamic image formed from multiple rendered images; The query results display interface shows the associated video formed from the multiple rendered images.

8. The method according to claim 7, characterized in that, The query results display interface also shows a prompt message, which reminds the user to confirm whether the currently displayed rendered image is correct. The method further includes: The information determined based on the detected operation in response to the prompt message, and the fifth image captured in real time by the camera, are sent to the server as the third query information; or The information determined based on the detected operation in response to the prompt information will be sent to the server as supplementary information to the first query information or the second query information.

9. The method according to claim 7 or 8, characterized in that, The query results display interface also shows some related information corresponding to the rendered image. The associated information comes from the server and is information related to the second or fourth image corresponding to the rendered image. The associated information includes at least one of the following: time information corresponding to the second or fourth image, event information related to the second or fourth image, character information related to the second or fourth image, and description information of the target object in the first or third image.

10. The method according to claim 9, characterized in that, The first area of ​​the query results display interface also displays thumbnails corresponding to each of the second or fourth images arranged in chronological order.

11. The method according to claim 10, characterized in that, The first area also displays the time information of the second or fourth image corresponding to each of the thumbnails.

12. The method according to claim 10, characterized in that, The method further includes: Receive a fourth operation for the currently displayed rendered image, the long image, or the thumbnail; In response to the fourth operation, the currently displayed content is switched to the rendered first image or the rendered third image corresponding to the fourth operation; The fourth operation includes at least one of the following operations on the currently displayed rendered image, the corresponding physical key on the terminal device, or the thumbnail: swipe operation, scroll operation, move operation, click operation, double-click operation, and long-press operation.

13. The method according to claim 12, characterized in that, When a rendered image is displayed in full screen in the query results display interface, in response to the fourth operation, the currently displayed content is switched to the rendered first image or the rendered third image corresponding to the fourth operation, including any one of the following switching methods: According to the gradient speed determined by the operation speed of the fourth operation, the current rendered image and the next rendered image are switched in a gradient manner in full screen. The gradient manner includes: displaying a first object that exists in the current rendered image but does not exist in the next rendered image in a gradually disappearing manner until the switch is completed, and displaying a second object that does not exist in the current rendered image but exists in the next rendered image in a gradually appearing manner until the switch is completed. According to the movement speed determined by the operation speed of the fourth operation, the currently rendered image displayed in full screen is moved out of the query result display interface and the next rendered image is moved into the query result display interface. During the movement, the query result display interface includes a portion of the currently rendered image and a portion of the next rendered image.

14. The method according to claim 10, characterized in that, The method further includes: Receive a fifth operation for the currently displayed rendered image, long image, or thumbnail; Display the associated information corresponding to the fifth operation.

15. The method according to claim 14, characterized in that, Displaying the associated information corresponding to the fifth operation, including any of the following methods: The information bar in the query results display interface displays the associated information corresponding to the fifth operation; The pop-up window above the query results display interface displays the associated information corresponding to the fifth operation; If the associated information corresponding to the fifth operation includes an image, part or all of the long image formed by stitching together the associated information will be displayed in the query results display interface.

16. The method according to any one of claims 1-3, characterized in that, The method further includes: During the display of the first image, if it is detected that the duration during which the first image remains unchanged exceeds a duration threshold, the first query information is sent to the server.

17. The method according to claim 7, characterized in that, The method further includes: Before displaying the rendered image, remove irrelevant objects from the rendered image.

18. An information display device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1-17 when executing the instructions.

19. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-17.

20. A computer program product comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, characterized in that, When the computer-readable code is run in an electronic device, the processor in the electronic device performs the method according to any one of claims 1-17.

Images