Information acquisition method and device in driving, electronic equipment and storage medium

Abstract

The invention relates to an information acquisition method and device in driving, electronic equipment and a storage medium. The method comprises the following steps: in response to a received preset instruction, controlling an associated camera device to shoot; extracting candidate pictures from the media data acquired by the associated camera device; displaying the candidate pictures; in response to a selection operation on the candidate pictures, determining a target picture; and performing intelligent search based on the target picture to obtain target information. By adopting the method, the user in the driving process can conveniently and timely obtain the target information.

Description

Technical Field

[0001] The present disclosure relates to the technical field of terminals, and in particular, to a method, an apparatus, an electronic device, and a storage medium for obtaining information during driving. Background Art

[0002] Image recognition search is a very convenient function that can recognize pictures and search for information related to the picture content. This function can help users conveniently meet some needs, such as recognizing things, searching for related products, etc. In the process of implementing the solution of the present disclosure, the inventors found that when a user encounters the need to search for information about a related object (such as a scenery like a mountain, animals, or plants) during driving, due to reasons such as the user being unable to take pictures in time during driving, the shooting angle and range of the mobile phone being relatively limited, and the pictures taken being inconvenient to find, the user cannot conveniently and timely obtain pictures of the related object, and thus cannot perform image recognition search to obtain information related to the related object. Summary of the Invention

[0003] Based on this, it is necessary to provide a method, an apparatus, an electronic device, and a storage medium for obtaining information during driving that can improve the convenience and timeliness of information acquisition for the above-mentioned technical problems.

[0004] According to a first aspect of certain exemplary embodiments of the present disclosure, there is provided a method for obtaining information during driving. The method is applied to an electronic device and includes: in response to receiving a preset instruction, controlling an associated imaging device to take a picture; extracting candidate pictures from the media data obtained through the associated imaging device; displaying the candidate pictures; in response to a selection operation on the candidate pictures, determining a target picture; and performing an intelligent search based on the target picture to obtain target information.

[0005] In some embodiments, before controlling the associated imaging device to take a picture, the method further includes: based on the preset instruction, determining shooting orientation information; and based on the shooting orientation information, determining the associated imaging device from all the imaging devices connected to the electronic device.

[0006] In some embodiments, extracting candidate pictures from the media data obtained through the associated imaging device includes: determining whether the preset instruction contains object information; in response to the preset instruction containing object information, extracting multiple pictures that contain the object information and whose image quality meets a first preset condition from the media data obtained through the associated imaging device as candidate pictures; and in response to the preset instruction not containing object information, extracting multiple pictures whose image quality meets the first preset condition from the media data as candidate pictures.

[0007] In some embodiments, determining whether the preset instruction contains object information includes: detecting the input type of the preset instruction; in response to the input type of the preset instruction being voice input, determining whether the preset instruction contains object information according to the semantic analysis result of the preset instruction; in response to the input type of the preset instruction being a specific operation, determining whether the preset instruction contains object information according to the operation form of the specific operation.

[0008] In some embodiments, after extracting candidate pictures from the media data obtained by the associated camera device, it further includes: storing the candidate pictures on the electronic device.

[0009] In some embodiments, performing an intelligent search based on the target picture to obtain target information includes: performing image recognition on the target picture to obtain an image recognition result; performing a search based on the image recognition result to obtain target information.

[0010] In some embodiments, after obtaining the target information, the method further includes: presenting the target information in a preset manner, where the preset manner includes: voice broadcast, and / or, card-style display.

[0011] According to a second aspect of certain exemplary embodiments of the present disclosure, there is provided an information acquisition device during driving, the device includes: a shooting control module, configured to control the associated camera device to shoot in response to receiving a preset instruction; a candidate picture extraction module, configured to extract candidate pictures from the media data obtained by the associated camera device; a picture display module, configured to display the candidate pictures; a picture determination module, configured to determine a target picture in response to a selection operation on the candidate picture; an information search module, configured to perform an intelligent search based on the target picture to obtain target information.

[0012] In some embodiments, the device further includes a direction determination module and a camera device determination module. The direction determination module is configured to determine shooting direction information based on the preset instruction; the camera device determination module is configured to determine the associated camera device from all the camera devices connected to the electronic device based on the shooting direction information.

[0013] In some embodiments, the candidate picture extraction module determines whether the preset instruction contains object information; in response to the preset instruction containing object information, extracting multiple pictures that contain object information and whose image quality meets a first preset condition from the media data obtained by the associated camera device as candidate pictures; in response to the preset instruction not containing object information, extracting multiple pictures whose image quality meets the first preset condition from the media data as candidate pictures.

[0014] In some embodiments, the candidate picture extraction module detects the input type of a preset instruction; in response to the input type of the preset instruction being voice input, it determines whether the preset instruction contains object information according to the semantic analysis result of the preset instruction; in response to the input type of the preset instruction being a specific operation, it determines whether the preset instruction contains object information according to the operation form of the specific operation.

[0015] In some embodiments, the device further includes a picture storage module, which is used to store candidate pictures on the electronic device.

[0016] In some embodiments, the information search module performs image recognition on the target picture to obtain an image recognition result; and performs a search based on the image recognition result to obtain target information.

[0017] In some embodiments, the device further includes an information presentation module. The information presentation module presents the target information in a preset manner, where the preset manner includes: voice broadcast, and / or, card-style display.

[0018] According to a third aspect of certain exemplary embodiments of the present disclosure, there is provided an electronic device including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the information acquisition method during driving provided in any of the above embodiments.

[0019] According to a fourth aspect of certain exemplary embodiments of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon. When the processor executes the computer program, it implements the steps of the information acquisition method during driving provided in any of the above embodiments.

[0020] For the above-mentioned information acquisition method, device, electronic device, and storage medium during driving, when a user wants to know information about a certain object, the user can send a preset instruction to the electronic device. In response to receiving the preset instruction, the electronic device controls an associated imaging device to take a picture, and then automatically extracts and displays candidate pictures from the media data obtained through the associated imaging device. The user can select a target picture from the candidate pictures. The electronic device determines the target picture in response to the selection operation on the candidate picture, and then performs an intelligent search based on the target picture to obtain target information. In the above solution, the electronic device can control the associated imaging device to take pictures, extract candidate pictures from the media data captured by these imaging devices for the user to select, and automatically perform an intelligent search to obtain target information after the user completes the selection. The above solution allows the user to obtain target information only by triggering a preset instruction and performing a selection operation, improving the convenience and timeliness of the user's information acquisition during driving, and enabling the user to obtain a better information acquisition experience during driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of the application environment of the information acquisition method during driving according to the present disclosure;

[0022] Figure 2 Schematic flowchart of the information acquisition method during driving in an exemplary embodiment of the present disclosure;

[0023] Figure 3 Schematic flowchart of the step of extracting candidate pictures from media data obtained by an associated imaging device in some exemplary embodiments of the present disclosure;

[0024] Figure 4 Schematic flowchart of the step of determining an associated imaging device in some exemplary embodiments of the present disclosure;

[0025] Figure 5 Schematic diagram of two candidate pictures in an exemplary embodiment of the present disclosure;

[0026] Figure 6 Schematic diagram of the display effect of target information in an exemplary embodiment of the present disclosure;

[0027] Figure 7 Block diagram of the structure of the information acquisition device during driving in an exemplary embodiment of the present disclosure;

[0028] Figure 8 Internal structure diagram of an electronic device in an exemplary embodiment of the present disclosure. Detailed implementation manners

[0029] The following detailed implementation manners are provided to help the reader obtain a comprehensive understanding of the methods, devices, terminals, and / or storage media described herein. However, various changes, modifications, and equivalents of the methods, devices, terminals, and / or storage media described herein will be apparent after understanding the disclosure of the present disclosure. For example, the order of operations described herein is merely an example and is not limited to those set forth herein, but may be changed as will be apparent after understanding the disclosure of the present disclosure, except for operations that must occur in a specific order. In addition, descriptions of features known in the art may be omitted for greater clarity and conciseness.

[0030] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Instead, the examples described herein are provided only to illustrate some of the many possible ways of implementing the methods, terminals, and / or storage media described herein, many of which will be apparent after understanding the present disclosure.

[0031] The terms used herein are for describing various examples only and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular forms are also intended to include the plural forms. The terms "comprising", "including" and "having" specify the presence of the recited features, quantities, operations, components, elements and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof. Unless otherwise specified, " / " means "or", for example, A / B may mean A or B; "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present invention, "a plurality of" means two or more than two.

[0032] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this disclosure pertains after understanding this disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and shall not be interpreted in an idealized or overly formal manner.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of this disclosure and the above-mentioned drawings are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in some of the following exemplary embodiments do not represent all embodiments consistent with this disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0034] In addition, in the description of the examples, when a detailed description of a related structure or function that is considered to be well-known will cause a blurred interpretation of this disclosure, such a detailed description will be omitted.

[0035] Hereinafter, the embodiments will be described in detail with reference to the drawings. However, the embodiments can be implemented in various forms and are not limited to the examples described herein.

[0036] In order to better understand the technical solutions provided by this disclosure, the deficiencies or defects existing in the related art will be described in more detail below.

[0037] In the related art, users can conveniently obtain relevant information about an object through the image recognition search function. Different from the common searches in the past, the input of the image recognition search is not keywords but pictures. The search engine can identify the content of the pictures through technologies such as image recognition, determine which objects are included in the pictures, and then conduct searches based on the recognition results to obtain relevant information about one or more objects in the pictures. For example, when a user sees a plant on the road and thinks it is very beautiful and wants to know what kind of plant it is, the user can take a photo of the plant with a mobile phone and then conduct an image recognition search. The results may include, but are not limited to, the scientific name, variety, morphological characteristics, distribution by origin, relevant pictures and / or videos, and other information related to the plant. Since users can easily obtain pictures of relevant objects in most cases, users can conveniently obtain relevant information about relevant objects through image recognition search. However, in some specific situations, for example, when the user is in a moving vehicle and the object is outside the vehicle, it is inconvenient for the user to obtain a picture of the relevant object, and thus the function of image recognition search cannot be smoothly used to obtain relevant information about the relevant object. The following uses an example to illustrate the above problem. In an example, assume that the user is Xiaoming, a travel enthusiast. Xiaoming is on a solo road trip and sees a group of wolves chasing and hunting an animal on the way. Xiaoming wants to know what kind of animal it is, but he doesn't know the variety of the animal and can't accurately describe it. If he could take a photo of the animal, he could conduct an object recognition search. However, Xiaoming is driving at this time and it is not convenient to take a photo with his mobile phone. If he stops to take a photo, he may miss this moment, and there are also risks when stopping on the road (such as being hit by a vehicle coming from behind / attacked by wild animals, etc.); the driving recorder in the vehicle may record the process of the wolves hunting. Xiaoming can find the video frame containing the animal in the video recorded by the driving recorder and use this video frame for image recognition search. However, this method is relatively cumbersome to operate and requires operations such as exporting the video, cropping or taking screenshots of the video. Moreover, Xiaoming cannot ensure that the driving recorder has recorded the hunting process. Therefore, it is possible that he watches the entire video and only then discovers that the wonderful moment has not been recorded, which is time-consuming and laborious and may not yield good results. Therefore, when Xiaoming sees an animal being hunted by wolves outside the vehicle while in the car, it is difficult for him to conveniently and timely obtain a photo of the animal, and thus he cannot obtain relevant information about the animal.

[0038] To overcome the deficiencies or defects existing in the related art, the present disclosure provides a method for obtaining information during driving, which enables users during driving to conveniently and timely obtain target information.

[0039] The method for obtaining information during driving provided by the present disclosure can be applied to the application environment as Figure 1 shown.

[0040] The electronic device and the imaging device are connected in a wired or wireless manner, so that the electronic device can be linked with the multiple imaging devices through this connection. That is, the electronic device can instruct the imaging device to take pictures through this connection, and the imaging device can transmit the captured media data to the electronic device through this connection. Among them, the electronic device can be a mobile intelligent terminal such as a laptop computer, a smart phone, a tablet computer, a portable wearable device, etc. The imaging device is other equipment including a camera, usually multiple, such as a driving recorder, a car machine, and other electronic devices. One or more driving recorders can be installed on the vehicle where the user is located, and the electronic device can be linked with some or all of the driving recorders. One or more vehicle-mounted cameras capable of photographing external objects of the vehicle can be installed on the vehicle, and these vehicle-mounted cameras can take pictures under the control of the car machine. The electronic device can indirectly control some or all of these vehicle-mounted cameras to take pictures by linking with the car machine.

[0041] In a schematic embodiment of the present disclosure, as Figure 2 shown, a method for obtaining information during driving is provided. Taking the electronic device in Figure 1 as an example, the method includes the following steps:

[0042] Step S110, in response to receiving a preset instruction, control the associated imaging device to take pictures.

[0043] The electronic device has the function of image recognition and search. When the user needs to obtain information about a target object, the user can trigger a preset instruction to start the image recognition and search function to instruct the electronic device to execute the steps in the method provided in this embodiment.

[0044] The user can trigger the preset instruction in various ways. For example, the user can perform voice interaction with the electronic device and trigger the preset instruction through a voice command, such as "Identify that thing", "Search the building in front", "Identify the big tree on the left". Specifically, after receiving the user's voice command, the electronic device can analyze the content of the voice command to identify the user's intention. When the user has the intention of image recognition and search, the preset instruction is triggered. Among them, the operation of identifying whether the user has the intention of image recognition and search can be implemented by using intention recognition technology. Another example is that the user can trigger the preset instruction through a pre-configured operation for quickly starting the image recognition and search function (which can be simply referred to as a quick start operation). The quick start operation can be any operation, such as double-clicking the screen, clicking a specific button, recognizing a specific gesture, etc.

[0045] The target object refers to the object that the user wants to obtain relevant information about, usually an external object of the vehicle, such as plants, animals, scenery, buildings, vehicles, etc. outside the vehicle.

[0046] After receiving a preset instruction, the electronic device can send a control instruction to an associated camera device through a connection to control the associated camera device to take pictures. The data obtained by the associated camera device during shooting is media data, which can be one or more pictures and / or one or more video clips.

[0047] The shooting operations indicated by the control instructions sent by the electronic device to different camera devices can be the same or different. For example, the electronic device can send the same control instruction to all associated camera devices to instruct each camera device to take a set of pictures (or a video clip). For another example, the control instruction sent by the electronic device to the driving recorder is used to instruct to take a set of pictures, while the control instruction sent by the electronic device to the in-vehicle unit is used to instruct the in-vehicle unit to control the relevant vehicle-mounted cameras to take a set of pictures and one or more video clips.

[0048] Step S120: Extract candidate pictures from the media data obtained by the associated camera device.

[0049] In some exemplary embodiments of the present disclosure, the media data obtained by the associated camera device is the media data captured by the associated camera device under the control of the electronic device. For example, the associated camera devices are the in-vehicle unit and the driving recorder. After receiving the control instruction sent by the electronic device for instructing to take a set of pictures, the in-vehicle unit and the driving recorder respectively take a set of pictures and then send the set of pictures to the electronic device. The electronic device uses the two sets of pictures received as the media data obtained by the associated camera device.

[0050] In some other exemplary embodiments of the present disclosure, the media data obtained by the associated camera device includes not only the media data captured by the associated camera device under the control of the electronic device, but also the media data captured by the associated camera device without the control of the electronic device. For example, based on the above example, the driving recorder has been shooting videos before receiving the control instruction sent by the electronic device. After receiving the control instruction sent by the electronic device, it intercepts video clips before and after the reception time of the control instruction (for example, the video clip corresponding to the time interval from the moment 5 seconds before the reception time to the moment 5 seconds after the reception time) and / or takes video frame screenshots of the videos before and after the reception time of the control instruction and sends them to the electronic device. The electronic device takes the set of pictures sent by the in-vehicle unit, the video clips intercepted from the driving recorder video and / or the video frame screenshots as the media data obtained by the associated camera device.

[0051] In still some other exemplary embodiments of the present disclosure, the associated camera device may further include the electronic device itself. Furthermore, the media data obtained by the associated camera device also includes the media data captured by the electronic device.

[0052] As described above, a user can trigger a preset instruction in various ways. In some ways, for example, when the user triggers a preset instruction through a voice instruction, object-related information (hereinafter referred to as object information) may be introduced. If such information can be recognized, this information can be utilized to assist in extracting candidate pictures, which helps to accurately extract candidate pictures containing the target object. In some exemplary embodiments of the present disclosure, the user can trigger a preset instruction through a voice instruction or a quick start operation. At this time, the step of extracting a candidate picture from the media data obtained by the associated imaging device includes steps S121 to S123 as Figure 3 shown, and each step will be described below.

[0053] Step S121: Determine whether the preset instruction contains object information.

[0054] When determining whether the preset instruction contains object information, the input type of the preset instruction can be detected first. Among them, the input type of the preset instruction is related to the way the user triggers the preset instruction. This input type can include voice input and non-voice input. Voice input means that the user triggers the preset instruction by means of a voice instruction, and non-voice input means that the user triggers the preset instruction by means of a quick start operation.

[0055] In some exemplary embodiments, when the user triggers a preset instruction through a quick start operation, no object information is introduced. Therefore, if it is determined after detection that the input type of the preset instruction is non-voice input, then it is not necessary to analyze the relevant content of the preset instruction, and it can be directly determined that the preset instruction does not contain object information.

[0056] In some other exemplary embodiments, the electronic device determines whether the preset instruction contains object information according to the operation form of a specific operation in response to the input type of the preset instruction being the specific operation.

[0057] In this embodiment, the user can trigger a preset instruction through a specific operation. The operation form of the specific operation can be various. For example, inputting an instruction through a virtual or physical button, inputting an instruction through a gesture (including gestures made by touching the screen and gestures made without touching the screen), inputting an instruction through sensors in the electronic device (such as an acceleration sensor, a gyroscope, an ambient light sensor, etc.), etc., some or all of these operation forms. Among them, some operation forms are only used to trigger the preset instruction, and the number of these operation forms can be one or more. Another part of the operation forms can introduce object information when triggering the preset instruction, and the number of these operation forms can also be one or more. When the number of these operation forms is multiple, different operation forms can be used to introduce different object information. Exemplarily, assume that the electronic device supports the user to trigger the preset instruction through three operation forms: shaking the electronic device, clicking a specific shortcut button on the screen, and making a specific gesture in a way of touching or not touching the screen. Among them, the two operation forms of shaking the electronic device and clicking a specific shortcut button on the screen can be set to only trigger the preset instruction without introducing object information, while the operation form of making a specific gesture in a way of touching or not touching the screen can be set to trigger the preset instruction and introduce object information at the same time. Gesture 1 indicates that the introduced object information is a building, and gesture 2 indicates that the introduced object information is a vehicle. Further, if it is detected that the user triggers the preset instruction through the operation form of shaking the electronic device or clicking a specific button, it is determined that the preset instruction does not contain object information. If it is detected that the user triggers the preset instruction through the operation form of making a specific gesture, it is determined that the preset instruction contains object information, and the object information included in the preset instruction is determined through the gesture type of the specific gesture. If the gesture type is gesture 1, it is determined that the object information is a building. If the gesture type is gesture 2, it is determined that the object information is a vehicle.

[0058] If it is determined through detection that the input type of the preset instruction is voice input, it is further determined whether the preset instruction contains object information according to the semantic analysis result of the preset instruction. Among them, the semantic analysis result of the preset instruction can be obtained by performing speech recognition on the voice instruction related to the preset instruction (that is, the voice information sent by the user) to obtain the speech text information, and then performing semantic analysis on the speech text information.

[0059] Among them, the semantic analysis result may include the named entity recognition result. The named entity recognition result may include the entity information extracted from the speech text information by using the named entity recognition technology (Named Entity Recognition, abbreviated as NER) during the semantic analysis process, such as animals, plants, scenery, etc. At this time, it indicates that the preset instruction contains object information, and the object information is the entity information. The named entity recognition result may also be empty, that is, it means that the speech text information does not contain entity information, and it also indicates that the preset instruction does not contain object information.

[0060] Step S122: In response to the preset instruction containing object information, extract multiple pictures that contain the object information and whose image quality meets the first preset condition from the media data obtained by the associated camera device as candidate pictures.

[0061] The media data obtained by the associated camera device may include pictures and video clips. For the pictures in the media data, image recognition technology can be used to identify whether the pictures contain the object corresponding to the object information, so as to determine whether the pictures contain the object information. For example, assuming that the voice instruction issued by the user is "search for the mountain in front", the recognized object information is "mountain", and image recognition technology can be used to identify whether the pictures in the media data contain the target object of the mountain. If it contains, it is determined that the pictures contain the object information "mountain". For the video clips in the media data, it is necessary to first split them into a series of video frames, and then perform the above-mentioned image recognition processing for pictures on each video frame to determine whether each video frame contains object information. The specific process can refer to the above-mentioned processing process for pictures.

[0062] After obtaining the pictures that contain object information (including the pictures in the media data and the video frames), extract multiple pictures whose image quality meets the first preset condition as candidate pictures. The present disclosure does not limit the number of candidate pictures, which can be set according to actual needs. The first preset condition is used to screen pictures with higher quality, and the condition content can be formulated based on one or more aspects such as clarity, contrast, color accuracy, dynamic range, noise, etc. The present disclosure does not impose special restrictions on it. In addition, the condition content can be flexibly adjusted according to the changes in the actual situation.

[0063] Step S123: In response to the preset instruction not containing object information, extract multiple pictures whose image quality meets the first preset condition from the media data as candidate pictures.

[0064] If the preset instruction does not contain object information, split the video clips in the media data into video frames, and then screen out candidate pictures from the video frames and the pictures originally contained in the media data based on the first preset condition.

[0065] Further, after extracting candidate pictures from the media data obtained by the associated imaging device, the candidate pictures can also be stored on the electronic device so that the user can further process or search for the candidate pictures when needed later.

[0066] Step S130: Display the candidate pictures.

[0067] After obtaining the candidate pictures, the electronic device displays the candidate pictures on the screen for the user to select. The present disclosure does not impose any special restrictions on the display method and display style of the candidate pictures.

[0068] Step S140: Determine the target picture in response to the selection operation on the candidate picture.

[0069] The user can determine the target picture, i.e., the picture for searching, from the candidate pictures through the selection operation. The target picture can be one or more. The present disclosure does not impose any special restrictions on the specific implementation form of the selection operation. For example, the user can touch, click, etc. on the screen to select one or more candidate pictures as the target picture. Alternatively, the user can specify one or more candidate pictures as the target picture through voice interaction.

[0070] Step S150: Perform an intelligent search based on the target picture to obtain target information.

[0071] The target information is information related to the target object in the target picture searched based on the target picture.

[0072] In some exemplary embodiments of the present disclosure, the step of performing an intelligent search based on the target picture to obtain target information may include the electronic device sending a picture recognition search request carrying the target picture to a picture recognition search engine. The picture recognition search engine may be a server with picture recognition search capabilities provided remotely. The picture recognition search engine responds to the picture recognition search request, performs a picture recognition search operation using the target picture as the query image, and returns the search results to the electronic device. The electronic device receives the search results returned by the picture recognition search engine and uses the search results as the target information. Alternatively, when the content of the search results is relatively large, specific information is filtered out from the search results based on a preset rule as the target information. The specific information may be one or more types of information that the user may be more interested in.

[0073] In some other exemplary embodiments of the present disclosure, the step of performing an intelligent search based on a target picture to obtain target information may include: The electronic device performs image recognition on the target picture to obtain an image recognition result, and then performs a search based on the image recognition result to obtain the target information. In this embodiment, the electronic device performs image recognition on the target picture based on image recognition technology to obtain an image recognition result. The image recognition result may include object information of the target object included in the target picture. Then, an information search request carrying the object information is sent to an information search engine. The information search engine may be a server with information search capabilities provided at a remote end. The information search engine responds to the information search request, performs a search using the object information as a query text (i.e., query), and returns the search result to the electronic device. The electronic device receives the search result returned by the information search engine and uses the search result as the target information. Alternatively, when there is a large amount of information searched based on the object information, specific information may be extracted from the search result as the target information based on the above preset rules.

[0074] In still some exemplary embodiments of the present disclosure, the electronic device may first detect whether the image quality of the target picture meets a second preset condition. If the image quality of the target picture meets the second preset condition, the operations of sending an image recognition search request to the image recognition search engine or an information search request to the information search engine based on the target picture, and determining the target information based on the search results returned by the image recognition search engine or the information search engine, as introduced in the above embodiments, are performed. If the image quality of the target picture does not meet the second preset condition, a reverse image search is performed based on the target picture. This reverse image search is used to search for pictures that are the same as or similar to the target picture. The electronic device may send a reverse image search request carrying the target picture to the image search engine. The image search engine may be a server located remotely and having the ability of reverse image search (i.e., searching for pictures by pictures). In response to the reverse image search request, the image search engine searches for pictures that are the same as or similar to the target picture and returns the search results to the electronic device. The electronic device may screen out pictures from the searched pictures that have a similarity to the target picture exceeding a preset threshold and whose image quality meets the second preset condition, and use the screened-out pictures as the final target pictures for intelligent search, so as to obtain the target information. If the electronic device cannot screen out pictures from the searched pictures that have a similarity to the target picture exceeding the preset threshold and whose image quality meets the second preset condition, the target picture is still used for intelligent search. The implementation manner of the intelligent search may refer to the above exemplary embodiments and will not be elaborated herein. This embodiment takes into account that when the quality of the target picture is low, for example, the picture is not clear enough, the image recognition result of the target picture may not be accurate enough, which may in turn lead to inaccurate search results. Therefore, when it is determined that the image quality of the target picture is low, a reverse image search is first performed using the target picture. If pictures with higher similarity and better image quality can be obtained, then intelligent search is performed based on these pictures, so as to improve the accuracy of the search results.

[0075] In some exemplary embodiments of the present disclosure, the electronic device itself has the ability of image recognition search. Furthermore, the electronic device can directly perform relevant operations of image recognition search based on the target picture and obtain the target information based on the search results. The relevant operations of image recognition search may refer to the image recognition operation and the information search operation introduced in the above embodiments and will not be elaborated herein.

[0076] After obtaining the target information through intelligent search, the electronic device presents the target information in a preset manner so that the user can understand the target information. The preset manner includes: voice broadcast, and / or, card display. The preset manner may also include text display. The voice broadcast may be a voice broadcast of the text information in the target information. The card display may be a display of relevant pictures and text information of the target object. The text display may be a display of the text information in the target information.

[0077] In the above method for obtaining information during driving, when a user wants to know the relevant information about an object, a preset instruction can be sent to the electronic device. In response to receiving the preset instruction, the electronic device controls the associated imaging device to take a picture. Then, candidate pictures are automatically extracted from the media data obtained through the associated imaging device and displayed. The user can select a target picture from the candidate pictures. The electronic device determines the target picture in response to the selection operation on the candidate picture, and then performs an intelligent search based on the target picture to obtain the target information. In the above solution, the electronic device can control the associated imaging device to take a picture, extract candidate pictures from the media data captured by these imaging devices for the user to select, and automatically perform an intelligent search to obtain the target information after the user completes the selection. The above solution allows the user to obtain the target information only by triggering the preset instruction and performing the selection operation, improving the convenience and timeliness of the user obtaining information during driving, and enabling the user to have a better information acquisition experience during driving.

[0078] Generally, there are multiple imaging devices on the vehicle where the user is located that can be linked with the electronic device. These imaging devices can capture objects outside the vehicle through one or more cameras. The orientations and shooting angles of different cameras can be different, so that one or more cameras can capture the target object as much as possible. Under the control of the electronic device, all cameras take pictures to take pictures as soon as possible to avoid missing wonderful moments. Then, candidate pictures are extracted from the captured media data. The operation of extracting candidate pictures can refer to the above embodiments and will not be elaborated here. Further, in some cases, only some cameras may be able to capture the object (hereinafter referred to as the target object) that the user wants to know the relevant information about. At this time, if all cameras are controlled to take pictures, there will be a situation where the media data captured by some cameras does not contain the target object, which will increase the workload of the subsequent operation of screening candidate pictures.

[0079] In response to this, in some exemplary embodiments, it can be through the steps as Figure 4 shown, that is, step S210 and step S220, to determine the imaging devices that can capture the target object before taking pictures, and only use these imaging devices as the associated imaging devices. The following describes each step.

[0080] S210: Based on the preset instruction, determine the shooting azimuth information.

[0081] The position of the target object can be described using its position relative to the vehicle where the user is located. For example, the target object is in front of the vehicle, left front, right rear, etc. The shooting azimuth information is the azimuth that can capture the target object, and it can be determined in various ways.

[0082] In some exemplary embodiments, when the preset instruction is triggered by a user's voice instruction, semantic analysis can be performed on the voice instruction information associated with the preset instruction to identify whether the voice instruction information contains orientation information. Specifically, named entity recognition technology can be used to identify the orientation information contained in the voice instruction information, such as "front", "left", "right front", etc. When the voice instruction information contains orientation information, the orientation information can be extracted as the shooting orientation information. For example, if the voice instruction information is "Search for the mountain in front", the "front" in it can be extracted as the shooting orientation information. In this embodiment, by identifying the orientation information clearly stated by the user in the voice instruction, the shooting orientation information can be accurately determined.

[0083] In some other exemplary embodiments, the in-vehicle computer can use eye-tracking technology to track the gaze direction of the user (such as the driver) (indicating the position where the relevant person is looking). The in-vehicle computer can record the obtained gaze direction information of the user, for example, "Time 1 - front", "Time 2 - left front", "Time 3 - front". When the electronic device receives the preset instruction, it can request the in-vehicle computer to obtain the corresponding gaze direction based on the trigger time of the preset instruction, and then determine the shooting orientation information based on the gaze direction returned by the in-vehicle computer. In this embodiment, through the gaze direction of the user tracked by the in-vehicle computer using eye-tracking technology, it can be used to assist in determining the shooting orientation information when the voice instruction information of the user does not contain orientation information or the user triggers the preset instruction through a quick start operation.

[0084] In still some other exemplary embodiments, when the electronic device is also involved in shooting, when the electronic device has displayed a live view window, the pose information of the electronic device can be obtained, and then the shooting orientation information can be determined based on the pose information of the electronic device. Among them, the orientation of the mobile phone can be judged based on sensor data such as the direction and rotation angle of the electronic device, and then the shooting orientation information can be determined. The above sensor data can be obtained through sensors such as an accelerometer and a gyroscope sensor equipped on the electronic device. In this embodiment, when the electronic device is involved in shooting, the orientation of the electronic device during shooting is analyzed, and then when the voice instruction information of the user does not contain orientation information or the user triggers the preset instruction through a quick start operation, it can be used to assist in determining the shooting orientation information based on this orientation.

[0085] S220: Based on the shooting orientation information, determine the associated imaging device from all the imaging devices connected to the electronic device.

[0086] An electronic device can maintain the shooting azimuths of the cameras that can be used to shoot external objects controlled by each imaging device, and then determine the cameras that can shoot the azimuth corresponding to the shooting azimuth information based on the shooting azimuth information, as well as the imaging devices associated with the cameras. Finally, the determined imaging devices are taken as the associated imaging devices. Moreover, when the electronic device controls the associated imaging devices to shoot, for any associated imaging device, if the imaging device is associated with multiple cameras and only some of the cameras of the imaging device can shoot the azimuth corresponding to the shooting azimuth information, the electronic device needs to specify the unique identifier of the camera that needs to be shot in the control instruction so that the imaging device only controls the cameras that can shoot the relevant azimuth to shoot.

[0087] To facilitate the understanding of the solutions and their beneficial effects of the exemplary embodiments of the present disclosure, the information acquisition method during driving of the present disclosure will be described in detail below with an exemplary embodiment.

[0088] In an exemplary embodiment, a mobile phone serves as the above-mentioned electronic device. The imaging devices connected to the mobile phone wirelessly include Dashcam A, Dashcam B, and a car head unit. The car head unit can control in-vehicle camera C and in-vehicle camera D to shoot external objects. The two dashcams have been shooting, while in-vehicle camera C and in-vehicle camera D were not in the shooting state before being triggered by a preset instruction. The shooting azimuths of the two dashcams and in-vehicle camera C and in-vehicle camera D are shown in Table 1 below, that is:

[0089] Table 1:

[0090]

[0091]

[0092] This embodiment includes the following steps:

[0093] 1. The user issues a voice command "Identify the animal on the left". The mobile phone receives and analyzes the intent of the voice command, determines that the user has the intent of image recognition search, and triggers a preset command;

[0094] 2. In response to the preset command, the mobile phone performs semantic analysis on "Identify the mountain in front", obtains "in front" as the azimuth information, and obtains "mountain" as the object information; the mobile phone determines the shooting azimuth information as "ahead", and based on this shooting azimuth information, determines that the associated imaging devices are Dashcam A and the car head unit;

[0095] 3. The mobile phone sends control instructions for instructing to capture a set of pictures to the driving recorder A and the in-vehicle computer. Among them, the control instruction sent to the driving recorder A includes the trigger time of the preset instruction, and the control instruction sent to the in-vehicle computer includes the unique identifier of the in-vehicle camera D and the shooting azimuth information. In response to the control instruction, the driving recorder A captures a set of pictures at a preset frequency, and sends the captured set of pictures and a section of video before and after the trigger time of the intercepted preset instruction to the mobile phone. In response to the control instruction, the in-vehicle computer controls the in-vehicle camera D to rotate to a position where it can capture the front, and controls the in-vehicle camera D to capture a set of pictures at a preset frequency, and then sends the set of pictures captured by the in-vehicle camera D to the mobile phone.

[0096] 4. The mobile phone saves the pictures and video clips (hereinafter referred to as media data) sent by the driving recorder A and the in-vehicle computer locally.

[0097] 5. The mobile phone splits the video clips in the media data into multiple video frames, performs image recognition on each video frame and each picture in the media data, screens out the pictures including "mountain", and then extracts two pictures whose image quality meets the first preset condition as candidate pictures. For example Figure 5 as shown in Figure a and Figure b.

[0098] 6. The mobile phone displays the candidate pictures on the screen.

[0099] 7. The user clicks on the screen to select Figure b.

[0100] 8. The mobile phone sends a picture recognition search request containing Figure b to the picture recognition search engine.

[0101] 9. The mobile phone receives the search results returned by the picture recognition search engine, extracts the pictures and text descriptions of "mountain" from the search results as target information, and displays them in the form of cards. The display effect is as Figure 6 shown.

[0102] It should be understood that although Figures 2 - 4 the steps in the flowchart of Figures 2 - 4 are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps has no strict order restriction, and these steps can be executed in other orders. Moreover,

[0103] at least a part of the steps inFigure 7 As shown, an information acquisition device during driving is provided, including: a shooting control module 110, a candidate picture extraction module 120, a picture display module 130, a picture determination module 140, and an information search module 150, where: The shooting control module 110 is configured to control an associated camera device to take pictures in response to receiving a preset instruction; The candidate picture extraction module 120 is configured to extract candidate pictures from the media data obtained by the associated camera device; The picture display module 130 is configured to display the candidate pictures; The picture determination module 140 is configured to determine a target picture in response to a selection operation on the candidate pictures; The information search module 150 is configured to perform an intelligent search based on the target picture to obtain target information.

[0104] In some exemplary embodiments of the present disclosure, the device further includes an orientation determination module (not shown in the figure) and a camera device determination module (not shown in the figure). The orientation determination module is configured to determine shooting orientation information based on a preset instruction; The camera device determination module is configured to determine an associated camera device from all the camera devices connected to the electronic device based on the shooting orientation information.

[0105] In some exemplary embodiments of the present disclosure, the candidate picture extraction module 120 determines whether the preset instruction contains object information; In response to the preset instruction containing object information, multiple pictures that contain the object information and whose image quality meets the first preset condition are extracted from the media data obtained by the associated camera device as candidate pictures; In response to the preset instruction not containing object information, multiple pictures whose image quality meets the first preset condition are extracted from the media data as candidate pictures.

[0106] In some exemplary embodiments of the present disclosure, the candidate picture extraction module 120 detects the input type of the preset instruction; In response to the input type of the preset instruction being voice input, it is determined whether the preset instruction contains object information according to the semantic analysis result of the preset instruction; In response to the input type of the preset instruction being a specific operation, it is determined whether the preset instruction contains object information according to the operation form of the specific operation.

[0107] In some exemplary embodiments of the present disclosure, the device further includes a picture storage module (not shown in the figure), and the picture storage module is configured to store the candidate pictures on the electronic device.

[0108] In some exemplary embodiments of the present disclosure, the information search module 150 performs image recognition on the target picture to obtain an image recognition result; A search is performed based on the image recognition result to obtain target information.

[0109] In certain exemplary embodiments of the present disclosure, the device further includes an information presentation module (not shown in the figure). The information presentation module presents target information in a preset manner, where the preset manner includes: voice broadcast, and / or, card display.

[0110] For the specific limitations of the information acquisition device during driving, reference can be made to the limitations of the information acquisition method during driving in the above text, which will not be elaborated here. Each module in the above information acquisition device during driving can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0111] In one embodiment, an electronic device is provided, and its internal structure diagram can be as Figure 8 shown. The electronic device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with external electronic devices through a network connection. When the computer program is executed by the processor, it implements an information acquisition method during driving. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball, or a touchpad provided on the housing of the electronic device, or an external keyboard, touchpad, or mouse, etc.

[0112] Those skilled in the art can understand that Figure 8 the structure shown in is only a block diagram of some structures related to the solution of the present disclosure, and does not constitute a limitation on the electronic device to which the solution of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0113] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: in response to receiving a preset instruction, controlling an associated imaging device to take a picture; extracting candidate pictures from the media data obtained through the associated imaging device; displaying the candidate pictures; in response to a selection operation on the candidate pictures, determining a target picture; performing an intelligent search based on the target picture to obtain target information.

[0114] In some exemplary embodiments of the present disclosure, before the processor executes a computer program to implement the above steps of controlling the associated imaging device to take pictures, the following steps are further implemented, including: determining shooting orientation information based on a preset instruction; and determining an associated imaging device from all the imaging devices connected to the electronic device based on the shooting orientation information.

[0115] In some exemplary embodiments of the present disclosure, when the processor executes a computer program to implement the above step of extracting candidate pictures from the media data obtained through the associated imaging device, the following steps are specifically implemented: determining whether the preset instruction contains object information; in response to the preset instruction containing object information, extracting multiple pictures that contain the object information and whose image quality meets a first preset condition from the media data obtained through the associated imaging device as candidate pictures; in response to the preset instruction not containing object information, extracting multiple pictures whose image quality meets the first preset condition from the media data as candidate pictures.

[0116] In some exemplary embodiments of the present disclosure, when the processor executes a computer program to implement the above step of determining whether the preset instruction contains object information, the following steps are specifically implemented: detecting the input type of the preset instruction; in response to the input type of the preset instruction being voice input, judging whether the preset instruction contains object information according to the semantic analysis result of the preset instruction; in response to the input type of the preset instruction being a specific operation, determining whether the preset instruction contains object information according to the operation form of the specific operation.

[0117] In some exemplary embodiments of the present disclosure, after the processor executes a computer program to implement the above step of extracting candidate pictures from the media data obtained through the associated imaging device, the following steps are further implemented: storing the candidate pictures on the electronic device.

[0118] In some exemplary embodiments of the present disclosure, when the processor executes a computer program to implement the above step of performing an intelligent search based on the target picture to obtain target information, the following steps are specifically implemented: performing image recognition on the target picture to obtain an image recognition result; and performing a search based on the image recognition result to obtain the target information.

[0119] In some exemplary embodiments of the present disclosure, after the processor executes a computer program to implement the above step of performing an intelligent search based on the target picture to obtain target information, the following steps are further implemented: presenting the target information in a preset manner, where the preset manner includes: voice broadcast, and / or, card-style display.

[0120] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: in response to receiving a preset instruction, controlling an associated imaging device to take a picture; extracting candidate pictures from the media data obtained by the associated imaging device; displaying the candidate pictures; in response to a selection operation on the candidate pictures, determining a target picture; and performing an intelligent search based on the target picture to obtain target information.

[0121] In some exemplary embodiments of the present disclosure, before the computer program is executed by the processor to implement the above step of controlling the associated imaging device to take a picture, the following steps are further implemented, including: based on the preset instruction, determining shooting azimuth information; and based on the shooting azimuth information, determining the associated imaging device from all the imaging devices connected to the electronic device.

[0122] In some exemplary embodiments of the present disclosure, when the computer program is executed by the processor to implement the above step of extracting candidate pictures from the media data obtained by the associated imaging device, the following steps are specifically implemented: determining whether the preset instruction contains object information; in response to the preset instruction containing object information, extracting multiple pictures that contain the object information and whose image quality meets a first preset condition from the media data obtained by the associated imaging device as candidate pictures; and in response to the preset instruction not containing object information, extracting multiple pictures whose image quality meets the first preset condition from the media data as candidate pictures.

[0123] In some exemplary embodiments of the present disclosure, when the computer program is executed by the processor to implement the above step of determining whether the preset instruction contains object information, the following steps are specifically implemented: detecting the input type of the preset instruction; in response to the input type of the preset instruction being voice input, judging whether the preset instruction contains object information according to the semantic analysis result of the preset instruction; and in response to the input type of the preset instruction being a specific operation, determining whether the preset instruction contains object information according to the operation form of the specific operation.

[0124] In some exemplary embodiments of the present disclosure, after the computer program is executed by the processor to implement the above step of extracting candidate pictures from the media data obtained by the associated imaging device, the following step is further implemented: storing the candidate pictures on the electronic device.

[0125] In some exemplary embodiments of the present disclosure, when the computer program is executed by the processor to implement the above step of performing an intelligent search based on the target picture to obtain target information, the following steps are specifically implemented: performing image recognition on the target picture to obtain an image recognition result; and performing a search based on the image recognition result to obtain target information.

[0126] In certain exemplary embodiments of the present disclosure, after the computer program is executed by a processor to implement the above steps of performing intelligent search based on a target picture to obtain target information, the following steps are further implemented: presenting the target information in a preset manner, where the preset manner includes: voice broadcast, and / or, card-style display.

[0127] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided by the present disclosure can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink), DRAM (SLDRAM), memory bus (Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0128] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not conflict, they should be considered as the scope described in this specification.

[0129] The above embodiments only represent several implementation manners of the present disclosure. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present disclosure, several modifications and improvements can be made, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure patent should be subject to the appended claims.

Claims

1. A method for obtaining information during driving, which is applied to an electronic device, characterized in that, The method includes: Controlling an associated imaging device to capture images in response to receiving a preset instruction; Extracting candidate pictures from the media data obtained by the associated imaging device; Displaying the candidate pictures; Determining a target picture in response to a selection operation on the candidate picture; Performing an intelligent search based on the target picture to obtain target information.

2. The method according to claim 1, characterized in that, Before controlling the associated imaging device to capture images, the method further includes: Determining shooting azimuth information based on the preset instruction; Determining the associated imaging device from all the imaging devices connected to the electronic device based on the shooting azimuth information.

3. The method according to claim 1, characterized in that, Extracting candidate pictures from the media data obtained by the associated imaging device includes: Determining whether the preset instruction contains object information; In response to the preset instruction containing object information, extracting multiple pictures that contain the object information and whose image quality meets a first preset condition from the media data obtained by the associated imaging device as candidate pictures; In response to the preset instruction not containing object information, extracting multiple pictures whose image quality meets the first preset condition from the media data as candidate pictures.

4. The method according to claim 3, characterized in that, Determining whether the preset instruction contains object information includes: Detecting the input type of the preset instruction; In response to the input type of the preset instruction being voice input, determining whether the preset instruction contains object information according to the semantic analysis result of the preset instruction; In response to the input type of the preset instruction being a specific operation, determining whether the preset instruction contains object information according to the operation form of the specific operation.

5. The method according to claim 1, wherein After extracting candidate pictures from the media data obtained by the associated imaging device, it further includes: Storing the candidate pictures on the electronic device.

6. The method according to claim 1, characterized in that, Performing an intelligent search based on the target picture to obtain target information includes: Performing image recognition on the target picture to obtain an image recognition result; Performing a search based on the image recognition result to obtain target information.

7. The method according to claim 1, wherein After obtaining the target information, the method further includes: Presenting the target information in a preset manner, where the preset manner includes: voice broadcast, and / or, card-style display.

8. An information acquisition device during driving, characterized in that, The device includes: A shooting control module, configured to control an associated imaging device to capture images in response to receiving a preset instruction; A candidate picture extraction module, configured to extract candidate pictures from the media data obtained by the associated imaging device; A picture display module, configured to display the candidate pictures; A picture determination module, configured to determine a target picture in response to a selection operation on the candidate picture; An information search module, configured to perform an intelligent search based on the target picture to obtain target information.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 7.

Images