Data analysis method, electronic device, wearable device system, and storage medium

Abstract

The embodiment of the application provides a data analysis method, an electronic device, a wearable device system and a storage medium, which are applied to the electronic device and belong to the technical field of artificial intelligence. The method comprises the following steps: in response to problem information of a target object, obtaining original environment data of the target object. Then, complexity analysis is performed according to the problem information and the original environment data, and complexity level information is obtained. Subsequently, the network signal strength of a target network is obtained, and the electronic device is connected to a cloud server through the target network. According to the complexity level information and the network signal strength, a target model is selected from candidate models, and the candidate models comprise at least one of a local model deployed on the electronic device and a cloud model deployed on the cloud server. Then, the original environment data and the problem information are analyzed according to the target model to obtain target answer data, and the target answer data is pushed to the target object. The embodiment of the application can improve the response speed of the electronic device.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a data analysis method, an electronic device, a wearable device system and a storage medium. BACKGROUND

[0002] In the related art, surrounding environment data (for example, video and voice) is collected by different electronic devices (for example, wearable devices) and sent to a cloud service center for processing, and then the processing result is fed back to the user to help the user safely complete the activity. However, as the amount of data processed increases, the load of cloud processing also rises, thereby increasing the data transmission delay and ultimately leading to a decrease in the response speed of the electronic device. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a data analysis method, an electronic device, a wearable device system and a storage medium, which aims to improve the response speed of the electronic device.

[0004] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application provides a data analysis method applied to an electronic device, and the method comprises:

[0005] In response to question information of a target object, original environment data of the target object is obtained;

[0006] Complexity analysis is performed according to the question information and the original environment data, and complexity level information is obtained;

[0007] Network signal strength of a target network is obtained; wherein the electronic device is connected to a cloud server through the target network;

[0008] A target model is selected from candidate models according to the complexity level information and the network signal strength; wherein the candidate models include at least one of a local model deployed on the electronic device and a cloud model deployed on the cloud server;

[0009] Question answering is performed on the original environment data and the question information according to the target model, and target answer data is obtained;

[0010] The target answer data is pushed to the target object.

[0011] In some embodiments, the target model is selected from the candidate models according to the complexity level information and the network signal strength, comprising:

[0012] If the network signal strength is greater than or equal to a preset signal strength, the local model or the cloud model is taken as the target model according to the complexity level information;

[0013] if the network signal strength is less than the preset signal strength, taking the local model as the target model.

[0014] In some embodiments, the complexity level information includes a first complexity level and a second complexity level, and the first complexity level is greater than the second complexity level; and the taking the local model or the cloud model as the target model according to the complexity level information includes:

[0015] if the complexity level information is the first complexity level, taking the cloud model as the target model;

[0016] if the complexity level information is the second complexity level, taking the local model as the target model.

[0017] In some embodiments, the complexity level information is obtained by performing complexity analysis according to the problem information and the original environment data, and the complexity analysis includes:

[0018] generating a difficulty coefficient according to the problem information to obtain the difficulty coefficient;

[0019] performing data computation amount evaluation on the original environment data to obtain target computation amount data;

[0020] determining the complexity level information according to at least one of the difficulty coefficient and the target computation amount data.

[0021] In some embodiments, the original environment data includes audio data and environment image data of the target object; and before the obtaining target answer data by performing problem solving on the original environment data and the problem information according to the target model, the method further includes:

[0022] in response to the target model being the cloud model, performing scene detection on the environment image data to obtain candidate image data;

[0023] determining a target region of a target type in the candidate image data, and performing mask processing on the target region to obtain target image data; wherein the target type represents a type that is prohibited from being disclosed;

[0024] performing text conversion processing on the audio data to obtain target text data;

[0025] sending the target image data and the target text data to the cloud server.

[0026] In some embodiments, before the pushing the target answer data to the target object, the method further includes:

[0027] perform data quality evaluation on the original environment data to obtain a data quality score;

[0028] perform intention clarity degree evaluation on the question information to obtain an intention clarity degree score;

[0029] perform feature vectorization on the question information to obtain a question vector;

[0030] perform feature vectorization on the preset local knowledge base to obtain a knowledge base vector;

[0031] perform similarity calculation according to the question vector and the knowledge base vector to obtain field similarity data;

[0032] perform integration calculation according to the data quality score, the intention clarity degree score and the field similarity data to obtain confidence data of the target answer data;

[0033] determine to update the target answer data according to the confidence data and a preset confidence threshold.

[0034] In some embodiments, the determining to update the target answer data according to the confidence data and a preset confidence threshold comprises:

[0035] if the confidence data is less than the preset confidence threshold, selecting low confidence factor data from the data quality score, the intention clarity degree score and the field similarity data;

[0036] generating prompt information according to the low confidence factor data;

[0037] updating the target answer data according to the prompt information.

[0038] To achieve the above object, a second aspect of embodiments of the present application proposes an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0039] To achieve the above object, a third aspect of embodiments of the present application proposes a wearable device system, which comprises an electronic device as described in the second aspect, an image acquisition module, an audio acquisition module, an audio playback module, a data display module and a data transmission module, the electronic device is connected with the image acquisition module, the audio acquisition module, the audio playback module, the display module and the data transmission module respectively.

[0040] The image acquisition module is configured to acquire image data; the audio acquisition module is configured to acquire audio data; the audio playing module is configured to play audio data; the display module is configured to display image data or text data; and the data transmission module is configured to send the image data and the audio data to a cloud server, and is further configured to receive data sent by the cloud server.

[0041] To achieve the above object, a fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method in the first aspect.

[0042] The data analysis method, the electronic device, the wearable device system and the storage medium provided by the present application, by acquiring the original environment data of the target object when responding to the question information of the target object, then evaluating the complexity of the question and the current network signal strength, and then selecting the most suitable model, that is, the target model from multiple candidate models for data processing. Finally, the target answer data analyzed by the target model is pushed to the target object. It is clear that the powerful computing resources of the cloud are used to process complex tasks, and the opportunity for local processing of lightweight models is selected. Not only effectively reduces the data transmission amount, relieves the processing load of the cloud, thereby reducing the response delay, improving the real-time response speed of the device, and finally ensuring to provide more efficient assistance support for the target object. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a flowchart of the data analysis method provided by the embodiment of the present application;

[0044] Figure 2 is a flowchart of step S102 in Figure 1

[0045] Figure 3 is a flowchart of step S104 in Figure 1

[0046] Figure 4 is another flowchart of the data analysis method provided by the embodiment of the present application;

[0047] Figure 5 is another flowchart of the data analysis method provided by the embodiment of the present application;

[0048] Figure 6 is a flowchart of step S507 in Figure 5

[0049] Figure 7 is a hardware structure schematic diagram of the electronic device provided by the embodiment of the present application;

[0050] ​​​Figure 8 is a schematic diagram of module connection of a wearable device system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0052] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0054] First, the terms involved in the present application are analyzed:

[0055] Artificial intelligence (AI): is a new technical science that studies, develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, and artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0056] Natural language processing (NLP): NLP uses computers to process, understand, and apply human languages (such as Chinese, English, etc.). NLP is a branch of artificial intelligence and is an interdisciplinary subject of computer science and linguistics, and is also commonly referred to as computational linguistics. Natural language processing includes syntax analysis, semantic analysis, and discourse understanding. Natural language processing is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, etc. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and language computing-related linguistic research related to language processing.

[0057] Information extraction: Information extraction is a text processing technology that extracts specified types of entities, relationships, events, and other factual information from natural language text and forms structured data output. Information extraction is a technology for extracting specific information from text data. Text data is composed of specific units such as sentences, paragraphs, and chapters, and text information is composed of small specific units such as words, phrases, sentences, paragraphs, or combinations of these specific units. Extracting noun phrases, names, and places from text data is text information extraction, and of course, the information extracted by text information extraction technology can be various types of information.

[0058] Image captioning: Image captioning generates natural language descriptions for images and uses the generated descriptions to help applications understand the semantics expressed in the image visual scene. For example, image captioning can convert image retrieval into text retrieval, which can be used to classify images and improve image retrieval results. People can usually describe the details of the image visual scene by just glancing at it, but automatically adding descriptions to images is a comprehensive and difficult computer vision task that requires converting the complex information contained in the image into a natural language description. Compared with ordinary computer vision tasks, image captioning not only needs to identify objects from images, but also needs to associate the identified objects with natural semantics and describe them in natural language. Therefore, image captioning needs to extract deep features of images, associate them with semantic features, and convert them for generating descriptions.

[0059] For the visually impaired, hearing impaired and some elderly people, these groups of people with weaker sensory functions, they can carry different electronic devices (such as wearable devices) to collect the surrounding environment data (such as video and voice) and send them to the cloud service center for processing, and then feed back the processing results to the user, to help the user safely complete the activity. However, as the amount of data processed increases, the load of cloud processing also rises, resulting in increased data transmission delay and ultimately leading to a decrease in device response speed.

[0060] In addition, since the analysis data needs to upload the collected image data and audio data to the cloud server, there is a risk of data leakage during transmission to the cloud, thereby affecting the user privacy and security issues.

[0061] Therefore, the embodiments of the present application provide a data analysis method, an electronic device, a wearable device system and a storage medium, which aims to improve the response speed of the electronic device.

[0062] The data analysis method, the electronic device, the wearable device system and the storage medium provided by the embodiments of the present application are specifically described by the following embodiments, and first, the data analysis method in the embodiments of the present application is described.

[0063] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (AI) is to use digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0064] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0065] The data analysis method provided by the embodiments of the present application relates to the technical field of artificial intelligence. The data analysis method provided by the embodiments of the present application can be applied to a terminal, can also be applied to a server end, and can also be software running in the terminal or the server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as a stand-alone physical server, can also be configured as a server cluster or a distributed system formed by multiple physical servers, and can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; and the software can be an application that implements the data analysis method, but is not limited to the above forms.

[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0067] It should be noted that in each specific embodiment of the present application, when it is necessary to process relevant data related to the identity or characteristics of the user according to user information, user behavior data, user history data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to normally operate will be obtained.

[0068] Figure 1 is an optional flowchart of the data analysis method provided by the embodiments of the present application, Figure 1 The method in the method can be applied to an electronic device, and can include but is not limited to steps S101 to S106.

[0069] Step S101, in response to the question information of the target object, obtaining the original environment data of the target object.

[0070] Step S102, according to the question information and the original environment data, performing complexity analysis to obtain complexity level information.

[0071] Step S103, obtaining the network signal strength of the target network.

[0072] Step S104, according to the complexity level information and the network signal strength, selecting the target model from the candidate models.

[0073] Step S105, according to the target model, performing question answering on the original environment data and the question information to obtain target answer data.

[0074] Step S106, pushing the target answer data to the target object.

[0075] The steps S101 to S106 shown in the embodiments of the present application, by acquiring the original environment data of the target object in response to the question information of the target object, then evaluating the complexity of the question and the current network signal strength, and then selecting the most suitable model, that is, the target model from multiple candidate models for data processing. Finally, the target answer data analyzed by the target model is pushed to the target object. It is clear that the powerful computing resources of the cloud are used to process complex tasks, and the opportunity of selecting a lightweight model for local processing. Not only effectively reduces the data transmission amount, relieves the processing load of the cloud, thereby reducing the response delay, improving the real-time response speed of the device, and finally ensuring to provide more efficient assistance support for the target object.

[0076] In step S101 of some embodiments, the target object is a person carrying an electronic device. The question information can be a query or requirement input by the target object to the electronic device, and the form of the question information can be text data, voice data and image data. The input method of the question information can be through keyboard or key input, or through voice acquisition module input, not limited to this.

[0077] It should be noted that the original environment data includes at least one of the following data: audio data around the target object, image data collected from the surrounding environment, environmental temperature data, and distance data between the target object and the surrounding obstacles, without being limited thereto. The specific composition of the original environment data obtained each time needs to be determined according to the requirements of the problem information. Exemplarily, the problem information of the target object is "who is the person in front of me", and it can be determined that the original environment data is the image data collected from the surrounding environment. Subsequently, the problem information changes to "now I want to go to the nearest library, can you navigate for me", and the original environment data can include real-time position data, image data collected from the surrounding environment, audio data around the target object, and distance data between the target object and the surrounding obstacles.

[0078] In step S102 of some embodiments, the complexity level information is a quantitative index for measuring the difficulty of the current data processing task, so as to determine the model for analyzing the data this time subsequently. In some embodiments, a plurality of complexity level information can be set.

[0079] Specifically, referring to Figure 2 , step S102 can include but is not limited to steps S201 to S203:

[0080] Step S201, difficulty coefficient generation is performed according to the problem information, and the difficulty coefficient is obtained.

[0081] Step S202, data calculation amount evaluation is performed on the original environment data, and target calculation amount data is obtained.

[0082] Step S203, complexity level information is determined according to at least one of the difficulty coefficient and the target calculation amount data.

[0083] In step S201 of some embodiments, the problem information can be analyzed by natural language processing technology to analyze the task type to which the problem information belongs. Exemplarily, if the problem information indicates that only face recognition and surrounding environment image recognition are required at present, it can be determined that the task type is recognition type. If the problem information indicates that the telephone number or address of the contact person needs to be queried, it can be determined that the task type is local query type. The problem information can be analyzed to determine the number of modalities of the original environment data, and if the number of modalities of the required data is greater than a preset threshold (which can be 3), it can be determined that the task type is multi-modality data processing type. The above-mentioned embodiment in which the real-time navigation service is provided for the target object belongs to the multi-modality data processing type.

[0084] A difficulty coefficient mapping table can be stored locally in the electronic device, which stores the mapping relationship between the task type and the difficulty coefficient. After the task type is determined, the difficulty coefficient mapping table is queried according to the task type to obtain the difficulty coefficient. It can be understood that the difficulty coefficients of the local query class, the recognition class and the multi-modal data processing class should be gradually increased.

[0085] In step S202 of some embodiments, in addition to the task difficulty implied by the question information affecting the complexity level information, the amount of calculation of the original environment data is also a key influencing factor. The target calculation data includes the data calculation amount of each modality in the original environment data. Exemplarily, for a 1080p (resolution of 1920x1080) 30fps video, assuming that a simple processing (such as color adjustment) needs to be performed on each pixel, the target calculation amount, that is, the pixel processing amount per second is 1920x1080x30=62,208,000 times. For audio data, the target calculation amount can be calculated by the following formula: target calculation amount=sampling rate x bit depth x channel number x time length x calculation amount per sample. Assuming that a 16kHz sampling rate, 16bit bit depth, single-channel voice, its time length is 10 seconds, and it is assumed that 10 floating point operations (Floating Point Operations Per Second, FLOPs) are required per sample per second, the target calculation amount is 16000Hzx16bitx1x10sx10=25,600,000FLOPs.

[0086] In step S203 of some embodiments, a difficulty coefficient threshold and a calculation amount threshold corresponding to each modality data are pre-set. In this embodiment, the complexity level information includes a first complexity level and a second complexity level, and the first complexity level is greater than the second complexity level. In other words, if the complexity level information is the first complexity level, it indicates that it is difficult to analyze the current question information and the original environment data. If the complexity level information is the second complexity level, it indicates that it is relatively easy to analyze the current question information and the original environment data, and the answer result obtained is also more reliable.

[0087] Specifically, if at least one of the following scenarios is met, it can be determined that the complexity level information is the first complexity level: the difficulty coefficient is greater than the preset difficulty coefficient threshold; the target calculation amount data of at least one modality is greater than the calculation amount threshold corresponding to the modality. If the difficulty coefficient is less than or equal to the difficulty coefficient threshold, and all target calculation amount data is less than or equal to the calculation amount threshold corresponding to the modality, it is determined that the complexity level information is the second complexity level.

[0088] It needs to be explained that in other embodiments, when the target object repeatedly inputs the same or very similar question information in a short time, the complexity level information can also be directly determined as the first complexity level.

[0089] The steps S201 to S203 shown in the embodiments of the present application can realize more accurate and reasonable measurement of the complexity of the processing task by generating the difficulty coefficient of the question information and evaluating the target calculation amount data of the original environment data, and determining the complexity level information according to at least one of the two. This is helpful for reasonably selecting a processing model and a strategy according to the actual complexity, and improving the processing efficiency and the accuracy of the results.

[0090] In step S103 of some embodiments, the electronic device is connected to the cloud server through a target network. The target network can be Wi-Fi, 5G, 4G, 3G, 2G, and wired network. At the operating system level of the electronic device, the network state detection interface provided by the device system can be used to obtain the signal strength parameter of the currently connected network in real time, which is generally in dBm (decibel milliwatt), and the larger the value, the stronger the signal. Exemplarily,

[0091] In step S104 of some embodiments, the candidate model includes at least one of a local model deployed on the electronic device and a cloud model deployed on the cloud server. It needs to be explained that the local model in the present embodiment is obtained by pre-training a small model using a large model through knowledge distillation, and then performing model parameter pruning and quantization operation on the small model optimized by training. Therefore, the local model in the present embodiment maintains good performance while the storage resources occupied can be significantly reduced. The cloud model refers to a machine learning or deep learning model trained, stored and deployed in a cloud computing environment. Unlike the model running on the local device, the cloud model relies on the computing resources of the cloud server and is accessed and called through the network.

[0092] Specifically, referring to Figure 3 , step S104 can include but is not limited to steps S301 to S302:

[0093] In step S301, if the network signal strength is greater than or equal to the preset signal strength, the local model or the cloud model is selected as the target model according to the complexity level information.

[0094] In step S302, if the network signal strength is less than the preset signal strength, the local model is selected as the target model.

[0095] In step S301 of some embodiments, if the complexity level information is the first complexity level, the cloud model is selected as the target model. If the complexity level information is the second complexity level, the local model is selected as the target model.

[0096] In step S302 of some embodiments, when the network signal strength is weak, the local model is directly selected for analysis and processing regardless of the complexity level, thereby avoiding the inability to connect to the cloud server, resulting in no response of the electronic device and no output of the analysis result.

[0097] The steps S301 to S302 shown in the embodiments of the present application rationally select the target model by comprehensively considering the network signal strength and the complexity level information. When the network signal is good, the computing resources of the cloud or the local are flexibly allocated according to the task complexity, so that simple tasks can be processed locally to reduce unnecessary data transmission overhead, and complex tasks can be efficiently completed with the powerful computing power and rich data support of the cloud, thereby improving the overall data processing efficiency and quality and ensuring that the user can obtain accurate answer data as soon as possible. On the other hand, when the network signal strength is insufficient, the local model is directly selected for data analysis, thereby avoiding problems such as data transmission delay and interruption caused by poor network, ensuring the stability and usability of the embodiment under different network environments, and enabling the target object to obtain the corresponding environmental analysis result by relying on the device, regardless of the network coverage, to better understand the surrounding environment and safely complete the activity.

[0098] Before step S105 of some embodiments, since the original environmental data can include audio data and environmental image data of the target object, when the cloud model is involved for data processing, there is a risk of data leakage in the process of transmitting the original environmental data to the cloud server. Therefore, before sending the original environmental data to the cloud server, the original environmental data needs to be desensitized. Specifically, please refer to Figure 4 The data analysis method provided by the embodiments of the present application can also include but is not limited to steps S401 to S404:

[0099] Step S401, in response to the target model being a cloud model, performing scene detection on the environmental image data to obtain candidate image data.

[0100] Step S402, determining a target region of a target type in the candidate image data, and performing mask processing on the target region to obtain target image data.

[0101] Step S403, performing text conversion processing on the audio data to obtain target text data.

[0102] Step S404, sending the target image data and the target text data to the cloud server.

[0103] In step S401 of some embodiments, scene detection is performed on the environment image data by a scene detection algorithm in the local model, and a scene label is obtained for each environment image, such as library, park, living room, kitchen, bedroom, and bathroom, wherein the bedroom and bathroom belong to scenes involving privacy. The image data involving privacy scenes are screened out from the candidate image data, i.e., the environment image data obtained.

[0104] In step S402 of some embodiments, the target type represents a type of object that should not be exposed, such as a safe, a window of a residence (to avoid revealing the address), and a toilet. The candidate image data is identified by a visual detection algorithm, and the region of the target type object in the candidate image data is framed according to the coordinate data of the identification frame, i.e., the target region.

[0105] In some embodiments, the mask processing can set the RGB values of all pixels in the target region to 0 or 255, so that the target region is white or black. Alternatively, the target region can be replaced with a mosaic pattern. The image after mask processing is the target image data.

[0106] In step S403 of some embodiments, an audio feature extraction algorithm is used to extract acoustic features that can represent the speech content from the preprocessed audio data. Common acoustic features include Mel-Frequency Cepstral Coefficients (MFCC), Linear Predictive Cepstral Coefficients (LPCC), Line Spectral Pairs (LSP), and Linear Predictive Coding (LPC). The extracted audio feature vectors are input into a selected local speech recognition model in chronological order. The speech recognition model decodes the audio features and gradually predicts the corresponding text content. Finally, the entire audio corresponding text information is output as the target text data, so that the voice features of the target object are not leaked during transmission.

[0107] In step S404 of some embodiments, first, the target image data and the target text data are packaged and integrated according to a predetermined format, for example, the JSON (Java Script Object Notation) format can be used, the image data is embedded in the JSON object in the form of Base64 encoding, and the target text data is added to the JSON object as another field to form a complete data structure. Then, a suitable network communication protocol is selected for data transmission, such as the HTTP protocol and the HTTPS protocol. Then, the packaged data is sent to the designated receiving address of the cloud server according to the selected protocol. During the sending process, a corresponding timeout mechanism and error callback function are set, for example, if no response confirmation information is received from the cloud server within a specified time, it is determined that the sending is timed out, and the corresponding error handling program is triggered, which may attempt to resend the data. If an error code is returned by the server, corresponding remedial measures can be taken according to different error codes, such as resending after adjusting the data format, prompting the user that the network is abnormal, etc., to ensure that the data can be as accurately and completely as possible to be sent to the cloud server for subsequent processing.

[0108] The steps S401 to S404 shown in the embodiments of the present application first acquire candidate image data by performing scene detection on the environmental image data, which can filter out image scene content that may involve privacy leakage from a large number of original environmental images, improving the pertinence and efficiency of subsequent data processing. Then, the target image data is obtained by determining the target region of the prohibited public type and performing mask processing. At the same time, the audio data is converted into target text data to ensure that the voice features of the target object are not leaked. This ensures that the cloud server can obtain useful image information for analysis while strictly protecting the content involving privacy, effectively avoiding the risk of privacy leakage during data transmission and use, and fully considering data security and the privacy rights and interests of users.

[0109] In step S105 of some embodiments, the target answer data is the result data formed after the target model processes and analyzes the question or demand raised by the user, aiming to meet the expectations of the target object.

[0110] Specifically, if the target model is a local model, the processor in the electronic device calls the locally stored model algorithm program and related parameters, arranges the original environmental data and question information according to the input format required by the model, and then inputs the local model for operation and analysis, for example, the local image recognition model performs object recognition and scene classification on the collected environmental image, and then outputs the corresponding target answer data, such as the type and position of the obstacle in the picture as the target answer data.

[0111] If the target model is a cloud model, the electronic device packs the original environment data and the question information according to the agreed communication protocol and sends them to the cloud server through the target network. After receiving the data, the cloud model running on the cloud server processes the data by using the powerful computing resources thereof, such as performing deep semantic analysis on a large amount of video data, behavior prediction, and the like. After the processing is completed, the generated target answer data is returned to the electronic device according to the above communication protocol.

[0112] Please refer to Figure 5 In some embodiments, the data analysis method provided by the embodiments of the present application before step S106 can further include but is not limited to steps S501 to S507:

[0113] Step S501, data quality evaluation is performed on the original environment data to obtain a data quality score.

[0114] Step S502, the intention clarity degree of the question information is evaluated to obtain an intention clarity degree score.

[0115] Step S503, the question information is feature vectorized to obtain a question vector.

[0116] Step S504, the pre-set local knowledge base is feature vectorized to obtain a knowledge base vector.

[0117] Step S505, similarity calculation is performed according to the question vector and the knowledge base vector to obtain field similarity data.

[0118] Step S506, the data quality score, the intention clarity degree score, and the field similarity data are integrated and calculated to obtain confidence data of the target answer data.

[0119] Step S507, the confidence data and the pre-set confidence threshold are used to determine whether to update the target answer data.

[0120] In step S501 of some embodiments, an image clarity evaluation algorithm in the target model is used to evaluate the data quality of the image data to measure whether the image data is clear and complete. Illustratively, the gradient amplitude of the image is calculated, and the greater the gradient amplitude, the clearer the image. Commonly used are Sobel operator and Laplacian operator to calculate the gradient information of the image. Or a frequency domain-based method is used, which analyzes the proportion of high-frequency components by performing fast Fourier transform on the image. The greater the proportion of high-frequency components, the richer the image details and the clearer the image. A clarity threshold is set, for example, the average gradient amplitude of the image after being processed by the Sobel operator is greater than a certain value, which can be 50, and the specific value can be adjusted according to actual conditions and experience. Therefore, a corresponding score is given in the clarity dimension.

[0121] For audio data, the energy amplitude of the audio signal can be counted, compared with the preset reasonable volume range, and a score is given according to the interval in which the audio amplitude is located. The audio is preliminarily denoised by using an audio denoising algorithm, the change of the audio signal before and after denoising is compared, and the energy proportion of the residual noise after denoising is calculated to measure the purity of the audio quality. The completeness of the audio length can also be detected to determine whether there is an obvious missing part, and a corresponding score is given in the completeness dimension.

[0122] Finally, the quality scores corresponding to the various modal data in the original environment data are weighted and summed according to the preset importance weight data to obtain the final data quality score.

[0123] In step S502 of some embodiments, the question information can be parsed by using natural language processing technology to view whether the sentence structure is complete and the words are accurate and standard. At the same time, a semantic understanding model is used to analyze the semantic relationship of the words in the question to determine whether the overall semantics is clear. For example, if the question sentence has grammatical errors or semantic ambiguity, a score is deducted in this dimension. In addition, the keywords in the question are extracted by a keyword extraction algorithm (such as TF-IDF algorithm and TextRank algorithm) to view whether the keywords can accurately reflect the core intent and the number is reasonable. If the number of keywords is too small or the keywords cannot accurately reflect the key of the question, or the logical association between the current question and the context is determined to be weak or contradictory to the context, it will affect the intent clarity score.

[0124] The grammar and semantic analysis, keyword extraction and analysis, context association judgment and other aspects are integrated, and weighted summation is performed according to certain weight data to obtain the intent clarity score of the question information.

[0125] In step S503 of some embodiments, a pre-trained word vector model (such as commonly used models such as Word2Vec, GloVe, BERT, etc.) can be used to convert each word in the question information into a corresponding vector representation, and finally the vector representations corresponding to all words are fused to obtain a question vector corresponding to the question information.

[0126] In step S504 of some embodiments, similar to the processing of the question information, each knowledge item (which can be a knowledge content expressed in the form of a sentence or a paragraph, etc.) in the knowledge base is converted into a word vector by using a pre-trained word vector model, and then integrated into a fixed-dimensional knowledge item vector by a suitable method. The vectors of all knowledge items are fused to obtain a knowledge base vector.

[0127] In step S505 of some embodiments, the field similarity data represents the similarity between the question information and the existing knowledge base, and whether an external knowledge base needs to be introduced for additional reasoning can be inferred according to the field similarity data. The similarity calculation method can be cosine similarity, Euclidean distance, Manhattan distance, etc., without limitation.

[0128] In step S506 of some embodiments, the data quality score, the intention clarity score, and the field similarity data can be weighted and summed to obtain the confidence data by using preset weight data. It can be understood that the data quality score, the intention clarity score, the field similarity data, and the confidence data have the same value range, which can be [0, 100].

[0129] In step S507 of some embodiments, specifically, referring to Figure 6 , step S507 includes but is not limited to steps S601 to S603:

[0130] In step S601, if the confidence data is less than the preset confidence threshold, the low-confidence factor data is selected from the data quality score, the intention clarity score, and the field similarity data.

[0131] In step S602, the prompt information is generated according to the low-confidence factor data.

[0132] In step S603, the target answer data is updated according to the prompt information.

[0133] In step S601 of some embodiments, the preset confidence threshold can be 60 points (full score is 100 points). The low-confidence factor data refers to the data that has the greatest impact on the confidence data, and the data corresponding to the maximum weight data can be selected as the low-confidence factor data. When each weight data is the same, the data with the smallest value is selected as the low-confidence factor data.

[0134] In other embodiments, if the confidence data is greater than the preset confidence threshold, step S106 can be directly executed.

[0135] In step S602 of some embodiments, for example, if the low confidence factor data is a data quality score, the prompt information can be "The collected environmental image is relatively blurred. Please check whether the camera has stains or adjust the shooting angle, focal length, etc. to re-collect a clear image to obtain a more accurate answer." If the image has a problem of occlusion and poor integrity, the prompt information can be "The image is partially occluded, causing the loss of key information. Please ensure that there are no obstacles when shooting to obtain a complete image and try again." For the case of low-quality audio data, such as too low volume, the prompt information can be "The audio volume is too low, making it difficult to accurately analyze the content. Please increase the volume or collect audio data again by approaching the sound source." If the audio quality is noisy and noisy, the prompt information can be "The audio has a lot of noise, affecting the analysis result. Please change to a quiet environment to re-collect audio to facilitate accurate information acquisition."

[0136] If the low confidence factor data is an intent clarity score, the prompt information can be "The syntax structure of the question you raised is incorrect / the semantics is not clear enough. It is recommended to modify it to '[correct syntax expression or more clear semantic expression example]' so that the system can accurately understand your needs and provide an answer." For example, the original question is "How to do that thing", the prompt can be "Your question semantics is ambiguous, and it is not clear what 'that thing' refers to. Please specify the specific item name to facilitate the system to answer your question."

[0137] If the low confidence factor data is domain similarity data, the prompt information can be "The question you raised has a low matching degree with the existing knowledge in the local knowledge base. It may need to further expand the question background information or the system will try to query more related knowledge online to improve the answer. Please wait patiently", or inform the target object "The current question involves a new field, and the knowledge base does not cover related content. If you can provide more basic information related to it, it will help the system better answer your question" and the like, guiding the target object to supplement information or letting the target object know the reason why the answer may be limited.

[0138] In step S603 of some embodiments, the prompt information generated in step S602 is added to the target answer data, realizing the update of the target answer data.

[0139] The steps S601 to S603 shown in the embodiments of the present application, when the confidence degree data is lower than the pre-set confidence threshold, the low confidence factor data is accurately located, and the corresponding prompt information is generated based on the low confidence factor data, guiding the target object to adjust the input information and data, thereby improving the overall interactive experience and service efficiency, and enhancing the applicability of the data analysis method provided by the embodiments in complex and variable scenarios.

[0140] The steps S501 to S507 shown in the embodiments of the present application realize the reliability and accuracy of the target answer data from multiple aspects by performing data quality evaluation on the original environment data and performing intention clarity evaluation on the problem information. The confidence data of the finally obtained target answer data is compared with the preset threshold to determine whether to update the answer data, which can effectively guarantee the accuracy and scientificity of data analysis.

[0141] In step S106 of some embodiments, the pushing manner of the target answer data can be converting the target answer data into text data and then displaying the target answer data to the target object through a display module. The target answer data can also be converted into audio data and then played through an audio playing module.

[0142] The embodiments of the present application further provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above data analysis method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0143] Please refer to Figure 7 , Figure 7 The hardware structure of the electronic device of another embodiment is shown, which includes:

[0144] The processor 701 can be implemented in the form of a general CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0145] The memory 702 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 702 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 702 and are called and executed by the processor 701 to implement the data analysis method of the embodiments of the present application.

[0146] The input / output interface 703 is used to realize information input and output.

[0147] The communication interface 704 is configured to realize the communication interaction between the device and other devices, and can realize the communication through a wired manner (for example, a USB, a network cable, or the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, or the like).

[0148] The bus 705 is configured to transmit information between various components (for example, the processor 701, the memory 702, the input / output interface 703, and the communication interface 704) of the device.

[0149] The processor 701, the memory 702, the input / output interface 703, and the communication interface 704 are connected to each other through the bus 705.

[0150] Please refer to Figure 8 The embodiment of the present application further provides a wearable device system, which comprises:

[0151] The electronic device, the image acquisition module, the audio acquisition module, the audio playback module, the data display module, and the data transmission module are connected to each other.

[0152] The image acquisition module is configured to acquire image data, and can be a miniature camera. The audio acquisition module is configured to acquire audio data, and can be a microphone. The audio playback module is configured to play audio data. The display module is configured to display image data or text data. The data transmission module is configured to send the image data and the audio data to a cloud server, and is further configured to receive data sent by the cloud server.

[0153] In some embodiments, the above modules can be integrated in the same hardware device, or can be formed in different combinations and integrated in different wearable devices. For example, the display module, the audio playback module, the audio acquisition module, the image acquisition module, and the electronic device can be integrated in the same glasses device, and the display module is integrated in the lens. The display module and the audio playback module can be integrated in a watch device, and the remaining modules are integrated in the glasses device, and the like.

[0154] The specific implementation of the wearable device system is basically the same as the specific embodiments of the above data analysis method, and will not be described here.

[0155] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above data analysis method is realized.

[0156] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0157] The data analysis method, wearable device system, electronic device and storage medium provided by the embodiments of the present application, by obtaining the original environment data of the target object when responding to the question information of the target object, then evaluating the complexity of the question and the current network signal strength, and then selecting the most suitable model, that is, the target model from the plurality of candidate models for data processing. Finally, the target answer data analyzed by the target model is pushed to the target object. It is clear that the powerful computing resources of the cloud are used to process complex tasks, and the opportunity for local processing of lightweight models is selected, which not only effectively reduces the data transmission amount and relieves the processing load of the cloud, thereby reducing the response delay and improving the real-time response speed of the device, and ultimately ensures that more efficient assistance is provided to the target object.

[0158] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0159] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures shown, or combine certain steps, or different steps.

[0160] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0161] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0162] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a given step or its integral presence in the process, method, system, article, or apparatus having been made with a wider scope. The use of notation such as "first", "second", "third", etc. does not generally limit the areas, but is used to connect like elements or to distinguish one claim from another. These terms can be used interchangeably when appropriate. Terms concerning the relative position of elements can be interpreted such that their use adheres to their normal meaning, but they can also be interpreted to mean the opposite according to specific claims.

[0163] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0164] In several embodiments provided by the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above-mentioned units is only a logical functional division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0165] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the application.

[0166] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0167] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0168] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A data analysis method, characterized in that, Applied to electronic devices, the method includes: In response to the problem information of the target object, the original environmental data of the target object is obtained; The complexity level information is obtained by analyzing the complexity of the problem information and the original environmental data. Specifically, this includes: generating a difficulty coefficient based on the problem information; evaluating the computational workload of the original environmental data to obtain target computational workload data; and determining the complexity level information based on the difficulty coefficient and the target computational workload data. The network signal strength of the target network is obtained; wherein the electronic device is connected to the cloud server through the target network; The target model is selected from the candidate models based on the complexity level information and the network signal strength; wherein, the candidate models include at least one of the local model deployed on the electronic device and the cloud model deployed on the cloud server; Based on the target model, the original environmental data and the problem information are used to answer the questions, thereby obtaining the target answer data; The target answer data is pushed to the target object; The original environmental data includes audio data and environmental image data of the target object. Before answering the question based on the original environmental data and the question information according to the target model to obtain the target answer data, the method further includes: responding to the target model being the cloud model, performing scene detection on the environmental image data to obtain candidate image data; determining the target region of the target type in the candidate image data and performing masking processing on the target region to obtain target image data; wherein the target type represents a type that is prohibited from being disclosed; performing text conversion processing on the audio data to obtain target text data; and sending the target image data and the target text data to the cloud server. Before pushing the target answer data to the target object, the method further includes: performing a data quality assessment on the original environmental data to obtain a data quality score; performing an intent clarity assessment on the question information to obtain an intent clarity score; integrating and calculating the data quality score, the intent clarity score, and domain similarity data to obtain confidence data for the target answer data; and determining whether to update the target answer data based on the confidence data and a preset confidence threshold.

2. The method according to claim 1, characterized in that, The step of selecting the target model from the candidate models based on the complexity level information and the network signal strength includes: If the network signal strength is greater than or equal to the preset signal strength, the local model or the cloud model is used as the target model according to the complexity level information; If the network signal strength is less than the preset signal strength, then the local model is used as the target model.

3. The method according to claim 2, characterized in that, The complexity level information includes a first complexity level and a second complexity level, wherein the first complexity level is greater than the second complexity level; the step of using the local model or the cloud model as the target model based on the complexity level information includes: If the complexity level information is the first complexity level, the cloud model will be used as the target model; If the complexity level information is the second complexity level, the local model is used as the target model.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes obtaining the domain similarity data: The problem information is vectorized into feature vectors to obtain the problem vector; The preset local knowledge base is vectorized to obtain a knowledge base vector; Similarity is calculated based on the question vector and the knowledge base vector to obtain domain similarity data.

5. The method according to claim 1, characterized in that, The step of updating the target answer data based on the confidence data and a preset confidence threshold includes: If the confidence data is less than the preset confidence threshold, low confidence factor data is selected from the data quality score, the intent clarity score, and the domain similarity data; A prompt message is generated based on the low confidence factor data; Update the target answer data according to the prompt information.

6. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 5.

7. A wearable device system, characterized in that, The wearable device system includes: The electronic device, image acquisition module, audio acquisition module, audio playback module, data display module, and data transmission module as described in claim 6, wherein the electronic device is connected to the image acquisition module, the audio acquisition module, the audio playback module, the display module, and the data transmission module respectively; The image acquisition module is used to acquire image data; the audio acquisition module is used to acquire audio data; the audio playback module is used to play audio data; the display module is used to display image data or text data; the data transmission module is used to send image data and audio data to a cloud server, and the data transmission module is also used to receive data sent by the cloud server.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images