Data analysis method, electronic equipment, wearable equipment system and storage medium

Abstract

The embodiment of the invention provides a data analysis method, electronic equipment, a wearable equipment system and a storage medium, is applied to the electronic equipment, and belongs to the technical field of artificial intelligence. The method comprises the steps of obtaining original environment data of a target object in response to question information of the target object; performing complexity analysis according to the problem information and the original environment data to obtain complexity level information; the network signal intensity of the target network is acquired, and the electronic equipment is networked with the cloud server through the target network. And selecting a target model from candidate models according to the complex level information and the network signal intensity, wherein the candidate models comprise at least one of a local model deployed in the electronic equipment and a cloud model deployed in a cloud server. And then performing question answering on the original environment data and the question information according to the target model to obtain target answer data, and pushing the target answer data to the target object. The embodiment of the invention can improve the response speed of the electronic equipment.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data analysis method, an electronic device, a wearable device system, and a storage medium. Background Art

[0002] In related technologies, surrounding environment data (such as video and voice) is collected through different electronic devices (such as wearable devices) and sent to a cloud service center for processing, and then the processing results are fed back to the user to help the user complete the activity safely. However, as the amount of data processed continues to increase, the load of cloud processing also increases, which increases the delay in data transmission and ultimately leads to a decrease in the response speed of electronic devices. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a data analysis method, an electronic device, a wearable device system and a storage medium, aiming to improve the response speed of the electronic device.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a data analysis method, which is applied to an electronic device, and the method includes:

[0005] In response to the question information of the target object, acquiring original environment data of the target object;

[0006] Performing complexity analysis based on the problem information and the original environment data to obtain complexity level information;

[0007] Acquire the network signal strength of the target network; wherein the electronic device is connected to the cloud server via the target network;

[0008] Selecting a target model from candidate models according to the complexity level information and the network signal strength; wherein the candidate models include at least one of a local model deployed on the electronic device and a cloud model deployed on the cloud server;

[0009] Answer the original environment data and the question information according to the target model to obtain target answer data;

[0010] The target answer data is pushed to the target object.

[0011] In some embodiments, selecting a target model from candidate models according to the complexity level information and the network signal strength includes:

[0012] If the network signal strength is greater than or equal to a preset signal strength, using the local model or the cloud model as the target model according to the complexity level information;

[0013] If the network signal strength is less than the preset signal strength, the local model is used as the target model.

[0014] In some embodiments, the complexity level information includes a first complexity level and a second complexity level, and the first complexity level is greater than the second complexity level; and taking the local model or the cloud model as the target model according to the complexity level information includes:

[0015] If the complexity level information is the first complexity level, taking the cloud model as the target model;

[0016] If the complexity level information is the second complexity level, the local model is used as the target model.

[0017] In some embodiments, performing complexity analysis according to the problem information and the original environment data to obtain complexity level information includes:

[0018] Generate a difficulty coefficient according to the problem information to obtain a difficulty coefficient;

[0019] Performing data computation amount evaluation on the original environment data to obtain target computation amount data;

[0020] The complexity level information is determined according to at least one of the difficulty coefficient and the target calculation amount data.

[0021] In some embodiments, the original environmental data includes audio data and environmental image data of the target object; before answering the question based on the target model to the original environmental data and the question information to obtain the target answer data, the method further includes:

[0022] In response to the target model being the cloud model, performing scene detection on the environmental image data to obtain candidate image data;

[0023] Determine a target area of ​​a target type in the candidate image data, and perform mask processing on the target area to obtain target image data; wherein the target type represents a type that is prohibited from being disclosed;

[0024] Performing text conversion processing on the audio data to obtain target text data;

[0025] The target image data and the target text data are sent to the cloud server.

[0026] In some embodiments, before pushing the target answer data to the target object, the method further includes:

[0027] Performing data quality assessment on the original environmental data to obtain a data quality score;

[0028] Evaluate the clarity of intent of the question information to obtain a clarity of intent score;

[0029] Performing feature vectorization on the problem information to obtain a problem vector;

[0030] Perform feature vectorization on the preset local knowledge base to obtain a knowledge base vector;

[0031] Perform similarity calculation based on the question vector and the knowledge base vector to obtain domain similarity data;

[0032] Performing integrated calculation based on the data quality score, the intention clarity score and the domain similarity data to obtain confidence data of the target answer data;

[0033] The target answer data is updated according to the confidence data and a preset confidence threshold.

[0034] In some embodiments, the updating of the target answer data according to the confidence data and a preset confidence threshold comprises:

[0035] If the confidence data is less than the preset confidence threshold, selecting low confidence factor data from the data quality score, the intention clarity score and the domain similarity data;

[0036] Generate prompt information based on low confidence factor data;

[0037] The target answer data is updated according to the prompt information.

[0038] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0039] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application proposes a wearable device system, comprising: the electronic device as described in the second aspect, an image acquisition module, an audio acquisition module, an audio playback module, a data display module and a data transmission module, wherein the electronic device is respectively connected to the image acquisition module, the audio acquisition module, the audio playback module, the display module and the data transmission module;

[0040] Among them, the image acquisition module is used to acquire image data; the audio acquisition module is used to acquire audio data; the audio playback module is used to play audio data; the display module is used to display image data or text data; the data transmission module is used to send image data and audio data to a cloud server, and the data transmission module is also used to receive data sent by the cloud server.

[0041] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0042] The data analysis method, electronic device, wearable device system and storage medium proposed in this application obtain the original environmental data of the target object when responding to the question information of the target object, and then evaluate the complexity of the problem and the current network signal strength, and then select the most suitable model from multiple candidate models, that is, the target model for data processing. Finally, the target answer data obtained by the target model analysis is pushed to the target object. It clarifies the use of powerful computing resources in the cloud to handle complex tasks, and the timing of selecting lightweight models for local processing, which not only effectively reduces the amount of data transmission and alleviates the processing load on the cloud, thereby reducing response delays and improving the real-time response speed of the device, and ultimately ensures that more efficient auxiliary support is provided to the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of the data analysis method provided in an embodiment of the present application;

[0044] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0045] Figure 3 yes Figure 1 Flow chart of step S104 in FIG.

[0046] Figure 4 is another flow chart of the data analysis method provided in an embodiment of the present application;

[0047] Figure 5 is another flow chart of the data analysis method provided in an embodiment of the present application;

[0048] Figure 6 yes Figure 5 Flow chart of step S507 in FIG.

[0049] Figure 7 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0050] Figure 8 Schematic diagram of module connection of the wearable device system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] First, some nouns involved in this application are analyzed:

[0055] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0056] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0057] Information Extraction: A text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data are all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

[0058] Image caption generates a natural language description for an image and uses the generated description to help applications understand the semantics expressed in the image visual scene. For example, image captioning can convert image retrieval into text retrieval, which is used to classify images and improve image retrieval results. People usually only need to take a quick glance to describe the details of the image visual scene, while automatically adding descriptions to images is a comprehensive and arduous computer vision task that requires converting the complex information contained in the image into a natural language description. Compared with ordinary computer vision tasks, image captioning not only requires identifying objects from images, but also requires associating the identified objects with natural semantics and describing them in natural language. Therefore, image description requires people to extract deep features of images, associate them with semantic features, and convert them for generating descriptions.

[0059] For people with weaker sensory functions, such as the visually impaired, the hearing impaired, and some elderly people, they can carry different electronic devices (such as wearable devices) to collect surrounding environment data (such as video and voice) and send it to the cloud service center for processing, and then feed back the processing results to the user to help the user complete the activity safely. However, as the amount of data processed continues to increase, the load of cloud processing also increases, which increases the delay in data transmission and ultimately leads to a decrease in device response speed.

[0060] In addition, since data analysis requires uploading the collected image data and audio data to the cloud server, there may be a risk of data leakage during transmission to the cloud, thus affecting user privacy and security.

[0061] Based on this, the embodiments of the present application provide a data analysis method, an electronic device, a wearable device system and a storage medium, aiming to improve the response speed of the electronic device.

[0062] The data analysis method, electronic device, wearable device system and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the data analysis method in the embodiments of the present application is described.

[0063] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0064] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0065] The data analysis method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The data analysis method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data analysis method, etc., but is not limited to the above forms.

[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0067] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0068] Figure 1 is an optional flow chart of the data analysis method provided in the embodiment of the present application. Figure 1 The method is applied to an electronic device and may include but is not limited to steps S101 to S106.

[0069] Step S101, in response to the problem information of the target object, obtaining the original environment data of the target object.

[0070] Step S102, performing complexity analysis based on the problem information and the original environment data to obtain complexity level information.

[0071] Step S103, obtaining the network signal strength of the target network.

[0072] Step S104: selecting a target model from candidate models according to the complexity level information and the network signal strength.

[0073] Step S105, answering the original environment data and question information according to the target model to obtain target answer data.

[0074] Step S106: Push the target answer data to the target object.

[0075] Steps S101 to S106 shown in the embodiment of the present application, by obtaining the original environmental data of the target object when responding to the question information of the target object, and then evaluating the complexity of the problem and the current network signal strength, and then selecting the most suitable model from multiple candidate models, that is, the target model for data processing. Finally, the target answer data obtained by the target model analysis is pushed to the target object. It clarifies the use of powerful computing resources in the cloud to handle complex tasks, and the timing of selecting lightweight models for local processing, which not only effectively reduces the amount of data transmission and alleviates the processing load of the cloud, thereby reducing response delays and improving the real-time response speed of the device, and ultimately ensures that more efficient auxiliary support is provided for the target object.

[0076] In step S101 of some embodiments, the target object is a person carrying an electronic device. The question information may be a query or request input by the target object to the electronic device, and the question information may be in the form of text data, voice data, and image data. The question information may be input through a keyboard or key input, or may be input through a voice acquisition module, but is not limited thereto.

[0077] It should be noted that the original environmental data includes at least one of the following data: audio data around the target object, image data collected from the surrounding environment, ambient temperature data, and distance data between the target object and surrounding obstacles, but is not limited thereto. The specific composition of the original environmental data acquired each time needs to be determined according to the needs of the question information. Exemplarily, the question information of the target object is "Who is the person in front of me?", and it can be determined that the original environmental data is the image data collected from the surrounding environment. Subsequently, the question information changes to "I want to go to the nearest library now, can you navigate for me?", then the original environmental data may include real-time location data, image data collected from the surrounding environment, audio data around the target object, and distance data between the target object and surrounding obstacles, etc.

[0078] In step S102 of some embodiments, the complexity level information is a quantitative indicator to measure the difficulty of the current data processing task, so as to determine the model for analyzing the data later. In some embodiments, multiple complexity level information can be set.

[0079] Specifically, see Figure 2 Step S102 may include but is not limited to steps S201 to S203:

[0080] Step S201, generating a difficulty coefficient according to question information to obtain a difficulty coefficient.

[0081] Step S202, evaluating the data computing amount of the original environment data to obtain target computing amount data.

[0082] Step S203, determining complexity level information according to at least one of the difficulty coefficient and the target calculation amount data.

[0083] In step S201 of some embodiments, the question information can be semantically parsed by natural language processing technology to analyze the task type to which the question information belongs. Exemplarily, if the question information indicates that only the face and the surrounding environment image need to be recognized at present, the task type can be determined to be a recognition type. If the question information indicates that the contact's phone number or address needs to be queried, the task type can be determined to be a local query type. The question information can be analyzed to determine the number of modalities of the original environmental data. If the number of modalities of the required data is greater than a preset threshold (which can be 3), the task type can be determined to be a multimodal data processing type. The provision of real-time navigation services for the target object mentioned in the above embodiments belongs to the multimodal data processing type.

[0084] A difficulty coefficient mapping table can be stored locally in the electronic device, and the table stores the mapping relationship between the task type and the difficulty coefficient. After the task type is determined, the difficulty coefficient mapping table is queried according to the task type to obtain the difficulty coefficient. It is understandable that the difficulty coefficients of the local query class, the recognition class, and the multimodal data processing class should gradually increase.

[0085] In step S202 of some embodiments, in addition to the difficulty of the task contained in the problem information affecting the complexity level information, the computational amount of the original environmental data is also a key influencing factor. The target computational amount data includes the data computational amount of each modality in the original environmental data. Exemplarily, for a 30fps video of 1080p (resolution of 1920×1080), assuming that each pixel needs to be processed once (such as color adjustment), the target computational amount, that is, the pixel processing amount per second is 1920×1080×30=62,208,000 times. For audio data, the target computational amount can be calculated by the following analytical formula: target computational amount = sampling rate×bit depth×number of channels×duration×computational amount per sample. Assume a monophonic speech with a sampling rate of 16kHz, a bit depth of 16 bits, and a duration of 10 seconds. Assume that 10 floating point operations (FLOPs) are required per sample per second. The target computational effort is 16000Hz×16bit×1×10s×10=25,600,000FLOPs.

[0086] In step S203 of some embodiments, a difficulty coefficient threshold and a calculation amount threshold corresponding to each modal data are preset. In this embodiment, the complexity level information includes a first complexity level and a second complexity level, and the first complexity level is greater than the second complexity level. In other words, if the complexity level information is the first complexity level, it indicates that it is difficult to analyze the current problem information and the original environment data. If the complexity level information is the second complexity level, it indicates that it is relatively easy to analyze the current problem information and the original environment data, and the solution result obtained is more reliable.

[0087] Specifically, if at least one of the following scenarios is met, the complexity level information can be determined to be the first complexity level: the difficulty coefficient is greater than the preset difficulty coefficient threshold; the target calculation amount data of at least one modality is greater than the calculation amount threshold of the corresponding modality. If the difficulty coefficient is less than or equal to the difficulty coefficient threshold, and all target calculation amount data are less than or equal to the calculation amount threshold of the corresponding modality, the complexity level information is determined to be the second complexity level.

[0088] It should be supplemented that, in other embodiments, when the target object repeatedly inputs the same or very similar question information within a short period of time, the complexity level information can also be directly determined to be the first complexity level.

[0089] Steps S201 to S203 shown in the embodiment of the present application can achieve a more accurate and reasonable measurement of the complexity of the processing task by generating a difficulty coefficient for the problem information and evaluating the target computing amount data for the original environment data, and determining the complexity level information based on at least one of the two. This helps to reasonably select the processing model and strategy according to the actual complexity, and improve the processing efficiency and the accuracy of the results.

[0090] In step S103 of some embodiments, the electronic device is connected to the cloud server through a target network. The target network can be Wi-Fi, 5G, 4G, 3G, 2G, or a wired network. At the operating system level of the electronic device, the signal strength parameter of the currently connected network can be obtained in real time using the network status detection interface provided by the device system. The unit is generally dBm (decibel milliwatt). The larger the value, the stronger the signal. For example,

[0091] In step S104 of some embodiments, the candidate models include at least one of a local model deployed on the electronic device and a cloud model deployed on a cloud server. It should be noted that the local model in this embodiment is obtained by pre-training a small model using a large model through knowledge distillation, and performing model parameter pruning and quantization operations on the small model after training optimization. Therefore, while the local model in this embodiment maintains good performance, the storage resources occupied can be significantly reduced. A cloud model refers to a machine learning or deep learning model that is trained, stored, and deployed in a cloud computing environment. Unlike models running on local devices, cloud models rely on the computing resources of cloud servers and are accessed and called through the network.

[0092] Specifically, see Figure 3 Step S104 may include but is not limited to steps S301 to S302:

[0093] Step S301, if the network signal strength is greater than or equal to the preset signal strength, the local model or the cloud model is used as the target model according to the complexity level information.

[0094] Step S302: If the network signal strength is less than the preset signal strength, the local model is used as the target model.

[0095] In step S301 of some embodiments, if the complexity level information is the first complexity level, the cloud model is used as the target model. If the complexity level information is the second complexity level, the local model is used as the target model.

[0096] In step S302 of some embodiments, when the network signal strength is weak, regardless of the complexity, the local model is directly selected for analysis and processing, thereby avoiding the inability to connect to the cloud server, resulting in the electronic device not responding and being unable to output analysis results.

[0097] Steps S301 to S302 shown in the embodiment of the present application reasonably select the target model by comprehensively considering the network signal strength and complexity level information. When the network signal is good, the cloud or local computing resources are flexibly allocated according to the complexity of the task, so that simple tasks can be selected for local fast processing to reduce unnecessary data transmission overhead, and complex tasks can be completed efficiently with the help of the powerful computing power and rich data support of the cloud, thereby improving the overall data processing efficiency and quality, and ensuring that users can obtain accurate answer data as soon as possible. On the other hand, when the network signal strength is insufficient, the local model is directly selected for data analysis, avoiding data transmission delays and interruptions caused by poor network, and ensuring the stability and availability of this embodiment in different network environments, so that no matter what network coverage is in, the target object can rely on the device to obtain the corresponding environmental analysis results, helping it to better understand the surrounding environment and complete activities safely.

[0098] Before step S105 in some embodiments, since the original environmental data may include the audio data and environmental image data of the target object, when a cloud model is required for data processing, there is a risk of data leakage during the process of the electronic device transmitting the original environmental data to the cloud server. Therefore, before sending the original environmental data to the cloud server, the original environmental data needs to be desensitized. For details, please refer to Figure 4 The data analysis method provided in the embodiment of the present application may also include but is not limited to steps S401 to S404:

[0099] Step S401, in response to the target model being a cloud model, performing scene detection on the environmental image data to obtain candidate image data.

[0100] Step S402 , determining a target area of ​​the target type in the candidate image data, and performing mask processing on the target area to obtain target image data.

[0101] Step S403, converting the audio data into text to obtain target text data.

[0102] Step S404, sending the target image data and the target text data to the cloud server.

[0103] In step S401 of some embodiments, scene detection is performed on the environmental image data through the scene detection algorithm in the local model, and the scene label of each environmental image can be obtained, for example: library, park, living room, kitchen, bedroom and bathroom, where bedroom and bathroom are scenes involving privacy, and images involving privacy scenes are screened out. Candidate image data refers to image data involving privacy scenes in the environmental image data that has just been remembered.

[0104] In step S402 of some embodiments, the target type represents a type that is prohibited from being disclosed, such as a safe, a window of a residence (to avoid disclosing the address), and a toilet, etc. The candidate image data is identified by a visual detection algorithm, and a region of the target type object in the candidate image data is selected according to the coordinate data frame of the identification frame, that is, the target region.

[0105] In some embodiments, the mask processing may be to set the RGB values ​​of all pixels in the target area to 0 or 255 to achieve the effect of the target area being white or black, or to replace the target area with a mosaic pattern. The image after the mask processing is the target image data.

[0106] In step S403 of some embodiments, an audio feature extraction algorithm is used to extract acoustic features that can characterize speech content from preprocessed audio data. Commonly used acoustic features include Mel-Frequency Cepstral Coefficients (MFCC), Linear Predictive Cepstral Coefficients (LPCC), Line Spectral Pairs (LSP) and Linear Predictive Coding (LPC), etc. The extracted audio feature vector is input into the selected local speech recognition model in chronological order. The speech recognition model decodes the audio features, gradually predicts the corresponding text content, and finally outputs the text information corresponding to the entire audio as the target text data, thereby avoiding the leakage of the sound features of the target object during the transmission process.

[0107] In step S404 of some embodiments, first, the target image data and the target text data are packaged and integrated according to a predetermined format. For example, the JSON (Java Script Object Notation) format can be used to embed the image data into the JSON object in a Base64 encoding manner, and the target text data is added to the JSON object as another field to form a complete data structure. Then, a suitable network communication protocol is selected for data transmission, such as the HTTP protocol and the HTTPS protocol. Then, the packaged data is sent to the designated receiving address of the cloud server according to the selected protocol. During the sending process, a corresponding timeout mechanism and an error callback function are set. For example, if the response confirmation information of the cloud server is not received within the specified time, it is determined that the sending timeout is triggered, and the corresponding error handling program may be tried to resend the data. If an error code returned by the server is received, corresponding remedial measures can be performed according to different error codes, such as resending after adjusting the data format, prompting the user of network abnormality, etc., to ensure that the data can be sent to the cloud server as accurately and completely as possible for subsequent processing.

[0108] In steps S401 to S404 shown in the embodiment of the present application, by first performing scene detection on the environmental image data to obtain candidate image data, it is possible to screen out image scene content that may involve privacy leakage from a large number of original environmental images, thereby improving the pertinence and efficiency of subsequent data processing. Then, the target image data is obtained by determining the target area of ​​the prohibited disclosure type and performing mask processing. At the same time, the audio data is converted into target text data to ensure that the sound characteristics of the target object will not be leaked. While ensuring that the subsequent cloud server can obtain useful image information for analysis, it strictly protects privacy-related content, effectively avoids the risk of privacy leakage during data transmission and use, and fully takes into account data security and user privacy rights.

[0109] In step S105 of some embodiments, the target answer data is based on the questions or requirements raised by the user, and is ultimately formed after being processed and analyzed by the target model to meet the expectations of the target object.

[0110] Specifically, if the target model is a local model, the processor in the electronic device calls the locally stored model algorithm program and related parameters, organizes the original environmental data and problem information in accordance with the input format required by the model, and inputs them. The local model performs calculations and analysis. For example, the local image recognition model performs object recognition, scene classification and other operations on the collected environmental images, and then outputs the corresponding target answer data, such as identifying the type and location of obstacles in the picture as target answer data.

[0111] If the target model is a cloud model, the electronic device packages the original environmental data and question information according to the agreed communication protocol through the target network and sends it to the cloud server. After the cloud model running on the cloud server receives the data, it uses its powerful computing resources to process it, such as deep semantic analysis and behavior prediction of large amounts of video data. After processing is completed, the generated target answer data is transmitted back to the electronic device according to the above communication protocol.

[0112] See also Figure 5 In some embodiments, before step S106, the data analysis method provided in the embodiment of the present application may also include but is not limited to steps S501 to S507:

[0113] Step S501, perform data quality assessment on the original environmental data to obtain a data quality score.

[0114] Step S502: Evaluate the clarity of the intention of the question information to obtain an intention clarity score.

[0115] Step S503, feature vectorization is performed on the question information to obtain a question vector.

[0116] Step S504, feature vectorization is performed on the preset local knowledge base to obtain a knowledge base vector.

[0117] Step S505, performing similarity calculation based on the question vector and the knowledge base vector to obtain domain similarity data.

[0118] Step S506, performing integrated calculation based on the data quality score, the intention clarity score and the domain similarity data to obtain the confidence data of the target answer data.

[0119] Step S507: determining to update the target answer data according to the confidence data and a preset confidence threshold.

[0120] In step S501 of some embodiments, the image clarity evaluation algorithm in the target model is used to evaluate the data quality of the image data to measure whether the image data is clear and complete. Exemplarily, the gradient amplitude of the image is calculated. The larger the gradient amplitude, the clearer the image. Commonly used operators include the Sobel operator and the Laplacian operator to calculate the gradient information of the image. Or based on the frequency domain method, the image is fast Fourier transformed to analyze the proportion of high-frequency components. The larger the proportion of high-frequency components, the richer the image details and the clearer it is. Set a clarity threshold, for example, the average gradient amplitude obtained after the image is processed by the Sobel operator is greater than a certain value, which can be 50. It can be adjusted according to actual conditions and experience, and then a corresponding score is given in the clarity dimension.

[0121] For audio data, the energy amplitude of the audio signal can be counted, and by comparing it with the preset reasonable volume range, a score can be given according to the interval in which the audio amplitude is. The audio denoising algorithm is used to perform preliminary denoising on the audio, and the changes in the audio signal before and after denoising are compared. The energy ratio of the remaining noise after denoising is calculated to measure the purity of the sound quality. It is also possible to give a corresponding score in the dimension of completeness by detecting whether the audio duration is complete and whether there are obvious missing parts.

[0122] Finally, the quality scores corresponding to the various modal data in the original environmental data are weighted and summed according to the preset importance weight data to obtain the final data quality score.

[0123] In step S502 of some embodiments, natural language processing technology can be used to analyze the grammatical structure of the question information to see whether the sentence structure is complete and whether the wording is accurate and standardized. At the same time, the semantic understanding model is used to analyze the semantic relationship of the words in the question to determine whether the overall semantics is clear. For example, if the question statement has grammatical errors or unclear semantics, points will be deducted in this dimension first. In addition, keywords in the question are extracted through keyword extraction algorithms (such as TF-IDF algorithm and TextRank algorithm) to see whether the keywords can accurately reflect the core intent and whether the number is reasonable. If there are too few keywords or the keywords cannot accurately reflect the key points of the problem, or if the logical relevance of the current question to the context is analyzed, it is determined that the relevance is not strong or inconsistent with the context, which will affect the score of intent clarity.

[0124] Based on comprehensive grammatical and semantic analysis, keyword extraction and analysis, contextual relevance judgment and other aspects, the data is weighted and summed up according to certain weights to obtain a score for the clarity of the intent of the question information.

[0125] In step S503 of some embodiments, a pre-trained word vector model (such as commonly used models such as Word2Vec, GloVe, BERT, etc.) can be used to convert each word in the question information into a corresponding vector representation, and finally the vector representations corresponding to all words are feature fused to obtain the question vector corresponding to the question information.

[0126] In step S504 of some embodiments, similar to the processing of question information, each knowledge item in the knowledge base (which may be knowledge content expressed in the form of a sentence or a paragraph, etc.) is converted into a word vector using a pre-trained word vector model, and then integrated into a knowledge item vector of fixed dimension by a suitable method. The vectors of all knowledge items are fused to obtain a knowledge base vector.

[0127] In step S505 of some embodiments, the domain similarity data indicates the similarity between the question information and the existing knowledge base, and whether an external knowledge base needs to be introduced for additional reasoning can be inferred based on the domain similarity data. The similarity calculation method can be cosine similarity, Euclidean distance, Manhattan distance, etc., but is not limited thereto.

[0128] In step S506 of some embodiments, the data quality score, the intention clarity score and the domain similarity data may be weighted and summed by using preset weight data to obtain confidence data. It is understandable that the value ranges of the data quality score, the intention clarity score, the domain similarity data and the confidence data are consistent, which may be [0,100].

[0129] In step S507 of some embodiments, specifically, see Figure 6 Step S507 includes but is not limited to steps S601 to S603:

[0130] Step S601: if the confidence data is less than a preset confidence threshold, low confidence factor data is selected from the data quality score, the intention clarity score and the domain similarity data.

[0131] Step S602: Generate prompt information according to the low confidence factor data.

[0132] Step S603: update the target answer data according to the prompt information.

[0133] In step S601 of some embodiments, the preset confidence threshold may be 60 points (out of 100 points). Low confidence factor data refers to data that has the greatest impact on confidence data, and data corresponding to the maximum weight data may be selected as low confidence factor data. When each weight data is the same, the data with the smallest value is selected as low confidence factor data.

[0134] In other embodiments, if the confidence data is greater than a preset confidence threshold, step S106 may be directly executed.

[0135] In step S602 of some embodiments, illustratively, if the low confidence factor data is a data quality score, the prompt message may be "The captured environmental image is relatively blurry. Please check whether the camera has stains or adjust the shooting angle, focal length, etc., and re-capture a clear image to obtain a more accurate answer." If the image has problems such as occlusion or poor integrity, the prompt message may be "Part of the image is blocked, resulting in the loss of key information. Please ensure that there are no obstacles blocking the shooting, and try again after obtaining the complete image." For the case of low audio data quality, such as too low volume, the prompt message may be "The audio volume is too low, and it is difficult to accurately analyze the content. Please increase the volume appropriately or get closer to the sound source to re-capture the audio data." If the sound quality is noisy and there is noise interference, the prompt message may be "There is a lot of noise in the audio, which affects the analysis results. Please change to a quiet environment to re-capture the audio to facilitate accurate information acquisition."

[0136] If the low confidence factor data is the intent clarity score, the prompt message may be "The grammatical structure of the question you asked is incorrect / the semantics is not clear enough. It is recommended to modify it to '[correct grammatical expression or clearer semantic expression example]' so that the system can accurately understand your needs and provide answers." For example, if the original question is "How to make that thing", the prompt may be "The semantics of your question are vague, and it is not clear what the specific 'that thing' is referring to. Please specify the specific name of the item so that the system can answer you."

[0137] If the low confidence factor data is domain similarity data, the prompt message may be "The question you asked has a low match with the existing knowledge in the local knowledge base. It may be necessary to further expand the background information of the question or the system will try to query more relevant knowledge online to improve the answer. Please wait patiently", or inform the target object "The current question involves a relatively new field, and the knowledge base does not yet cover the relevant content. If you can provide more basic information related to it, it will help the system to better answer you", etc., to guide the target object to supplement the information or let the target object know why the answer may be limited.

[0138] In step S603 of some embodiments, the prompt information generated in step S602 is added to the target answer data to update the target answer data.

[0139] In steps S601 to S603 shown in the embodiment of the present application, when the confidence data is lower than the preset confidence threshold, the low-confidence factor data is accurately located, and corresponding prompt information is generated based on the analysis of the low-confidence factor data to guide the target object to cooperate in adjusting the input information and data, thereby improving the overall interactive experience and service efficiency, and enhancing the applicability of the data analysis method provided in this embodiment in complex and changeable scenarios.

[0140] Steps S501 to S507 shown in the embodiment of the present application, by evaluating the data quality of the original environment data and evaluating the clarity of the intention of the question information, realize the comprehensive consideration of the reliability and accuracy of the target answer data from multiple aspects. The confidence data of the target answer data finally obtained is compared with the preset threshold to determine whether to update the answer data, which can effectively ensure the accuracy and scientificity of data analysis.

[0141] In step S106 of some embodiments, the target answer data may be pushed by converting the target answer data into text data and then displaying it to the target object through a display module, or by converting the target answer data into audio data and broadcasting it through an audio playback module.

[0142] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above data analysis method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0143] See also Figure 7 , Figure 7 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0144] The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0145] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 702, and the processor 701 calls and executes the data analysis method of the embodiment of this application;

[0146] Input / output interface 703, used to implement information input and output;

[0147] Communication interface 704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0148] A bus 705 that transmits information between the various components of the device (e.g., the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0149] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0150] See also Figure 8 The present application also provides a wearable device system, which includes:

[0151] As described above, the electronic device, image acquisition module, audio acquisition module, audio playback module, data display module and data transmission module, the electronic device is respectively connected to the image acquisition module, audio acquisition module, audio playback module, display module and data transmission module.

[0152] The image acquisition module is used to acquire image data and may be a micro camera. The audio acquisition module is used to acquire audio data and may be a microphone. The audio playback module is used to play audio data. The display module is used to display image data or text data. The data transmission module is used to send image data and audio data to a cloud server, and the data transmission module is also used to receive data sent by the cloud server.

[0153] In some embodiments, the above modules can be integrated into the same hardware device, or they can be integrated into different wearable devices in different combinations. For example, the display module, audio playback module, audio acquisition module, image acquisition module and electronic device can be integrated into the same eyewear device, where the display module is integrated on the lens. Alternatively, the display module and audio playback module can be integrated into a watch device, and the remaining modules are still integrated into the eyewear device, but not limited thereto.

[0154] The specific implementation of the wearable device system is basically the same as the specific implementation of the above-mentioned data analysis method, and will not be repeated here.

[0155] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned data analysis method is implemented.

[0156] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0157] The data analysis method, wearable device system, electronic device and storage medium provided in the embodiments of the present application obtain the original environmental data of the target object when responding to the question information of the target object, and then evaluate the complexity of the problem and the current network signal strength, and then select the most suitable model from multiple candidate models, that is, the target model for data processing. Finally, the target answer data obtained by the target model analysis is pushed to the target object. It clarifies the use of powerful computing resources in the cloud to handle complex tasks, and the timing of selecting lightweight models for local processing, which not only effectively reduces the amount of data transmission, alleviates the processing load of the cloud, thereby reducing response delays, and improving the real-time response speed of the device, and ultimately ensures that more efficient auxiliary support is provided for the target object.

[0158] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0159] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0160] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0161] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0162] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0163] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0164] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0165] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0166] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0167] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0168] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data analysis method, characterized in that: Applied to electronic equipment, the method comprises: In response to the question information of the target object, acquiring original environment data of the target object; Performing complexity analysis based on the problem information and the original environment data to obtain complexity level information; Acquire the network signal strength of the target network; wherein the electronic device is connected to the cloud server via the target network; Selecting a target model from candidate models according to the complexity level information and the network signal strength; wherein the candidate models include at least one of a local model deployed on the electronic device and a cloud model deployed on the cloud server; Answer the original environment data and the question information according to the target model to obtain target answer data; The target answer data is pushed to the target object.

2. The method according to claim 1, characterized in that: The selecting a target model from candidate models according to the complexity level information and the network signal strength includes: If the network signal strength is greater than or equal to a preset signal strength, using the local model or the cloud model as the target model according to the complexity level information; If the network signal strength is less than the preset signal strength, the local model is used as the target model.

3. The method according to claim 2, characterized in that The complexity level information includes a first complexity level and a second complexity level, and the first complexity level is greater than the second complexity level; and taking the local model or the cloud model as the target model according to the complexity level information includes: If the complexity level information is the first complexity level, taking the cloud model as the target model; If the complexity level information is the second complexity level, the local model is used as the target model.

4. The method according to any one of claims 1 to 3, characterized in that: The complexity level information is obtained by performing complexity analysis based on the problem information and the original environment data, including: Generate a difficulty coefficient according to the problem information to obtain a difficulty coefficient; Performing data computation amount evaluation on the original environment data to obtain target computation amount data; The complexity level information is determined according to at least one of the difficulty coefficient and the target calculation amount data.

5. The method according to any one of claims 1 to 3, characterized in that: The original environment data includes audio data and environment image data of the target object; Before answering the original environment data and the question information according to the target model to obtain target answer data, the method further includes: In response to the target model being the cloud model, performing scene detection on the environmental image data to obtain candidate image data; Determine a target area of ​​a target type in the candidate image data, and perform mask processing on the target area to obtain target image data; wherein the target type represents a type that is prohibited from being disclosed; Performing text conversion processing on the audio data to obtain target text data; The target image data and the target text data are sent to the cloud server.

6. The method according to any one of claims 1 to 3, characterized in that: Before pushing the target answer data to the target object, the method further includes: Performing data quality assessment on the original environmental data to obtain a data quality score; Evaluate the clarity of intent of the question information to obtain a clarity of intent score; Performing feature vectorization on the problem information to obtain a problem vector; Perform feature vectorization on the preset local knowledge base to obtain a knowledge base vector; Perform similarity calculation based on the question vector and the knowledge base vector to obtain domain similarity data; Performing integrated calculation based on the data quality score, the intention clarity score and the domain similarity data to obtain confidence data of the target answer data; The target answer data is updated according to the confidence data and a preset confidence threshold.

7. The method according to claim 6, characterized in that The updating of the target answer data according to the confidence data and a preset confidence threshold comprises: If the confidence data is less than the preset confidence threshold, selecting low confidence factor data from the data quality score, the intention clarity score and the domain similarity data; Generate prompt information based on low confidence factor data; The target answer data is updated according to the prompt information.

8. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

9. A wearable device system, characterized in that: The wearable device system comprises: The electronic device, the image acquisition module, the audio acquisition module, the audio playback module, the data display module and the data transmission module according to claim 8, wherein the electronic device is respectively connected to the image acquisition module, the audio acquisition module, the audio playback module, the display module and the data transmission module; Among them, the image acquisition module is used to acquire image data; the audio acquisition module is used to acquire audio data; the audio playback module is used to play audio data; the display module is used to display image data or text data; the data transmission module is used to send image data and audio data to a cloud server, and the data transmission module is also used to receive data sent by the cloud server.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images