Method and apparatus for maintenance guidance of a household appliance

Abstract

The application provides a household appliance maintenance guidance method and device, a unified vector database is constructed in advance based on text information, image information and audio information related to household appliance faults; on-site situation information of the household appliance is acquired, the on-site situation information includes video information, audio information and / or text information; the on-site situation information is subjected to image matching, audio matching and / or text matching based on the unified vector database, and a knowledge fragment is output as a search result; the search result is input into a large language model, and a household appliance maintenance guidance scheme is output. A search and reasoning framework based on direct fusion of multi-modal features can be constructed, image, audio and text features of the fault are directly analyzed, information dimension reduction loss is avoided, and the comprehensive accuracy of fault diagnosis is improved to more than 95%; through dynamic knowledge search and enhancement generation technology, the risk of "knowledge illusion" that may occur in the professional field of the general large language model is effectively reduced.

Description

Technical Field

[0001] This invention relates to the field of household appliance technology, and in particular to a method and apparatus for providing repair guidance for household appliances. Background Technology

[0002] At present, fault diagnosis and repair in the kitchen appliance industry mainly rely on the personal experience of repair technicians, which generally suffers from the following defects: (1) High skill dependence: The skill level of repair personnel varies, especially new technicians, resulting in low fault identification accuracy and repair efficiency. (2) Inefficient information retrieval: Traditional knowledge bases (such as manuals and documents) are mainly text-based, making it difficult to quickly and accurately locate faults in the face of complex audio and video fault information on site. (3) Loss of information dimensions: When processing fault pictures or videos, existing technical solutions often convert them into text descriptions first. This process will lose a lot of key visual (such as burn marks on parts, small cracks) and auditory (such as the frequency and rhythm of specific abnormal noises) details, which are precisely the core basis for accurate diagnosis. (4) Solidified knowledge system: Traditional knowledge bases are static and cannot absorb new cases and new skills generated in the front-line repair process. The knowledge system will gradually age and it will be difficult to cope with the endless new problems. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a method and apparatus for repairing household appliances, so as to construct a retrieval and reasoning framework based on direct fusion of multimodal features. By directly analyzing the image, audio and text features of the fault, information dimensionality reduction loss is avoided, and the overall accuracy of fault diagnosis is improved to over 95%. Through dynamic knowledge retrieval and enhanced generation technology, the risk of "knowledge illusion" that may occur in general large language models in professional fields is effectively reduced.

[0004] In a first aspect, embodiments of the present invention provide a method for providing repair guidance for household appliances. This method involves pre-constructing a unified vector database based on text, image, and audio data related to household appliance malfunctions. The method includes: acquiring on-site information about the household appliances, including video, audio, and / or text information; performing image matching, audio matching, and / or text matching on the on-site information based on the unified vector database, and outputting knowledge fragments as retrieval results; and inputting the retrieval results into a large language model to output a repair guidance scheme for the household appliances.

[0005] In optional embodiments of this application, the above method further includes: extracting text vectors corresponding to text data through a large text model, extracting image vectors corresponding to image data through a visual feature extraction model, and extracting audio vectors corresponding to audio data through an audio recognition model; and constructing a unified vector database based on text vectors, image vectors, and audio vectors.

[0006] In optional embodiments of this application, the step of extracting text vectors corresponding to text data through a large text model includes: performing semantic slicing on the text data using the large text model to obtain text vectors; the step of extracting image vectors corresponding to image data through a visual feature extraction model includes: determining the visual information of the image data encoded by the model as image vectors through visual features; and the step of extracting audio vectors corresponding to audio data through an audio recognition model includes: extracting the spectral features of the audio data through an audio recognition model to obtain audio vectors corresponding to the audio data.

[0007] In optional embodiments of this application, the aforementioned unified vector database includes: text vectors, image vectors, and audio vectors; text data, image data, and audio data; and metadata; the metadata includes: device model, component name, data source link, and associated fault tag.

[0008] In an optional embodiment of this application, the steps of performing image matching, audio matching, and text matching on-site situation information based on a unified vector database and outputting knowledge fragments as retrieval results include: determining query image vectors, query audio vectors, and query text vectors based on the on-site situation information; performing image matching on the query image vectors based on the unified vector database to determine the most similar visual scene as a knowledge fragment; performing audio matching on the query audio vectors based on the unified vector database to determine the most similar sound as a knowledge fragment; performing text matching on the query text vectors based on the unified vector database to determine the most similar problem description as a knowledge fragment; determining the association weights of multiple knowledge fragments, and sorting the multiple knowledge fragments based on the association weights to obtain retrieval results.

[0009] In an optional embodiment of this application, the step of sorting multiple knowledge segments based on association weights includes: determining the signal quality of multiple knowledge segments; adjusting the association weights of multiple knowledge segments based on the signal quality; and sorting the multiple knowledge segments based on the adjusted association weights.

[0010] In an optional embodiment of this application, the step of inputting the search results into a large language model and outputting a repair guidance scheme for household appliances includes: inputting the search results into a large language model and outputting textual repair guidance for household appliances; injecting knowledge fragments into a Prompt template; inputting the Prompt template into a large language model and outputting structured repair guidance; wherein, the information elements of the Prompt template include: textual knowledge information, key visual evidence information, key auditory evidence information, and information on answering user questions; and using the textual repair guidance and structured repair guidance as a repair guidance scheme for household appliances.

[0011] In optional embodiments of this application, the method further includes: obtaining a repair report of the completed repair of the household appliance; the repair report includes structured feedback and unstructured feedback, wherein the structured feedback is used to allow the repair technician to confirm repair-related issues through multiple-choice questions, and the unstructured feedback is used to allow the repair technician to input text through a text box; determining the target case marked by the repair technician based on the repair report; wherein the target case represents a case that has been successfully resolved and differs significantly from the existing unified vector database; adding the on-site situation information of the target case (which has been feature-vectorized) and the repair guidance plan of the target case to the unified vector database; adjusting the association weights in the unified vector database based on the repair report; if the repair report indicates that the target repair guidance plan is an incorrect repair guidance plan, labeling the knowledge fragments associated with the target repair guidance plan in the unified vector database with tags indicating low confidence.

[0012] In an optional embodiment of this application, the above-mentioned text big data model generates structured feedback based on the video information, audio information and / or text information involved in the repair; the text big data model evaluates the result of the repair based on the evaluation of the repair technician, and performs model learning based on the evaluation result.

[0013] Secondly, embodiments of the present invention also provide a repair guidance device for household appliances. A unified vector database is pre-constructed based on text, image, and audio data related to household appliance malfunctions. The device includes: a retrieval result output module, used to acquire on-site information about the household appliances, including video, audio, and / or text information; performing image matching, audio matching, and / or text matching on the on-site information based on the unified vector database, and outputting knowledge fragments as retrieval results; and a repair guidance scheme generation module, used to input the retrieval results into a large language model and output a repair guidance scheme for the household appliances.

[0014] The embodiments of the present invention bring the following beneficial effects:

[0015] This invention provides a method and apparatus for providing repair guidance for household appliances. The method involves first constructing a unified vector database based on text, image, and audio data related to household appliance malfunctions; acquiring on-site information about the household appliances, including video, audio, and / or text information; performing image matching, audio matching, and / or text matching on the on-site information based on the unified vector database, and outputting knowledge fragments as retrieval results; inputting the retrieval results into a large language model to output a repair guidance scheme for the household appliances. This approach constructs a retrieval and reasoning framework based on direct fusion of multimodal features. By directly analyzing the image, audio, and text features of the malfunction, it avoids information dimensionality reduction loss and improves the overall accuracy of malfunction diagnosis to over 95%. Furthermore, through dynamic knowledge retrieval and enhanced generation techniques, it effectively reduces the risk of "knowledge illusion" that may occur with general-purpose large language models in specialized fields.

[0016] Other features and advantages of this disclosure will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0017] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 A schematic diagram illustrating the construction of a unified vector database offline, provided as an embodiment of the present invention;

[0020] Figure 2 A flowchart illustrating a method for providing repair guidance for household appliances, as provided in an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram illustrating an online real-time diagnosis and interaction method provided in an embodiment of the present invention.

[0022] Figure 4 A flowchart illustrating another method for providing repair guidance for household appliances, as provided in an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of a closed-loop self-optimization mechanism provided in an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram illustrating an embodiment of the present invention for updating association weights and labeling low-confidence tags in a unified vector database.

[0025] Figure 7 This is a schematic diagram of a repair guidance device for household appliances provided in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] At present, fault diagnosis and repair in the kitchen appliance industry mainly rely on the personal experience of repair technicians, which generally suffers from the following defects: (1) High skill dependence: The skill level of repair personnel varies, especially new technicians, resulting in low fault identification accuracy and repair efficiency. (2) Inefficient information retrieval: Traditional knowledge bases (such as manuals and documents) are mainly text-based, making it difficult to quickly and accurately locate faults in the face of complex audio and video fault information on site. (3) Loss of information dimensions: When processing fault pictures or videos, existing technical solutions often convert them into text descriptions first. This process will lose a lot of key visual (such as burn marks on parts, small cracks) and auditory (such as the frequency and rhythm of specific abnormal noises) details, which are precisely the core basis for accurate diagnosis. (4) Solidified knowledge system: Traditional knowledge bases are static and cannot absorb new cases and new skills generated in the front-line repair process. The knowledge system will gradually age and it will be difficult to cope with the endless new problems.

[0028] Based on this, the present invention provides a method and apparatus for repairing household appliances, specifically a kitchen appliance fault diagnosis and repair guidance system based on multimodal feature fusion and closed-loop self-optimization, which can avoid information dimensionality reduction loss and improve the overall accuracy of fault diagnosis.

[0029] To facilitate understanding of this embodiment, a detailed description of a repair guidance method for household appliances disclosed in this embodiment of the invention will be provided first.

[0030] Example 1:

[0031] This invention provides a method for guiding the repair of household appliances, which involves pre-constructing a unified vector database based on text, image, and audio data related to household appliance malfunctions. See also... Figure 1The diagram illustrates an offline method for constructing a unified vector database. This embodiment can construct a multimodal knowledge base, i.e., a unified vector database, based on original data such as text, image, and audio data.

[0032] In some embodiments, text vectors corresponding to text data can be extracted using a large text model, image vectors corresponding to image data can be extracted using a visual feature extraction model, and audio vectors corresponding to audio data can be extracted using an audio recognition model; a unified vector database can be constructed based on text vectors, image vectors, and audio vectors.

[0033] In this embodiment, multimodal feature vectorization can be performed to convert unstructured raw data into machine-understandable, high-fidelity feature vectors.

[0034] In some embodiments, text data can be semantically sliced ​​using a large text model to obtain text vectors; visual information of image data encoded by the model can be determined as image vectors using visual features; and spectral features of audio data can be extracted using an audio recognition model to obtain audio vectors corresponding to the audio data.

[0035] 1. Text Data Processing: For PDF (Portable Document Format) manuals, SOP (Standard Operating Procedure) documents, etc., use large text models such as DeepSeek-V3 to perform semantic slicing and generate text embedding vectors.

[0036] 2. Image Data Processing: For fault images and video keyframes, use visual feature extraction models such as CLIP (Contrastive Language–Image Pre-training) and ViT (Vision Transformer) to generate image embedding vectors. These vectors can encode visual information such as color, texture, shape, and component status of the image.

[0037] 3. Video data processing: For audio tracks or individual fault recordings in maintenance videos, use audio recognition models such as AudioSpectrogram Transformer to extract their spectral features and generate audio embedding vectors. These vectors can distinguish between different types of mechanical friction sounds, electrical sounds, or gas leak sounds.

[0038] In some embodiments, the unified vector database mentioned above includes: text vectors, image vectors, and audio vectors; text data, image data, and audio data; metadata includes: device model, component name, data source link, and associated fault tag.

[0039] like Figure 1 As shown, this embodiment can construct a unified vector database based on text vectors, image vectors, and audio vectors. It utilizes professional vector databases that support multi-vector indexing, such as Milvus and Weaviate, to establish the unified vector database. This database can simultaneously store: three types of embedding vectors (text, image, and audio); raw data (text fragments, image files, audio / video clips); and metadata (such as device model, component name, data source link, associated fault tags, etc.).

[0040] Based on the above description, see Figure 2 The flowchart shown illustrates a repair guidance method for a household appliance, which includes the following steps:

[0041] Step S202: Obtain on-site situation information of household appliances, including video information, audio information and / or text information; perform image matching, audio matching and / or text matching on the on-site situation information based on a unified vector database, and output knowledge fragments as search results.

[0042] See Figure 3 The diagram illustrates an online real-time diagnostic and interactive system. Technicians can collect on-site information about home appliances via a mobile app (application). This information can include short videos, images, and voice / text descriptions. This embodiment can perform image matching, audio matching, and / or text matching on the on-site information based on a unified vector database, outputting corresponding knowledge fragments as retrieval results.

[0043] In some embodiments, query image vectors, query audio vectors, and query text vectors can be determined based on on-site situation information; image matching is performed on the query image vectors based on a unified vector database to determine the most similar visual scene as a knowledge fragment; audio matching is performed on the query audio vectors based on a unified vector database to determine the most similar sound as a knowledge fragment; text matching is performed on the query text vectors based on a unified vector database to determine the most similar question description as a knowledge fragment; the association weights of multiple knowledge fragments are determined, and the multiple knowledge fragments are sorted based on the association weights to obtain the retrieval results.

[0044] This embodiment can determine the query image vector, query audio vector, and query text vector based on on-site information. For example, keyframe images can be extracted from a video to generate a query image vector; audio can be extracted from a video to generate a query audio vector; and the technician's voice or text description can be used to generate a query text vector.

[0045] like Figure 3As shown, this embodiment can also perform cross-modal fusion retrieval, that is, perform parallel, cross-modal similarity searches. For example: audio matching: using the query audio vector to search for the most similar sound in the audio vectors of the knowledge base; image matching: using the query image vector to search for the most similar visual scene in the image vectors of the knowledge base; text matching: using the query text vector to search for the most similar question description in the text vectors of the knowledge base.

[0046] In this embodiment, a dynamic weighted fusion algorithm can be used to sort the three search results. For example, if the energy and features of the input audio signal are significant (such as strong abnormal noise), the weight of the audio matching result will be dynamically increased.

[0047] In some embodiments, the signal quality of multiple knowledge fragments can be determined; the association weights of the multiple knowledge fragments can be adjusted based on the signal quality; and the multiple knowledge fragments can be sorted based on the adjusted association weights.

[0048] The dynamic weighted fusion algorithm can be:

[0049] ;

[0050] The dynamic weight adjustment rules can be:

[0051]

[0052] Aq and Adb: Aq represents the query audio vector, which is the vector converted from the audio signal input by the user for retrieval; Adb is the audio vector in the knowledge base, used for similarity matching with the query audio vector.

[0053] Iq and Idb: Iq is the query image vector, which is the vector transformed from the image input by the user; Idb is the image vector in the knowledge base, used to calculate the similarity with the query image vector.

[0054] Tq and Tdb: Tq refers to the query text vector, which is the vector transformed from the text input by the user; Tdb is the text vector in the knowledge base, used to perform similarity matching with the query text vector.

[0055] Sim(Aq,Adb), Sim(Iq,Idb), and Sim(Tq,Tdb) represent the similarity scores between the query audio and the knowledge base audio, the query image and the knowledge base image, and the query text and the knowledge base text, respectively, and are used to measure the degree of matching between the two.

[0056] α, β, γ: These are dynamically adjusted weighting coefficients, corresponding to the weights of audio, image, and text matching results in the fusion score, respectively. They change dynamically based on signal quality (such as audio energy, image clarity, text confidence, etc.).

[0057] Audio_Energy: Peak spectral energy (dB) of the input audio, reflecting the strength of the audio energy. When the energy is high (such as a strong abnormal noise), the audio matching weight will be dynamically increased.

[0058] Image_Clarity_Score: A score (0-1) based on image blurriness / contrast. The higher the score, the clearer the image, and it affects the weight of the image matching result during fusion.

[0059] Text_Confidence: The confidence level of keyword matching in the text description. A high confidence level indicates a good match between the text and the knowledge base text, which will affect the weight of the text matching results.

[0060] Fused_Score: The fusion score is calculated by combining the similarity results of audio, image, and text matching through a dynamic weighted fusion algorithm with their respective weights α, β, and γ, and is used to rank the search results.

[0061] Max_Energy: This is a set reference value for the maximum audio energy, used to compare with the actual Audio_Energy to help determine the dynamic adjustment of the audio weight α.

[0062] Threshold: The threshold value. When Audio_Energy exceeds this threshold, the weight α of the audio matching result will be dynamically adjusted according to Audio_Energy / Max_Energy.

[0063] In this embodiment, the weights of audio, image, and text matching results in the fusion score can be dynamically changed based on signal quality (such as audio energy, image clarity, text confidence, etc.). This allows for flexible adjustment of association weights, thereby accurately sorting multiple knowledge fragments.

[0064] Step S204: Input the search results into the large language model and output the repair guidance plan for home appliances.

[0065] In some embodiments, the search results can be input into a large language model to output textual repair instructions for home appliances; knowledge fragments can be injected into a Prompt template; the Prompt template can be input into a large language model to output structured repair instructions; wherein, the information elements of the Prompt template include: textual knowledge information, key visual evidence information, key auditory evidence information, and information on answering user questions; the textual repair instructions and structured repair instructions are used as a repair guidance scheme for home appliances.

[0066] like Figure 3As shown, this embodiment can also perform enhanced reasoning and generation based on the retrieval results. The retrieval results are input into a large language model to output textual guidance schemes for the repair of home appliances. The RAG (Retrieval-Augmented Generation) architecture is adopted, which injects multiple high-scoring multimodal knowledge fragments (which may be text descriptions, a best-matching fault diagram, or a most similar abnormal sound audio) into a domain-optimized Prompt template (a structured framework for specific types of output).

[0067] For example, a Prompt instance could be: You are an experienced kitchen appliance repair expert. Please answer the user's question: {{query}} by combining the following high-fidelity knowledge: [textual knowledge: {{retrieved_text}}], [key visual evidence: {{retrieved_image_link}}], and [key auditory evidence: {{retrieved_audio_link}}].

[0068] like Figure 3 As shown, this embodiment can also perform system question and answer. The prompt is input into a large language model such as Qwen2.5-32b to generate structured repair steps. The answer not only includes text guidance but also directly embeds links to the most relevant images and video clips retrieved from the knowledge base, allowing technicians to view them at any time.

[0069] This invention provides a method for providing repair guidance for household appliances. The method involves first constructing a unified vector database based on text, image, and audio data related to household appliance malfunctions; then acquiring on-site information about the household appliance, including video, audio, and / or text information; performing image matching, audio matching, and / or text matching on the on-site information based on the unified vector database, and outputting knowledge fragments as retrieval results; finally, inputting the retrieval results into a large language model to output a repair guidance scheme for the household appliance. This approach constructs a retrieval and reasoning framework based on direct fusion of multimodal features. By directly analyzing the image, audio, and text features of the malfunction, it avoids information dimensionality reduction loss and improves the overall accuracy of malfunction diagnosis to over 95%. Furthermore, through dynamic knowledge retrieval and enhanced generation techniques, it effectively reduces the risk of "knowledge illusion" that may occur with general-purpose large language models in specialized fields.

[0070] Example 2:

[0071] This embodiment provides another method for repairing household appliances. This method is implemented based on the above embodiment, focusing on the specific implementation of the closed-loop self-optimization mechanism. (See [link to previous embodiment]). Figure 4The flowchart shown represents another method for repairing a household appliance, which includes the following steps:

[0072] Step S402: Obtain on-site information of household appliances, including video information, audio information and / or text information; perform image matching, audio matching and / or text matching on the on-site information based on a unified vector database, and output knowledge fragments as search results.

[0073] Step S404: Input the search results into the large language model and output the repair guidance plan for home appliances.

[0074] Step S406: Obtain the repair report of the household appliance repair completion; after feature vectorization, add the repair report to the unified vector database.

[0075] See Figure 5 The diagram shows a closed-loop self-optimization mechanism. In this embodiment, the repair report of the completed repair of household appliances can also be vectorized and added to a unified vector database.

[0076] The aforementioned repair report includes structured feedback and unstructured feedback. Structured feedback is used to allow repair technicians to confirm repair-related issues through multiple-choice questions, while unstructured feedback is used to allow repair technicians to input text through text boxes.

[0077] The structured feedback mentioned above can also be generated by the text-based large model based on video, audio, and / or text information involved in the repair. The text-based large model can evaluate the results of the repair based on the repair technician's assessment and conduct a new round of model learning based on the evaluation results.

[0078] For example, structured feedback involves using multiple-choice questions to confirm the root cause of the fault, the replaced parts, and whether the repair was successful. Unstructured feedback, on the other hand, is a text box for the technician to evaluate the effectiveness of the system guidance or to add new techniques or problems discovered during the repair process.

[0079] In some embodiments, target cases marked by maintenance technicians can be identified based on maintenance reports; wherein, target cases represent cases that have been successfully resolved and differ significantly from the existing unified vector database; the on-site situation information of the target cases that have undergone feature vectorization, as well as the maintenance guidance plan for the target cases, are added to the unified vector database.

[0080] like Figure 5As shown, this embodiment can automatically add knowledge to the database: For target cases marked as "successfully resolved" by maintenance technicians and that differ significantly from the existing knowledge base, the system (including the initial multimodal input and the solution confirmed by the technician) is treated as a high-quality Q&A pair, automatically vectorized, and added to the unified knowledge base after a small amount of manual review.

[0081] In some embodiments, the association weights in the unified vector database can also be adjusted based on the maintenance report.

[0082] See Figure 6 The diagram illustrates an update of association weights and labeling of low-confidence tags in a unified vector database. This embodiment also allows for dynamic adjustment of knowledge weights: the system statistically analyzes feedback data. If a solution is frequently adopted and receives positive feedback, its association weight and retrieval ranking in the knowledge base will automatically increase. Conversely, if a solution is frequently marked as "invalid," its weight will be reduced, achieving a process of natural selection in knowledge.

[0083] In some embodiments, if the repair report indicates that the target repair guidance is an incorrect repair guidance, the knowledge fragments associated with the target repair guidance in the unified vector database are labeled with tags indicating low confidence.

[0084] like Figure 6 As shown, this embodiment can also perform self-correction of "knowledge illusion": for guidance schemes generated by large models but proven to be wrong in front-line practice, the system can automatically label the associated knowledge fragments with "low credibility" based on negative feedback, thereby suppressing their appearance in future searches.

[0085] The method provided in this embodiment of the invention can establish a closed-loop self-optimizing knowledge base iteration mechanism. By utilizing real-time feedback data from front-line maintenance personnel, the system knowledge base can learn, correct, and continuously grow, forming a dynamically evolving "living" knowledge system.

[0086] This embodiment also provides a specific example, using "the range hood does not smoke" as an example for illustration.

[0087] 1. Technician input: Film a video of the motor running, which contains a clear metallic friction sound, and verbally describe that "the motor is turning, but it doesn't feel like there is any suction".

[0088] II. System Multimodal Analysis: Generate audio vectors (encoding the characteristics of metallic friction sound). Generate image vectors (encoding the motor appearance and turbine rotation status). Generate text vectors (from the speech-to-text conversion "motor is turning, but there is no suction").

[0089] III. Cross-modal fusion search results: Audio matching: Highly similar to the audio sample in the knowledge base for "turbine bearing failure or foreign object," (similarity 0.91). Text matching: Related to the text description "motor idling, insufficient suction," (similarity 0.85). Image matching: Matched with the image of "turbine rotating normally," ruling out turbine jamming. Fusion ranking: Due to the highly significant audio features, the audio matching result received the highest weight, and the system highly suspects a bearing or foreign object problem.

[0090] IV. Generating Guidance Schemes:

[0091] (1) Preliminary diagnosis: Based on the sound and images you provided, it is highly suspected that the internal bearing of the turbine is damaged or that foreign objects have been sucked in, causing friction.

[0092] (2) Maintenance Procedures: Safe Operation: Please ensure the equipment power is disconnected. Disassembly and Inspection: Remove the turbine air box (Refer to video: V-2035, click to view disassembly details). Problem Localization: Check the turbine bearing for looseness or metal debris (compare images: I-5012, bearing damage characteristics), and check for foreign objects in the air duct. Solution: If bearing damage is found, the entire turbine module needs to be replaced (Note: Please verify the equipment model when replacing...).

[0093] V. Closed-Loop Feedback: After the technician completes the repair, they select "Cause of Fault: Bearing Damage" in the app and click "Repair Successful." This case is then systematically studied, enhancing the correlation between "metallic friction noise" and "bearing damage."

[0094] In summary, the core of the home appliance repair guidance method provided in this invention lies in the fact that, offline, the system no longer forcibly converts all repair data into text. Instead, it utilizes multiple AI models to extract multimodal feature vectors from equipment manuals (text), fault diagrams (images), and repair videos (video keyframes + audio), constructing a unified, multi-dimensional vector knowledge base. During online diagnosis, the system directly analyzes the fault videos captured on-site, extracting feature vectors from their images, audio, and text descriptions, performing cross-modal fusion retrieval, and accurately matching the most similar fault scenarios (including similar images, similar sounds, and similar descriptions) in the knowledge base. Finally, using a RAG architecture, the retrieved high-fidelity multimodal knowledge is injected into a large language model to generate accurate repair guidance solutions, along with links to the most relevant original materials (images, video clips). Simultaneously, this invention designs a repair feedback loop, feeding back successful front-line cases into the knowledge base, enabling the system's self-learning and iteration.

[0095] The method provided in this invention can construct a retrieval and reasoning framework for the kitchen appliance field based on direct fusion of multimodal features. By directly analyzing the image, audio, and text features of faults, it avoids information loss due to dimensionality reduction and improves the overall accuracy of fault diagnosis. It can also establish a closed-loop self-optimizing knowledge base iteration mechanism in the kitchen appliance field. By utilizing real-time feedback data from front-line maintenance personnel, the system knowledge base can learn and correct itself, dynamically "evolving" the knowledge system and enhancing the real-time performance and accuracy of knowledge.

[0096] Example 3:

[0097] Corresponding to the above method embodiments, this invention provides a repair guidance device for household appliances, which pre-constructs a unified vector database based on text, image, and audio data related to household appliance malfunctions. (See also...) Figure 7 The diagram shows a structural schematic of a repair guidance device for a household appliance. The device includes:

[0098] The retrieval result output module 71 is used to obtain on-site information of household appliances, including video information, audio information and / or text information; based on a unified vector database, image matching, audio matching and / or text matching are performed on the on-site information respectively, and knowledge fragments are output as retrieval results;

[0099] The repair guidance solution generation module 72 is used to input the search results into the large language model and output the repair guidance solution for home appliances.

[0100] This invention provides a repair guidance device for household appliances. It first constructs a unified vector database based on text, image, and audio data related to household appliance malfunctions. Then, it acquires on-site information about the household appliance, including video, audio, and / or text information. Based on the unified vector database, it performs image matching, audio matching, and / or text matching on the on-site information, outputting knowledge fragments as retrieval results. The retrieval results are input into a large language model to output a repair guidance scheme for the household appliance. This approach constructs a retrieval and reasoning framework based on direct fusion of multimodal features. By directly analyzing the image, audio, and text features of the malfunction, it avoids information dimensionality reduction loss and improves the overall accuracy of malfunction diagnosis to over 95%. Through dynamic knowledge retrieval and enhanced generation techniques, it effectively reduces the risk of "knowledge illusion" that may occur with general-purpose large language models in specialized fields.

[0101] The aforementioned unified vector database construction module is used to extract text vectors corresponding to text data through a large text model, extract image vectors corresponding to image data through a visual feature extraction model, and extract audio vectors corresponding to audio data through an audio recognition model; and construct a unified vector database based on text vectors, image vectors, and audio vectors.

[0102] The aforementioned unified vector database construction module is used to perform semantic slicing on text data using a large text model to obtain text vectors; to determine the visual information of the model-encoded image data as image vectors using visual features; and to extract the spectral features of audio data using an audio recognition model to obtain the corresponding audio vectors.

[0103] The aforementioned unified vector database includes: text vectors, image vectors, and audio vectors; text data, image data, and audio data; and metadata. The metadata includes: device model, component name, data source link, and associated fault tags.

[0104] The aforementioned retrieval result output module is used to determine the query image vector, query audio vector, and query text vector based on the on-site situation information; perform image matching on the query image vector based on the unified vector database to determine the most similar visual scene as a knowledge fragment; perform audio matching on the query audio vector based on the unified vector database to determine the most similar sound as a knowledge fragment; perform text matching on the query text vector based on the unified vector database to determine the most similar question description as a knowledge fragment; determine the association weights of multiple knowledge fragments, and sort the multiple knowledge fragments based on the association weights to obtain the retrieval results.

[0105] The above-mentioned search result output module is used to determine the signal quality of multiple knowledge fragments; adjust the association weights of multiple knowledge fragments based on the signal quality; and sort the multiple knowledge fragments based on the adjusted association weights.

[0106] The aforementioned repair guidance generation module is used to input the search results into a large language model and output textual repair guidance for household appliances; inject knowledge fragments into a Prompt template; input the Prompt template into the large language model and output structured repair guidance; wherein, the information elements of the Prompt template include: textual knowledge information, key visual evidence information, key auditory evidence information, and information on answering user questions; and use the textual repair guidance and structured repair guidance as repair guidance schemes for household appliances.

[0107] The aforementioned device also includes: a unified vector database update module, used to obtain repair reports of completed repairs of household appliances; the repair report includes structured feedback and unstructured feedback, the structured feedback is used for repair technicians to confirm repair-related issues through multiple-choice questions, and the unstructured feedback is used for repair technicians to input text through text boxes; based on the repair report, target cases marked by the repair technician are identified; wherein, target cases represent cases that have been successfully resolved and differ significantly from the existing unified vector database; the on-site situation information of the target cases, which are vectorized, and the repair guidance plan of the target cases are added to the unified vector database; the association weights in the unified vector database are adjusted based on the repair report; if the repair report indicates that the target repair guidance plan is an incorrect repair guidance plan, the knowledge fragments associated with the target repair guidance plan in the unified vector database are labeled with low confidence tags.

[0108] The aforementioned unified vector database update module is also used by the text big model to generate structured feedback based on the video, audio and / or text information involved in this maintenance; the text big model evaluates the results of this maintenance based on the maintenance technician's evaluation, and learns the model based on the evaluation results.

[0109] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described household appliance repair guidance device can be referred to the corresponding process in the embodiments of the aforementioned household appliance repair guidance method, and will not be repeated here.

[0110] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0111] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0113] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for providing repair guidance for household appliances, characterized in that, The method involves pre-constructing a unified vector database based on text, image, and audio data related to household appliance malfunctions. Obtain on-site situation information of household appliances, including video information, audio information, and / or text information; perform image matching, audio matching, and / or text matching on the on-site situation information based on the unified vector database, and output knowledge fragments as search results; The search results are input into a large language model to output a repair guidance plan for the home appliance. The steps of performing image matching, audio matching, and text matching on the scene information based on the unified vector database, and outputting knowledge fragments as retrieval results, include: determining query image vectors, query audio vectors, and query text vectors based on the scene information; performing image matching on the query image vectors based on the unified vector database to determine the most similar visual scene as the knowledge fragment; performing audio matching on the query audio vectors based on the unified vector database to determine the most similar sound as the knowledge fragment; performing text matching on the query text vectors based on the unified vector database to determine the most similar problem description as the knowledge fragment; determining the association weights of multiple knowledge fragments, and sorting the multiple knowledge fragments based on the association weights to obtain retrieval results; The step of sorting multiple knowledge segments based on the association weights includes: determining the signal quality of the multiple knowledge segments; adjusting the association weights of the multiple knowledge segments based on the signal quality; and sorting the multiple knowledge segments based on the adjusted association weights. The method further includes: obtaining a repair report of the completed repair of the household appliance; determining a target case marked by the repair technician based on the repair report; wherein the target case represents a case that has been successfully resolved and differs significantly from the existing unified vector database; adding the on-site situation information of the target case (which has been feature vectorized) and the repair guidance plan for the target case to the unified vector database; and adjusting the association weights in the unified vector database based on the repair report. If the repair report indicates that the target repair guidance plan is an incorrect repair guidance plan, the knowledge fragments associated with the target repair guidance plan in the unified vector database are labeled with low confidence.

2. The method according to claim 1, characterized in that, The method further includes: The text vectors corresponding to the text data are extracted by the large text model, the image vectors corresponding to the image data are extracted by the visual feature extraction model, and the audio vectors corresponding to the audio data are extracted by the audio recognition model. A unified vector database is constructed based on the text vector, the image vector, and the audio vector.

3. The method according to claim 2, characterized in that, The steps for extracting text vectors from text data using a large text model include: semantically slicing the text data using a large text model to obtain text vectors; The steps of extracting image vectors corresponding to image data through a visual feature extraction model include: determining the visual information of the image data encoded by the model as an image vector through visual features; The step of extracting the audio vector corresponding to the audio data through an audio recognition model includes: extracting the spectral features of the audio data through an audio recognition model to obtain the audio vector corresponding to the audio data.

4. The method according to claim 2, characterized in that, The unified vector database includes: the text vector, the image vector, and the audio vector; the text data, the image data, and the audio data; and metadata. The metadata includes: device model, component name, data source link, and associated fault tags.

5. The method according to claim 1, characterized in that, The steps of inputting the search results into a large language model and outputting a repair guidance plan for the home appliance include: The search results are input into a large language model to output textual instructions for the repair of the home appliances; The knowledge fragments are injected into the Prompt template; the Prompt template is input into a large language model to output structured maintenance guidance; wherein, the information elements of the Prompt template include: textual knowledge information, key visual evidence information, key auditory evidence information, and information on answering user questions; The written repair instructions and the structured repair instructions will be used as the repair guidance scheme for the household appliances.

6. The method according to any one of claims 1-5, characterized in that, The repair report includes structured feedback and unstructured feedback. The structured feedback is used to allow repair technicians to confirm repair-related issues through multiple-choice questions, while the unstructured feedback is used to allow repair technicians to input text through text boxes.

7. The method according to claim 6, characterized in that, The method further includes: The text-based big data model generates the structured feedback based on the video, audio, and / or text information involved in this maintenance. The large text model evaluates the repair results based on the repair technician's assessment and then learns from the evaluation results.

8. A repair guidance device for household appliances, characterized in that, The device includes a pre-constructed unified vector database based on text, image, and audio data related to household appliance malfunctions. The retrieval result output module is used to obtain on-site situation information of household appliances, including video information, audio information and / or text information; based on the unified vector database, image matching, audio matching and / or text matching are performed on the on-site situation information, and knowledge fragments are output as retrieval results; The repair guidance solution generation module is used to input the search results into the large language model and output the repair guidance solution for the household appliance; The retrieval result output module is used to determine query image vectors, query audio vectors, and query text vectors based on the on-site situation information; perform image matching on the query image vectors based on the unified vector database to determine the most similar visual scene as the knowledge fragment; perform audio matching on the query audio vectors based on the unified vector database to determine the most similar sound as the knowledge fragment; perform text matching on the query text vectors based on the unified vector database to determine the most similar question description as the knowledge fragment; determine the association weights of multiple knowledge fragments, and sort the multiple knowledge fragments based on the association weights to obtain the retrieval results; The retrieval result output module is used to determine the signal quality of multiple knowledge segments; adjust the association weights of the multiple knowledge segments based on the signal quality; and sort the multiple knowledge segments based on the adjusted association weights. The device further includes: a unified vector database update module, used to obtain a repair report of the completed repair of the home appliance; determine target cases marked by the repair technician based on the repair report; wherein the target cases represent cases that have been successfully resolved and differ significantly from the existing unified vector database; add the on-site situation information of the target cases (which have been feature-vectorized) and the repair guidance plan for the target cases to the unified vector database; and adjust the association weights in the unified vector database based on the repair report; If the repair report indicates that the target repair guidance plan is an incorrect repair guidance plan, the knowledge fragments associated with the target repair guidance plan in the unified vector database are labeled with low confidence.

Images