After-sales installation quality detection method and device, electronic equipment and storage medium

Abstract

The invention provides an after-sales installation quality detection method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: inputting a product image after the after-sales installation into an image recognition model, and obtaining image recognition information; the image identification information comprises an identification result after identifying at least one target area in the product image, and the target area is determined according to a key point for performing quality detection on after-sales installation; and judging whether the after-sales installation is abnormal or not based on the image identification information and the business logic of the after-sales installation. According to the invention, a real-time after-sales installation quality detection mechanism and an after-sales installation image recognition model are constructed through an artificial intelligence visual technology, more accurate after-sales installation scene supervision is realized through a machine learning mode, real-time early warning of violation conditions is facilitated, timely tracing and rectification of problem files are promoted, and installation quality improvement is reversely promoted.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an after-sales installation quality detection method, device, electronic equipment and storage medium. Background Art

[0002] Due to the lack of supervision and factors such as staff mobility, the quality of after-sales installation services varies greatly, which in turn easily leads to product quality and customer complaints. Therefore, how to effectively supervise after-sales installation services has always been a key issue in the industry, and there is currently no good solution.

[0003] The reason is that under the existing technology, the after-sales installation service of home appliances lacks efficient quality inspection methods. The diversity of installation environments and the differences in the operations of installation masters lead to uneven installation quality, which makes it difficult to conduct comprehensive and timely inspections through traditional manual methods.

[0004] Therefore, a method is needed to automatically, quickly and accurately detect the after-sales installation quality, so as to solve the problems of low efficiency and insufficient accuracy of installation quality detection in the prior art. Summary of the invention

[0005] The present invention provides an after-sales installation quality detection method, device, electronic equipment and storage medium, which are used to solve the defects of low detection accuracy, large workload and poor real-time performance existing in the current manual detection.

[0006] The present invention provides an after-sales installation quality detection method, comprising the following steps: Inputting the product image after after-sales installation into the image recognition model, and obtaining image recognition information output by the image recognition model; the image recognition information includes a recognition result after recognizing at least one target area in the product image, and the target area is determined according to the key points of quality inspection of the after-sales installation; Based on the image recognition information and the business logic of the after-sales installation, determining whether the after-sales installation is abnormal; The image recognition model is obtained after training using image samples and identification information labels of each image sample.

[0007] According to an after-sales installation quality detection method provided by the present invention, the image recognition model includes an input layer, a feature extraction layer, an attention mechanism module, a dynamic sparse attention module and a classification layer; The step of inputting the product image after after-sales installation into the image recognition model and obtaining the image recognition information output by the image recognition model comprises: Input the product image to the input layer included in the image recognition model to perform image size and format conversion and normalization processing on the product image, and obtain a pre-processed standard image to input to the feature extraction layer; The convolution layer and the pooling layer in the feature extraction layer extract the image feature information in the standard image and input it into the attention mechanism module; The attention mechanism module calculates the attention weight of each feature point according to the feature map of the image feature information and the key points of the after-sales installation quality inspection; performs weighted summation on the feature map according to the calculated attention weight, obtains the weighted feature map and inputs it into the dynamic sparse attention module; The dynamic sparse attention module selects some key feature points from the weighted feature map, and generates target feature information related to the product image based on all the key feature points; The classification layer maps the target feature information to each classification label to obtain the image recognition information.

[0008] According to an after-sales installation quality detection method provided by the present invention, the image recognition model is trained based on the following method: Collecting historical after-sales installation image samples, and marking each image sample with the identification information label, wherein the identification information label includes at least one of position information and category information of key points in each target area on the image sample; Acquire text information on each of the image samples; fusing image information related to any image sample with the text information into multimodal information; Inputting any one of the image samples and the corresponding multimodal information into the initial model, and calculating the prediction loss of the initial model according to a comparison result between the recognition information result output by the initial model and the recognition information label corresponding to any one of the image samples; The model parameters of the initial model are iteratively updated according to the prediction loss calculated for each of the image samples until the training results converge, thereby obtaining the image recognition model.

[0009] According to an after-sales installation quality detection method provided by the present invention, the obtaining of text information on each of the image samples includes: Extracting text descriptions related to after-sales installation from each image sample; Extracting semantically relevant core words from the text description; Key content related to quality inspection of the after-sales installation is extracted from the core text as the text information.

[0010] According to an after-sales installation quality detection method provided by the present invention, the image information related to any image sample and the text information are fused into multimodal information, including: Inputting the image information and the text information into an image-text matching model to embed the image information and the text information into a feature vector, wherein the feature vector is used to represent the multimodal information; The image-text matching model is constructed based on the self-attention mechanism of the Transformer model.

[0011] According to an after-sales installation quality inspection method provided by the present invention, the identification information label of any of the image samples is marked in the following manner: Inputting any of the image samples into a visual language model, and obtaining target inspection items output by the visual language model and corresponding to key points of quality inspection for after-sales installation; According to the target inspection item, determining a target rectangular frame for pre-marking the target object of the image sample, wherein the target rectangular frame is used to characterize the position and size of the target object in the image sample; Determine the automatic annotation label according to the target rectangular frame; The identification information label is generated according to the verification result of the automatic marking label.

[0012] According to a method for detecting after-sales installation quality provided by the present invention, after collecting image samples after the completion of historical after-sales installation, it also includes preprocessing all the image samples, and the preprocessing specifically includes: After performing image cleaning on all the image samples, performing an enhancement operation on each obtained image sample, wherein the enhancement operation includes one or more of rotation, scaling, random cropping, and color gamut transformation; A blur evaluation is performed on at least one image sample obtained after the enhancement operation, so as to perform a blur enhancement operation on the image sample according to the evaluation result, wherein the blur enhancement operation includes one or more of adding noise, adjusting contrast, image degradation, image filtering, and image interpolation.

[0013] According to an after-sales installation quality inspection method provided by the present invention, when the key point of the after-sales installation quality inspection is the metal mixing valve inspection, judging whether the after-sales installation is abnormal based on the image recognition information and the business logic of the after-sales installation includes: When it is determined according to the image recognition information that a round water mixing valve or a plastic water mixing valve is detected from the product image, it is determined that the after-sales installation is normal, otherwise the next judgment is performed; When it is determined that the water heater is not detected from the product image, it is determined that the after-sales installation is normal, otherwise, the next judgment is performed; When it is determined that no metal water mixing valve is detected from the product image, it is determined that the after-sales installation is normal, otherwise, the next judgment is performed; When it is determined that the intersection-over-union ratio between the area where the metal water mixing valve detected from the product image is located and the product image meets a preset threshold, it is determined that the after-sales installation is normal; otherwise, it is determined that the after-sales installation is abnormal.

[0014] The present invention also provides an after-sales installation quality detection device, comprising: An after-sales installation image analysis unit, used to input the product image after after-sales installation into the image recognition model, and obtain image recognition information output by the image recognition model; the image recognition information includes a recognition result after recognizing at least one target area in the product image, and the target area is determined according to the key points of the quality inspection of the after-sales installation; an after-sales installation quality judgment unit, configured to judge whether the after-sales installation is abnormal based on the image recognition information and the business logic of the after-sales installation; The image recognition model is obtained after training using image samples and identification information labels of each image sample.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the after-sales installation quality detection method as described in any one of the above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the after-sales installation quality detection method as described in any one of the above is implemented.

[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the after-sales installation quality detection method as described in any one of the above is implemented.

[0018] The after-sales installation quality detection method, device, electronic device and storage medium provided by the present invention construct a real-time after-sales installation quality detection mechanism and a training method for an after-sales installation image recognition model through AI visual technology, thereby achieving more accurate supervision of after-sales installation scenarios, facilitating real-time early warning of violations, promoting timely tracing and rectification of problem files, and reversely promoting improvement of installation quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flow chart of the after-sales installation quality detection method provided by the present invention.

[0021] Figure 2 It is a schematic diagram of model training based on multimodal information of images and texts provided by the present invention.

[0022] Figure 3 It is a schematic diagram provided by the present invention for determining target inspection items using a visual language model to achieve accurate labeling of target objects.

[0023] Figure 4 It is a flowchart of judging whether the after-sales installation is abnormal based on the image recognition information and the business logic of the after-sales installation provided by the present invention.

[0024] Figure 5 It is a schematic diagram of the architecture of the after-sales installation quality inspection platform provided by the present invention.

[0025] Figure 6 It is a structural schematic diagram of the after-sales installation quality inspection device provided by the present invention.

[0026] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] It should be noted that in the description of the present invention, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0029] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0030] In the home appliance industry, the quality of after-sales installation services has always been a common focus of attention for consumers and companies. With the intelligent and diversified development of home appliance products, the complexity and professionalism of after-sales installation services are also constantly improving. However, the existing installation services in the home appliance industry generally lack effective supervision, and the mobility of installation engineers is high, resulting in uneven quality of after-sales installation services. Especially for products with strong installation attributes, such as water heaters, air conditioners, etc., the installation quality directly affects the use effect and safety of the product. Once improperly installed, it is easy to cause quality problems and customer complaints, causing unnecessary economic losses and brand damage to the company.

[0031] For example, a business unit with multiple products with strong installation attributes has a huge number of installation files per year, reaching tens of millions. However, the lack of effective management and control measures has made the closed-loop chain of rectification of installation problems long and the rectification efficiency low. Specifically, the existing after-sales installation quality inspection has the following main problems: (1) The detection scenarios are complex and changeable: After-sales scenarios include multiple categories and multiple detection items. There are also many business rules for the same detection item, and the detection rules are complex and difficult to distinguish. In addition, the installation environment varies greatly, the background is complex, and the installation master takes pictures at will, resulting in uncertain image shooting angles, variable sizes, uneven time and space distribution, dense occlusion and congestion, and imaging that changes at all times and has poor quality. These factors have greatly increased the difficulty of overall recognition and put forward very high requirements on the generalization ability of the image recognition model.

[0032] (2) Low efficiency of manual inspection: With more than 10 million files installed each year, manual inspection is difficult to cover all files, takes a long time to complete, and is prone to misjudgment and omission due to fatigue. In addition, the standards for manual inspection vary from person to person, and different inspection results are likely to be generated for complex scenarios and complex rules. At the same time, manual inspection also involves complex operations between multiple systems and multiple stakeholders, resulting in poor user experience and inability to achieve efficient inspection.

[0033] (3) Low data utilization efficiency: Manual inspections have a lag in discovering problems, and cannot provide real-time warnings and urge the installation master to make on-site corrections. Users are often required to make new appointments, resulting in a long installation problem rectification process. In addition, since the data from manual inspections cannot be fed back and used in real time, the data utilization efficiency is low and cannot provide strong support for the continuous improvement of installation quality.

[0034] In view of the above problems, the present invention proposes a new after-sales installation quality inspection method, which uses an image recognition model to identify the product image after the after-sales installation is completed, obtains image recognition information, and judges whether the after-sales installation is abnormal based on the image recognition information and the business logic of the after-sales installation. The method can realize fast and accurate inspection of the after-sales installation quality, improve the inspection efficiency and quality, and provide strong technical support for the after-sales installation service of home appliances.

[0035] Combine the following Figure 1-Figure 7 The present invention describes the after-sales installation quality detection method, device, electronic device and storage medium provided by the present invention.

[0036] Figure 1 It is a flow chart of the after-sales installation quality inspection method provided by the present invention, such as Figure 1 As shown, including but not limited to the following steps: Step 101: input the product image after after-sales installation into an image recognition model to obtain image recognition information output by the image recognition model.

[0037] The image recognition information includes a recognition result of at least one target area in the product image, and the target area is determined based on key points of quality inspection of the after-sales installation.

[0038] In the actual inspection process, the product images after after-sales installation are first obtained. These product images can be obtained by taking photos on site or from the storage system. Then, these product images are input into the pre-trained image recognition model in sequence. This image recognition model is built based on a deep learning algorithm. It can automatically identify the target area in the image and output the recognition results after image analysis of these target areas.

[0039] The target area is determined based on the key points of interest when conducting quality inspections on after-sales installations. For home appliance installation quality inspections, these key points directly guide the setting of the target area in the image recognition model.

[0040] For example, the key points of quality inspection for after-sales installation can be determined as: analysis of precision electrical connection status (such as cable joint quality inspection, insulation layer integrity inspection), complex mechanical structure assembly accuracy assessment (such as precision component position and posture detection, fastener preload force assessment, etc.), refrigerant circulation system and sealing inspection (such as pipeline system pressure and leakage detection, valve and interface sealing assessment, etc.).

[0041] Specifically, with regard to the quality inspection of cable joints, the present invention sets it as the key point of quality inspection. It not only pays attention to the physical connection status of the cable, such as whether it is inserted in place and whether it is tightened, but also uses high-resolution image recognition technology and image recognition models to analyze the metal surface status at the cable joint (such as whether there is oxidation, corrosion, scratches), as well as the contact area and contact pressure distribution inside the joint, so as to evaluate the reliability and long-term stability of the electrical connection.

[0042] For insulation integrity testing, image recognition models can be used to accurately identify tiny signs of damage, wear or aging of the cable insulation layer using edge detection and color analysis to ensure the safety of the electrical system.

[0043] For the position and posture detection of precision components, for key components such as compressors and motors, the 3D reconstruction technology and shape matching algorithm in image recognition are used to accurately measure their position deviation, tilt angle and rotation direction during the installation process to ensure the assembly accuracy of the mechanical structure.

[0044] For the assessment of fastener preload, image recognition and mechanical models can be combined to identify the degree of deformation of fasteners (such as bolts and nuts) and the stress distribution of surrounding structures, and indirectly assess whether the preload applied by the fasteners meets the design requirements, thereby ensuring the reliability and durability of the mechanical connection.

[0045] In addition, for pipeline system pressure and leakage detection, pressure sensors and infrared thermal imaging technology assisted by image recognition technology can be used to monitor the pressure changes and temperature distribution of the pipeline system in real time. Combined with machine learning algorithms, potential refrigerant leakage points can be quickly identified to ensure the efficiency and safety of the refrigeration system.

[0046] For the evaluation of the sealing of valves and interfaces, texture analysis and deep learning algorithms in image recognition can be used to accurately identify the sealing surface status of valves and interfaces (such as whether there are scratches, wear, corrosion), as well as the integrity of sealing materials (such as O-rings), thereby evaluating the sealing performance of the system.

[0047] The present invention predetermines the key points for quality inspection of the after-sales installation, and then uses a training model to focus on image recognition of target areas related to these key points. It can accurately capture and evaluate the inspection key points in the after-sales installation process, and provide strong technical support for real-time monitoring and quality control of home appliance installation quality.

[0048] Step 102: Based on the image recognition information and the business logic of the after-sales installation, determine whether the after-sales installation is abnormal.

[0049] The image recognition model is obtained after training using image samples and identification information labels of each image sample.

[0050] The image recognition information output by the image recognition model is used to combine with the business logic of after-sales installation. The business logic can be formulated according to the installation specifications and after-sales service standards of home appliances, for example, defining under what circumstances after-sales installation is considered abnormal.

[0051] In the comprehensive evaluation process, the image recognition information is aimed at the recognition results of the target area, such as the power cord connection status, screw tightening degree, interface docking status, etc. The recognition results are deeply analyzed and compared with the installation specifications, safety standards, performance requirements, etc. defined in the business logic. At the same time, considering the complexity and diversity of the actual installation environment and the differences between different product models, the business logic also includes flexible processing rules and decision-making mechanisms to deal with various special situations that may arise.

[0052] By combining image recognition information and business logic for comprehensive evaluation, the present invention can achieve a comprehensive, detailed and accurate judgment of the quality of after-sales installation. If an abnormality or non-compliance with the specification is found in the installation, an abnormality report or reminder can be generated in time so that the installation engineer or after-sales service team can check, repair or adjust it, thereby ensuring the installation quality and user satisfaction of home appliances.

[0053] The after-sales installation quality inspection method provided by the present invention constructs a real-time after-sales installation quality inspection mechanism and an after-sales installation image recognition model through artificial intelligence visual technology, so as to achieve more accurate supervision of after-sales installation scenarios through machine learning, facilitate real-time warning of violations, promote timely tracing and rectification of problem files, and reversely promote the improvement of installation quality.

[0054] Based on the content of the above embodiment, as an optional embodiment, the image recognition model includes an input layer, a feature extraction layer, an attention mechanism module, a dynamic sparse attention module and a classification layer; The step of inputting the product image after after-sales installation into the image recognition model and obtaining the image recognition information output by the image recognition model comprises: Input the product image to the input layer of the image recognition model to perform image size and format conversion and normalization processing on the product image, and obtain a pre-processed standard image to input into the feature extraction layer; The convolution layer and the pooling layer in the feature extraction layer extract the image feature information in the standard image and input it into the attention mechanism module; The attention mechanism module calculates the attention weight of each feature point according to the feature map of the image feature information and the key points of the after-sales installation quality inspection; performs weighted summation on the feature map according to the calculated attention weight, obtains the weighted feature map and inputs it into the dynamic sparse attention module; The dynamic sparse attention module selects some key feature points from the weighted feature map, and generates target feature information related to the product image based on all the key feature points; The classification layer maps the target feature information to each classification label to obtain the image recognition information.

[0055] The after-sales installation quality detection method provided by the present invention is a detection method based on an image recognition model. By using the attention mechanism and the dynamic sparse attention layer, it can automatically focus on the key areas in the image (such as the electric water heater, the mixing valve, the nozzle, the pipeline, etc.). The structure of the image recognition model is complex and efficient, and is particularly suitable for the after-sales installation quality detection of household appliances. The following is a detailed explanation of the image recognition model and its workflow.

[0056] The image recognition model can be one of the following deep learning models or a combination of them: a convolutional neural network, a network model with an attention mechanism, a Transformer model, or a lightweight model (such as MobileNet, yolov8, and other network models).

[0057] The input layer of the image recognition model preprocesses the input product image, including image resizing, format conversion, and normalization, to ensure that the processed product image meets the requirements of subsequent processing steps within the model and improve the processing efficiency and accuracy of the image recognition model.

[0058] The feature extraction layer contains multiple convolutional layers and pooling layers, which are used to extract image feature information from the product image preprocessed by the input layer. The convolutional layer extracts local features in the product image through convolution operations, and the pooling layer downsamples the feature map output by the convolutional layer to reduce the amount of calculation and extract the main image feature information.

[0059] The attention mechanism module is mainly used to determine the attention weight of each feature point based on the feature map of image feature information by calculating the local and global relationship of the feature map, and generate a weighted feature map, so that the model can focus on the feature points that are more critical to after-sales installation quality inspection and improve detection accuracy.

[0060] The dynamic sparse attention module is mainly used to select some key feature points from the weighted feature map, select sparse key feature points according to the attention weight, and construct the target feature information. Its role in the model is to further reduce redundant information and improve the model's sensitivity to key features.

[0061] The classification layer determines the quality of after-sales installation based on the target feature information and gives the corresponding classification results, that is, it maps the target feature information to each classification label and outputs the image recognition information.

[0062] Assume that it is necessary to conduct quality inspection on the washing machines it produces after installation to ensure that the installation quality meets the standards. The specific inspection process includes: Collect a large number of image samples of washing machines after after-sales installation, including those collected under normal installation and abnormal installation conditions.

[0063] The above image sample data is used to train the image recognition model, including the input layer, feature extraction layer, attention mechanism module, dynamic sparse attention module and classification layer. During the training process, the model parameters are continuously adjusted to optimize the model performance so that the model can accurately identify the quality of the washing machine installation.

[0064] Finally, the trained image recognition model is deployed to the after-sales service system for actual quality inspection. After-sales service personnel use a camera to take images of the washing machine after after-sales installation, and upload them to the image recognition model to obtain the corresponding image recognition results. Then, based on the image recognition results and the business logic requirements of the after-sales installation of the washing machine, it is possible to accurately determine whether the after-sales installation quality of the washing machine meets the standards, and give corresponding treatment suggestions.

[0065] The after-sales installation quality detection method provided by the present invention utilizes an attention mechanism module and a dynamic sparse attention module, so that the obtained image recognition model can automatically identify and focus on the key target areas in the product image that are most relevant to the key points of after-sales installation quality detection. This means that when processing images, the image recognition model can more accurately capture those details that are crucial to judging the installation quality, such as the connection status of the power cord, the tightness of the screws, etc. This improvement can significantly improve the model's recognition accuracy for after-sales installation quality and reduce misjudgments and missed judgments.

[0066] In practical applications, images of after-sales installation often have problems such as complex backgrounds, different lighting conditions, and various shooting angles. The attention mechanism module and the dynamic sparse attention module can help the image recognition model filter out key information from these complex factors, thereby enhancing the robustness of the model. In this way, even when faced with product images of different quality and from different sources, the model can stably output accurate recognition results.

[0067] In addition, the dynamic sparse attention module generates target feature information by selecting key feature points, which is actually a sparse representation of image feature information. This means that when processing product images, the image recognition model only needs to focus on those feature points that have an important impact on the recognition results, without having to process all the features of the entire product image. This can not only reduce the amount of computation of the image recognition model and improve computational efficiency, but also reduce the memory usage of the model, making it more suitable for running in resource-constrained environments.

[0068] Based on the content of the above embodiment, as an optional embodiment, the image recognition model is trained based on the following method: Collecting historical after-sales installation image samples, and marking each image sample with the identification information label, wherein the identification information label includes at least one of position information and category information of key points in each target area on the image sample; Acquire text information on each of the image samples; fusing image information related to any image sample with the text information into multimodal information; Inputting any one of the image samples and the corresponding multimodal information into the initial model, and calculating the prediction loss of the initial model according to a comparison result between the recognition information result output by the initial model and the recognition information label corresponding to any one of the image samples; The model parameters of the initial model are iteratively updated according to the prediction loss calculated for each of the image samples until the training results converge, thereby obtaining the image recognition model.

[0069] Figure 2is a schematic diagram of model training based on multimodal information of images and texts provided by the present invention, such as Figure 2 As shown, the present invention provides a method for after-sales installation quality inspection. When training the image recognition model, a multimodal information fusion method is used to combine image information and related text information (such as descriptions in after-sales archives) to enhance the trained image recognition model's ability to understand the after-sales installation scene of home appliances. In this way, the model can more accurately identify key elements in the image, such as the whole machine, mixing valve, sprinkler, etc., and determine whether their installation position and status meet the specifications.

[0070] First, image features are extracted from product images. These image features can describe key visual elements in the image, such as water heaters, metal mixing valves, sprinklers, etc. At the same time, text features are extracted from related text descriptions. These text features can contain relevant information such as product models and installation requirements.

[0071] The extracted image features and text features are enhanced. This step can generally be done by standardizing or normalizing the features or by using a specific algorithm to enhance the expressiveness of the features. Then, the enhanced image features and text features are fused into multimodal information. This step utilizes a cross-modal decoder that can process and integrate information from different modalities. The fused multimodal information is input into the image recognition model to obtain the recognition results output by the image recognition model, including the location and category information of key targets in the product image.

[0072] As an optional embodiment, the model can also be trained through multimodal information, mainly including: Collect image samples after the completion of historical after-sales installation, and annotate each image sample with identification information labels, including the location information and category information of key points. At the same time, obtain text information related to the image sample, such as installation specifications, product models, etc.

[0073] Image features are extracted from image samples, and text features are extracted from text information. The extracted features are enhanced to improve the recognition ability of the model. Image features and text features are fused to form multimodal information to provide richer input data for the model. The fused multimodal information is input into the initial model, and the prediction loss is calculated based on the comparison between the recognition information results output by the model and the actual recognition information labels. Then, the model parameters can be iteratively updated according to the prediction loss until the training results converge. The trained model is used to perform quality inspection on new after-sales installation images, output key target information, and make a comprehensive judgment based on the business logic to output installation abnormality information.

[0074] Figure 3is a schematic diagram of using a visual language model to determine target inspection items for achieving accurate labeling of target objects, as provided by the present invention. Figure 3 As shown in the figure, for the collected image samples, the corresponding image features can be extracted by inputting them into the image recognition model to be trained. At the same time, text extraction technology is used to obtain text information related to each image sample (such as leak detector, power meter, mixing valve, etc.). The text information may include descriptions in after-sales archives, which record in detail the installation process, installation conditions, feedback from installation engineers and other related information of home appliances. This text information provides additional context and semantic understanding for the image recognition model, which helps the model to understand the content and scenes in the image more accurately.

[0075] The present invention adopts a multimodal information fusion method to combine image information and related text information to enhance the ability of the trained image recognition model to understand the after-sales installation scenario of home appliances.

[0076] By fusing these two types of information, the model can make full use of the visual features in the image and the semantic information in the text to achieve a more comprehensive and accurate understanding of the after-sales installation scenario. This multimodal information fusion method helps to improve the robustness and generalization ability of the image recognition model, enabling it to better adapt to different after-sales installation scenarios and conditions.

[0077] Assume that there is an image sample of an electric water heater installation scene, which contains key components such as the electric water heater, mixing valve, and sprinkler. First, label the image sample with identification information, including the location and category of each key component.

[0078] Then, we obtain text information related to this image sample, such as "Electric water heater model: XXX; Installation location: Bathroom". Next, we enhance the image features and text features and fuse them into multimodal information. This multimodal information is input into the model. After the model outputs the recognition result, we can calculate the prediction loss by comparing this result with the actual recognition information label.

[0079] Then, the model parameters of the initial model are iteratively updated according to the predicted loss until the updated model can accurately identify key components such as electric water heaters, mixing valves, and sprinklers in the image samples, and can identify whether the installation complies with the specifications. In this way, a trained image recognition model is obtained, which can be used for after-sales installation quality detection.

[0080] The after-sales installation quality inspection method provided by the present invention utilizes multimodal information fusion technology to enable the trained image recognition model to learn richer feature representations from data of different modalities, which helps to improve its generalization ability when processing complex tasks. For example, combining image information and text information can enable the image recognition model to understand the image content while also utilizing text information for more accurate recognition and classification.

[0081] It should be noted that the above-mentioned embodiments of the present invention only provide a multimodal information training method based on the fusion of image information and text information, and it does not exclude that the present invention can also adopt pairwise fusion or centralized fusion of multi-form information such as text, image, voice, video, etc.

[0082] Based on the content of the above embodiment, as an optional embodiment, the obtaining of text information on each of the image samples includes: Extracting text descriptions related to after-sales installation from each image sample; Extracting semantically relevant core words from the text description; Key content related to quality inspection of the after-sales installation is extracted from the core text as the text information.

[0083] The present invention provides a method for extracting text information from an image sample, which mainly includes: Image samples containing electric water heater installation scenes are obtained from after-sales systems, user uploads, or on-site shooting. The collected image samples can be preprocessed, such as denoising, contrast enhancement, etc., to improve the recognition rate of the text information contained therein.

[0084] Optical Character Recognition (OCR) can be used to convert text in images into editable text format. OCR is a technology that uses computers to automatically recognize and analyze text information in images. It is widely used in document digitization, automated data input, electronic archive management and other fields. It converts text information in image samples into computer-editable and processable text descriptions related to after-sales installation through a series of image processing and pattern recognition algorithms.

[0085] Furthermore, the Natural Language Processing (NLP) system can be used to perform word segmentation, part-of-speech tagging, and syntactic analysis on the text description to understand the meaning and structure of the text description. Furthermore, based on the text processed by the NLP system, the NLP system is used to perform semantic analysis to identify and extract core text. This can be done by using named entity recognition (NER) technology to identify entities such as product models and locations, and by using syntactic analysis to parse sentence structures, thereby extracting core text. The content of these core texts can include, but is not limited to, the model of the electric water heater, installation location, accessories list, etc.

[0086] Key information extraction algorithms can identify key information in text based on predefined rules or models. For example, regular expressions can be used to match specific text patterns, or machine learning models can be used to predict which parts of the text are key information.

[0087] Finally, the extracted key information can be stored in a structured manner to form a format that can be used for subsequent processing (such as JSON, XML, etc.).

[0088] Collect installation site images corresponding to the after-sales archives as image samples, and ensure that the image samples contain elements related to the text description (such as electric water heaters, mixing valves, etc.).

[0089] Associate the extracted key information (such as product model, installation location, accessories list) with the corresponding image sample. This can be achieved by building an index or database to ensure that each image sample can be matched with the corresponding text description.

[0090] Use multimodal learning technology (such as image-text matching model) for alignment transfer. This step involves fusing language information with image modality, so that the resulting image recognition model can understand and identify key elements in the image and their relationship with the text description, which not only improves the performance and accuracy of the model, but also optimizes the user experience and improves decision-making efficiency. Through multimodal fusion technology, the model can understand and process information more comprehensively, thus playing a greater role in practical applications.

[0091] Based on the content of the above embodiment, as an optional embodiment, fusing the image information related to any image sample with the text information into multimodal information includes: Inputting the image information and the text information into an image-text matching model to embed the image information and the text information into a feature vector, wherein the feature vector is used to represent the multimodal information; The image-text matching model is constructed based on the self-attention mechanism of the Transformer model.

[0092] Aiming at the problem of complex elements in product images related to after-sales installation, the present invention integrates after-sales archive language with image modality through image language representation learning and alignment transfer in the data pre-labeling stage, and improves the target detection and result segmentation capabilities of complex imaging by combining with a large model. Taking an electric water heater as an example, the whole machine, mixing valve, sprinkler, and pipeline in a complex installation scene can be accurately segmented.

[0093] Among them, through image language representation learning and alignment transfer, the fusion of after-sales archive language and image modality can adopt the self-attention mechanism based on Transformer, and embed the information of two modalities, image information and text information, into a common feature vector for feature fusion and extraction.

[0094] As an optional embodiment, in the training process of the above-mentioned image recognition model, a key step is to fuse the image information and text information into multimodal information. This process is not simply to splice the two types of information together, but to embed them into a common feature vector through a specific image-text matching model. The obtained feature vector can fully and accurately represent the multimodal information, thereby providing strong support for subsequent image recognition tasks.

[0095] The core task of the image-text matching model is to map the two different modalities of information, image information and text information, into a common feature space, so that similar image and text pairs have similar feature representations in this space. In this way, the model can judge the relevance between images and texts based on the similarity of feature vectors, thereby achieving effective fusion of multimodal information.

[0096] In order to achieve effective matching and fusion of image and text information, the image-text matching model of the present invention can adopt a self-attention mechanism based on the Transformer model.

[0097] The Transformer model is a powerful sequence modeling tool that uses a self-attention mechanism to capture the dependencies between different positions in a sequence. In the image-text matching model, the self-attention mechanism is used to process image features and text features, allowing the model to capture the potential connection between images and text.

[0098] Specifically, image features can be extracted through models such as convolutional neural networks (CNN), while text features can be extracted through models such as recurrent neural networks (RNN), long short-term memory networks (LSTM) or Transformer. These features are then input into the image-text matching model, which uses a self-attention mechanism to calculate the similarity between image features and text features and fuse them into a common feature vector.

[0099] After being processed by the image-text matching model, the obtained feature vector has a strong representation ability. It can not only contain key information in the image (such as the location, shape, color, etc. of key points in the target area), but also text information related to the image (such as installation instructions, fault reports, user feedback, etc.). This fusion of multimodal information enables the model to understand the image content more accurately in subsequent image recognition tasks, improving the accuracy and robustness of recognition.

[0100] Taking the after-sales installation quality inspection of refrigerators in the home appliance industry as an example, product images (such as photos of the refrigerator after installation) and text information (such as installation instructions and user feedback) can be input into the image-text matching model. The model uses the Transformer-based self-attention mechanism to calculate the similarity between image features and text features and fuse them into a common feature vector. This feature vector is then input into the subsequent image recognition model to determine whether the installation quality of each component of the refrigerator meets the standards.

[0101] In this way, we can not only make full use of the two different modalities of information, image information and text information, but also improve the accuracy and robustness of the image recognition model. This is of great significance to home appliance companies because it can help companies better control product quality, improve after-sales service quality, and thus increase customer satisfaction.

[0102] Based on the content of the above embodiment, as an optional embodiment, the identification information label of any of the image samples is marked in the following manner: Inputting any of the image samples into a visual language model, and obtaining target inspection items output by the visual language model and corresponding to key points of quality inspection for after-sales installation; According to the target inspection item, determining a target rectangular frame for pre-marking the target object of the image sample, wherein the target rectangular frame is used to characterize the position and size of the target object in the image sample; Determine the automatic annotation label according to the target rectangular frame; The identification information label is generated according to the verification result of the automatic marking label.

[0103] The present invention provides a process for automatically generating and verifying image sample identification information labels in after-sales installation quality inspection. Aiming at the problem of large demand for image samples in model training, intelligent pre-labeling can be used to greatly reduce the labeling cost, shortening the single labeling frame from 30 seconds to 0.3 seconds per frame, reducing the labeling cost by a hundred times, making the project feasible for promotion.

[0104] First, the image samples to be processed (which may be photos or video frames of the product installation site) are input into a pre-trained visual language model, which can understand and parse the visual information in the image samples and associate it with language descriptions.

[0105] Based on the content of the input image sample, the visual language model outputs target inspection items corresponding to key points related to after-sales installation quality inspection. These key points may be the completion of the installation steps, the integrity of the parts, the accuracy of the installation position, etc. The visual language model converts these key points into specific target inspection items as the basis for subsequent annotation.

[0106] According to the target inspection items output by the visual language model, determine which objects (such as parts, installation locations, etc.) in the image sample are the focus of inspection. Then, use computer vision technology (such as edge detection, shape matching, etc.) to automatically draw target rectangles of these target objects in the image sample. These target rectangles are used to accurately characterize the position and size of the target objects in the image.

[0107] After the target rectangular boxes are determined, one or more pre-labeled labels can be preliminarily assigned to the objects within these target rectangular boxes. These pre-labeled labels are generated based on the target inspection items and the visual language model's understanding of the image information of the image samples.

[0108] Based on the target rectangle and pre-labeled labels, further processing is performed to generate more accurate automatic labeling labels. These automatic labeling labels may include the name, category, state (such as intact, damaged), location information (such as the position relative to a reference point), etc. of the object.

[0109] In order to improve the accuracy and reliability of labels, machine learning algorithms can also be used to optimize automatic labeling, for example by considering contextual information in the image, shape and color features of objects, etc.

[0110] After obtaining the automatic labeling label, it needs to be verified to ensure the accuracy and consistency of the automatic labeling label. The verification mainly includes manual inspection (professionals review the label) and / or automatic inspection (using another set of verification algorithms to verify the label). After the verification, the final identification information label can be generated based on the verification results. These identification information labels contain not only the basic information of the object, but also may contain specific information related to after-sales installation quality inspection (such as compliance of installation steps, integrity of parts, etc.).

[0111] As an optional embodiment, the above-mentioned visual language model can be a SAM large model. Taking the object of labeling a water heater as an example, the water heater is passed into the SAM large model as a prompt word. The SAM large model will segment the water heater part according to the input prompt word, process the segmented data to obtain the detected target rectangular frame, and obtain a large amount of pre-labeled data. Then, through manual confirmation, a data set of after-sales installation target detection objects is obtained (the data set is divided into training set, verification set, and test set in deep learning training. Its data representation is a combination of image samples and their corresponding labels. The content of the label indicates the category, coordinate position, etc. of the target object in the image sample. The purpose of constructing the data set is to guide model training). Finally, the yolov8 model is used to detect the key objects of the mixing valve, such as water heaters, round mixing valves, metal mixing valves, plastic mixing valves, etc.

[0112] The after-sales installation quality inspection method provided by the present invention can realize automatic pre-labeling of image samples through visual language models and computer vision technology, greatly reducing the workload of manual labeling. Automated labeling not only improves the labeling speed, but also can ensure the consistency of labeling to a certain extent, avoiding errors that may occur in manual labeling. The labeling process combines image information and text information, and through the fusion of multimodal information, it can more accurately identify key points and target objects in the image.

[0113] Based on the content of the above embodiment, as an optional embodiment, after collecting the image samples after the historical after-sales installation is completed, it also includes preprocessing all the image samples, and the preprocessing specifically includes: After performing image cleaning on all the image samples, performing an enhancement operation on each obtained image sample, wherein the enhancement operation includes one or more of rotation, scaling, random cropping, and color gamut transformation; The blurriness of at least one image sample obtained after the enhancement operation is evaluated, and a blur enhancement operation is performed on the image sample according to the evaluation result. The blur enhancement operation includes one or more of adding noise, adjusting contrast, image degradation, image filtering, and image interpolation.

[0114] In the after-sales installation quality inspection process, in order to improve the efficiency and accuracy of subsequent analysis, learning or quality control, these image samples after installation are first collected. These image samples not only record the actual status of the equipment installation, but also carry rich information, such as installation quality, environmental adaptability, etc.

[0115] However, image samples collected directly from the scene often have a variety of problems, such as inconsistent shooting angles, different lighting conditions, image blur, etc. These problems directly affect the effect of subsequent image analysis. Therefore, preprocessing these image samples has become an indispensable step.

[0116] The preprocessing process first includes image cleaning, which is a process of preliminary screening and optimization of the image. Specifically, the following operations can be performed: (1) Remove redundant information such as background and interference that are irrelevant to after-sales installation quality inspection from the image samples through image processing techniques such as image cropping and mask processing. At the same time, check and correct format errors in the image samples, such as image size mismatch and insufficient resolution. You can also adjust color distortion to ensure the authenticity and consistency of the image color in the image samples.

[0117] (2) Perform enhancement operations on each cleaned image sample. This step aims to improve the visual effect and recognition of the image through a series of technical means, including: 2.1) Rotate and scale image samples to simulate different viewing angles and size changes to enhance the generalization ability of the model.

[0118] 2.2) Randomly crop different areas in the image samples to increase the ability of subsequent models to recognize key information in the image.

[0119] 2.3) Adjust the color distribution of image samples, such as brightness, contrast, saturation, etc., to make them more in line with the needs of model analysis or model training.

[0120] (3) The image processing algorithm can be used to evaluate the blurriness of the image sample to determine the clarity level of the image sample. Then, based on the evaluation results, the image samples with higher blurriness can be enhanced, such as adding noise, adjusting contrast, image degradation (such as simulating lens stains, scratches, etc.), image filtering (deblurring filtering), image interpolation, etc., so that the image can present clearer and easier-to-recognize features while maintaining the original information, thereby improving the clarity and recognition of the image sample.

[0121] (4) Other image optimization schemes can also be adopted, such as using image processing tools such as filters to remove noise from image samples to improve the quality of image samples; sharpening image samples to enhance the clarity of image edges and details; standardizing and normalizing image samples to ensure the consistency of image data and improve the stability and accuracy of model training.

[0122] In summary, by systematically preprocessing the image samples after the historical after-sales installation, including image cleaning, enhancement operations, and blur evaluation and enhancement, the quality and usability of the images can be significantly improved, providing strong support for subsequent analysis, learning, or quality control. This process not only improves work efficiency, but also ensures the accuracy and reliability of the processing results.

[0123] Figure 4is a flow chart of judging whether the after-sales installation is abnormal based on the image recognition information and the business logic of the after-sales installation provided by the present invention, as an optional embodiment, such as Figure 4 As shown, when the key point of the quality inspection of the after-sales installation is the metal mixing valve inspection, judging whether the after-sales installation is abnormal based on the image recognition information and the business logic of the after-sales installation includes: When it is determined according to the image recognition information that a round water mixing valve or a plastic water mixing valve is detected from the product image, it is determined that the after-sales installation is normal, otherwise the next judgment is performed; When it is determined that the water heater is not detected from the product image, it is determined that the after-sales installation is normal, otherwise, the next judgment is performed; When it is determined that no metal water mixing valve is detected from the product image, it is determined that the after-sales installation is normal, otherwise, the next judgment is performed; When it is determined that the intersection-over-union ratio between the area where the metal water mixing valve detected from the product image is located and the product image meets a preset threshold, it is determined that the after-sales installation is normal; otherwise, it is determined that the after-sales installation is abnormal.

[0124] The present invention takes the after-sales installation quality inspection of a water heater as an example and provides a specific implementation method for determining whether the after-sales installation is abnormal based on the image recognition information and the business logic of the after-sales installation.

[0125] When conducting after-sales installation quality inspection of the water heater, other inspection items only need to be inspected for presence or absence, that is, only the image recognition information detected by the method provided in the above embodiment can be used for judgment. However, for the metal mixing valve, its inspection items are more complicated. The present invention designs a set of quality inspection methods combined with the business logic of the inspection items.

[0126] First, let's introduce the description of the business logic related to the following business requirements. The metal mixing valve is too close to the water heater, which causes leakage and leads to safety issues. Since there are few negative samples of images related to the metal mixing valve in this scene, the recognition accuracy of this detection item is low and cannot meet the quality detection requirements of this scene. In order to detect this type of installation quality problem, a special business logic is designed to solve this problem.

[0127] First, find all qualified contents during after-sales installation, and use the elimination method to find the contents that ultimately cause unqualified installation quality (excluding qualified contents is mainly to improve the accuracy of the algorithm, because when directly detecting metal mixing valves, there are fewer negative samples in the image, resulting in poor recognition accuracy. There are no requirements for installation specifications for round mixing valves and plastic mixing valves, and their recognition accuracy is higher).

[0128] Combination Figure 4As shown, during after-sales installation quality inspection, image recognition technology is first used to detect whether a round mixing valve or a plastic mixing valve exists in the product image.

[0129] If the image recognition information indicates that a round mixing valve or a plastic mixing valve is successfully detected from the product image, then the after-sales installation is considered normal according to the business logic. This is because round mixing valves or plastic mixing valves are acceptable choices in certain application scenarios and are not considered abnormal.

[0130] If the round mixing valve or the plastic mixing valve is not detected, the next determination step is continued, that is, whether there is a water heater in the product image is continued to be checked.

[0131] If the water heater is not detected in the image, according to the business logic, this is also considered as a normal after-sales installation. That is, since the detection object is not a water heater, there is no need to continue to judge whether the metal mixing valve is installed correctly. If the water heater is detected, the next determination step is performed, that is, checking whether a metal water mixing valve is not detected in the product image.

[0132] If no metal mixing valve is detected in the image, this is considered as a normal after-sales installation according to the business logic. This may be because the metal mixing valve is not required to be installed in certain situations, or it is replaced or omitted for some reason.

[0133] If a metal mixing valve is detected, the next more detailed judgment step is entered to judge the position accuracy of the detected metal mixing valve.

[0134] The intersection over union (IoU) between the area where the metal water mixing valve is located and the entire product image is calculated and compared with the preset threshold. If the IoU meets the preset threshold (the IoU value is greater than or equal to the preset threshold), it means that the metal water mixing valve is installed correctly and meets the requirements of the business logic, so the after-sales installation is determined to be normal. If the IoU does not meet the preset threshold (the IoU value is less than the preset threshold), it means that the metal water mixing valve is installed incorrectly, and there may be misalignment, offset or other abnormalities, so the after-sales installation is determined to be abnormal.

[0135] Among them, the IoU value between the area where the metal mixing valve is located and the entire product image is determined by calculating the intersection of the area where the metal mixing valve is located and the entire product image area, and the union of the area of ​​the entire product image plus the area of ​​the area where the metal mixing valve is located (if they do not completely overlap) minus the area of ​​the intersection area. The ratio between the above intersection and the union is determined.

[0136] The after-sales installation quality inspection method provided by the present invention can evaluate whether the installation position of the metal mixing valve is accurate by calculating the IoU between the area where the metal mixing valve is located and the entire product image area during the after-sales installation quality inspection. If the IoU value is high, it means that the position of the metal mixing valve has a high degree of overlap with the expected installation position and the installation position is accurate; conversely, if the IoU value is low, it may indicate that there is a deviation in the position of the metal mixing valve. IoU provides an objective and quantitative standard to evaluate the quality of the installation position. Compared with subjective judgment, the IoU value is more accurate and reliable, can reduce the influence of human factors, and improve the accuracy and consistency of the evaluation.

[0137] Figure 5 is a schematic diagram of the architecture of the after-sales installation quality inspection platform provided by the present invention, such as Figure 5 As shown, the entire platform includes multiple execution terminals, among which: Service engineers are mainly installation engineers who perform after-sales installation services. After completing the after-sales installation, they will take on-site photos of the installed products and upload the photographed product images to the company's switch system through the company's home service application.

[0138] The company's switch system (Compact Control Shelves, CCS) distributes the uploaded product images (installation engineers often call them order photos) to the AI ​​after-sales installation quality inspection system for real-time inspection.

[0139] A trained image recognition model is pre-loaded in the AI ​​after-sales installation quality inspection system, which is used to execute the after-sales installation quality inspection method provided by any of the above embodiments, and ultimately obtain information on whether each after-sales installation service is abnormal.

[0140] If after-sales installation service is confirmed to be abnormal, the CCS system will generate a corresponding installation file, and simultaneously feedback the abnormal installation problem to the on-site installation engineer in real time, and put forward rectification requirements, mainly through the front-end of the company's home service application to remind the installation of abnormalities. Abnormal information is pushed to relevant responsible persons (such as those related to categories, transportation, and outlets) in real time through various methods to ensure that the problem is handled in a timely manner.

[0141] The AI ​​after-sales installation quality inspection system will also analyze abnormal installation quality data in real time, deeply deconstruct and display quality abnormality data in different dimensions, and provide data support for improving the installation process and improving installation quality.

[0142] Furthermore, if the installation engineer refuses to make corrections, the service order that indicates that the after-sales installation does not meet the requirements and that no corrections have been made will be marked and recorded as an abnormal after-sales installation service order.

[0143] Furthermore, the AI ​​after-sales installation quality inspection system provides a multi-dimensional data dashboard, displaying data of concern to different user groups, such as installation quality, abnormality rate, etc.

[0144] In addition, the present invention can also use the product images collected each time as image samples for training, so that continuous iteration of massive data can be achieved, the image recognition model can be continuously optimized, and the accuracy and efficiency of detection can be improved.

[0145] The after-sales installation quality inspection platform provided by the present invention adopts an image recognition model for after-sales installation quality inspection, and the inspection accuracy can be increased to more than 95% in most cases. The real-time after-sales installation quality inspection method can effectively help improve the inspection efficiency by 97% (10s / file for manual inspection and 0.3s / file for AI after-sales system), truly realize the full-volume installation file automatic intelligent inspection and real-time early warning, promote rapid on-site rectification, and significantly shorten the closed-loop link of problem discovery-notification of rectification-on-site rectification-rectification completion.

[0146] Abnormal data from after-sales installation inspections can be intelligently generated into statistical reports based on multiple dimensions such as category, product model, inspection item, outlet, in-transit, number of violations, violation rate, etc., and pushed in a variety of ways. This can improve the visual monitoring and utilization of installation quality inspection results, and reversely promote the quality and service departments to improve after-sales quality.

[0147] Figure 6 Schematic diagram of the structure of the after-sales installation quality inspection device provided by the present invention. Figure 6 As shown, mainly including but not limited to: The after-sales installation image analysis unit 61 is used to input the product image after after-sales installation into the image recognition model, and obtain the image recognition information output by the image recognition model.

[0148] The image recognition information includes a recognition result of at least one target area in the product image, wherein the target area is determined based on key points of quality inspection for the after-sales installation; The after-sales installation quality judgment unit 62 is used to judge whether the after-sales installation is abnormal based on the image recognition information and the business logic of the after-sales installation.

[0149] The image recognition model is obtained after training using image samples and identification information labels of each image sample.

[0150] It should be noted that the after-sales installation quality inspection device provided by the present invention can execute the after-sales installation quality inspection method described in any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0151] The after-sales installation quality inspection device provided by the present invention constructs a real-time after-sales installation quality inspection mechanism and an after-sales installation image recognition model through artificial intelligence visual technology, so as to achieve more accurate supervision of after-sales installation scenarios through machine learning, facilitate real-time warning of violations, promote timely tracing and rectification of problem files, and reversely promote the improvement of installation quality.

[0152] Figure 7 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the after-sales installation quality detection method, the method comprising: inputting the product image after the after-sales installation is completed into the image recognition model, and obtaining the image recognition information output by the image recognition model; the image recognition information includes the recognition result after recognizing at least one target area in the product image, and the target area is determined according to the key points of the quality detection of the after-sales installation; based on the image recognition information and the business logic of the after-sales installation, judging whether the after-sales installation is abnormal; the image recognition model is obtained after training with the image sample and the recognition information label of each image sample.

[0153] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0154] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the after-sales installation quality inspection method provided in the above-mentioned embodiments, the method including: inputting the product image after the after-sales installation is completed into an image recognition model, and obtaining image recognition information output by the image recognition model; the image recognition information includes a recognition result after identifying at least one target area in the product image, and the target area is determined based on the key points of the quality inspection of the after-sales installation; based on the image recognition information and the business logic of the after-sales installation, judging whether the after-sales installation is abnormal; the image recognition model is obtained after training using image samples and the identification information label of each of the image samples.

[0155] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the after-sales installation quality inspection method provided in the above-mentioned embodiments, the method comprising: inputting a product image after the after-sales installation is completed into an image recognition model, and obtaining image recognition information output by the image recognition model; the image recognition information comprises a recognition result after identifying at least one target area in the product image, and the target area is determined based on the key points of the quality inspection of the after-sales installation; based on the image recognition information and the business logic of the after-sales installation, determining whether the after-sales installation is abnormal; the image recognition model is obtained after training using image samples and the identification information label of each of the image samples.

[0156] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0157] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for after-sales installation quality inspection, characterized in that: include: Inputting the product image after after-sales installation into the image recognition model, and obtaining image recognition information output by the image recognition model; the image recognition information includes a recognition result after recognizing at least one target area in the product image, and the target area is determined according to the key points of quality inspection of the after-sales installation; Based on the image recognition information and the business logic of the after-sales installation, determining whether the after-sales installation is abnormal; The image recognition model is obtained after training using image samples and identification information labels of each image sample.

2. The after-sales installation quality inspection method according to claim 1, characterized in that: The image recognition model includes an input layer, a feature extraction layer, an attention mechanism module, a dynamic sparse attention module and a classification layer; The step of inputting the product image after after-sales installation into the image recognition model and obtaining the image recognition information output by the image recognition model comprises: Input the product image to the input layer included in the image recognition model to perform image size and format conversion and normalization processing on the product image, and obtain a pre-processed standard image to input to the feature extraction layer; The convolution layer and the pooling layer in the feature extraction layer extract the image feature information in the standard image and input it into the attention mechanism module; The attention mechanism module calculates the attention weight of each feature point according to the feature map of the image feature information and the key points of the after-sales installation quality inspection; performs weighted summation on the feature map according to the calculated attention weight, obtains the weighted feature map and inputs it into the dynamic sparse attention module; The dynamic sparse attention module selects some key feature points from the weighted feature map, and generates target feature information related to the product image based on all the key feature points; The classification layer maps the target feature information to each classification label to obtain the image recognition information.

3. The after-sales installation quality inspection method according to claim 1, characterized in that: The image recognition model is trained based on the following method: Collecting historical after-sales installation image samples, and marking each image sample with the identification information label, wherein the identification information label includes at least one of position information and category information of key points in each target area on the image sample; Acquire text information on each of the image samples; fusing image information related to any image sample with the text information into multimodal information; Inputting any one of the image samples and the corresponding multimodal information into the initial model, and calculating the prediction loss of the initial model according to a comparison result between the recognition information result output by the initial model and the recognition information label corresponding to any one of the image samples; The model parameters of the initial model are iteratively updated according to the prediction loss calculated for each of the image samples until the training results converge, thereby obtaining the image recognition model.

4. The after-sales installation quality inspection method according to claim 3, characterized in that: The obtaining of text information on each of the image samples comprises: Extracting text descriptions related to after-sales installation from each image sample; Extracting semantically relevant core words from the text description; Key content related to quality inspection of the after-sales installation is extracted from the core text as the text information.

5. The after-sales installation quality inspection method according to claim 3, characterized in that: The step of fusing the image information related to any image sample with the text information into multimodal information includes: Inputting the image information and the text information into an image-text matching model to embed the image information and the text information into a feature vector, wherein the feature vector is used to represent the multimodal information; The image-text matching model is constructed based on the self-attention mechanism of the Transformer model.

6. The after-sales installation quality detection method according to any one of claims 1 to 5, characterized in that: The identification information label of any of the image samples is marked in the following manner: Inputting any of the image samples into a visual language model, and obtaining target inspection items output by the visual language model and corresponding to key points of quality inspection for after-sales installation; According to the target inspection item, determining a target rectangular frame for pre-marking the target object of the image sample, wherein the target rectangular frame is used to characterize the position and size of the target object in the image sample; Determine an automatic annotation label according to the target rectangular frame; The identification information label is generated according to the verification result of the automatic marking label.

7. The after-sales installation quality inspection method according to claim 3, characterized in that: After collecting the image samples after the historical after-sales installation is completed, it also includes preprocessing all the image samples, and the preprocessing specifically includes: After performing image cleaning on all the image samples, performing an enhancement operation on each obtained image sample, wherein the enhancement operation includes one or more of rotation, scaling, random cropping, and color gamut transformation; A blur evaluation is performed on at least one image sample obtained after the enhancement operation, so as to perform a blur enhancement operation on the image sample according to the evaluation result, wherein the blur enhancement operation includes one or more of adding noise, adjusting contrast, image degradation, image filtering, and image interpolation.

8. The after-sales installation quality inspection method according to claim 1, characterized in that: When the key point of the quality inspection of the after-sales installation is the metal mixing valve inspection, judging whether the after-sales installation is abnormal based on the image recognition information and the business logic of the after-sales installation includes: When it is determined according to the image recognition information that a round water mixing valve or a plastic water mixing valve is detected from the product image, it is determined that the after-sales installation is normal, otherwise the next judgment is performed; When it is determined that the water heater is not detected from the product image, it is determined that the after-sales installation is normal, otherwise, the next judgment is performed; When it is determined that no metal water mixing valve is detected from the product image, it is determined that the after-sales installation is normal, otherwise, the next judgment is performed; When it is determined that the intersection-over-union ratio between the area where the metal water mixing valve detected from the product image is located and the product image meets a preset threshold, it is determined that the after-sales installation is normal; otherwise, it is determined that the after-sales installation is abnormal.

9. An after-sales installation quality inspection device, characterized in that: include: An after-sales installation image analysis unit, used to input the product image after after-sales installation into the image recognition model, and obtain image recognition information output by the image recognition model; the image recognition information includes a recognition result after recognizing at least one target area in the product image, and the target area is determined according to the key points of the quality inspection of the after-sales installation; an after-sales installation quality judgment unit, configured to judge whether the after-sales installation is abnormal based on the image recognition information and the business logic of the after-sales installation; The image recognition model is obtained after training using image samples and identification information labels of each image sample.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the after-sales installation quality detection method according to any one of claims 1 to 8 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the after-sales installation quality detection method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the after-sales installation quality detection method according to any one of claims 1 to 8 is implemented.

Images