Method, device and equipment for detecting water pipe leakage point based on vision and sound

Abstract

The application relates to the technical field of artificial intelligence, and provides a method, device and equipment for jointly detecting a water pipe leakage point based on vision and sound, to solve the problem that there is no leakage anomaly detection method with high accuracy and good universality in related technologies. First, an image sample of a water pipe is taken as input of a vision network model to obtain a positioning result and a positioning confidence for representing a leakage point position in the image sample, and a vision classification result and a vision classification confidence for representing a leakage point category in the image sample; then, a sound sample of the water pipe is taken as input of a sound network model to obtain a sound classification result and a sound classification confidence for representing a sound sample category; the position of the leakage point is comprehensively obtained according to the positioning confidence and the sound classification confidence, and then the category information of the leakage point is comprehensively obtained according to the vision classification confidence and the sound classification confidence, so that the position and type of the leakage point are finally obtained.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus and equipment for detecting water pipe leaks based on a combination of vision and sound. Background Technology

[0002] In computer vision-based intelligent recognition scenarios, leak anomaly detection is a common application. Currently, the industry primarily uses visual recognition methods for intelligent leak detection in water pipes, employing leak point detection and classification to determine the leak location.

[0003] In related technologies, only visual recognition methods are used to detect water pipe leaks, and the leaks are located and classified separately to determine the final leak detection result.

[0004] The above method has the following problems:

[0005] 1. Some smaller leaks are easily missed;

[0006] 2. Changes in lighting conditions increase the difficulty of visual recognition.

[0007] Therefore, there is a lack of highly accurate and versatile methods for detecting leaks and anomalies in related technologies. Summary of the Invention

[0008] The purpose of this application is to provide a method, apparatus, and equipment for detecting water pipe leaks based on a combination of vision and sound, in order to solve the problem of the lack of a leak anomaly detection method with high accuracy and good versatility in related technologies.

[0009] In a first aspect, this application provides a method for detecting water pipe leaks based on a combination of vision and sound, the method comprising:

[0010] Using image samples of water pipes as input to a visual network model, we obtain localization results and localization confidence scores to characterize the location of leaks in the image samples, as well as first classification results and first classification confidence scores to characterize the type of leaks in the image samples.

[0011] Using the sound samples from the water pipes as input to the sound network model, we obtain a second classification result and a second classification confidence level to characterize the types of leaks in the sound samples.

[0012] If the location confidence level is determined to meet the leak location requirement, then based on whether the larger value of the first category confidence level and the second category confidence level meets the leak location confidence requirement, it is determined whether the leak location is a leak point and the corresponding leak point category when it is determined to be a leak point.

[0013] In one possible implementation, if the location confidence level is determined to meet the leak location requirement, then based on whether the larger of the first category confidence level and the second category confidence level meets the leak location confidence requirement, the step of determining whether the leak location is a leak point and the corresponding leak point category when it is determined to be a leak point includes:

[0014] If it is determined that the location confidence is greater than the target location confidence, the first category confidence and the second category confidence are compared, and the larger value is selected as the actual category confidence.

[0015] If the confidence level of the actual category is greater than the confidence level of the target category, then the location of the leak corresponding to the positioning result is determined as the leak point, and the category corresponding to the confidence level of the actual category is taken as the category of the leak point; or,

[0016] If the confidence level of the actual category is not greater than the confidence level of the target category, then the location of the leak corresponding to the positioning result is determined to be not a leak.

[0017] In one possible implementation, the method further includes:

[0018] If the location confidence level does not meet the requirements for the location of the leak, then the location of the leak corresponding to the location result is determined not to be a leak.

[0019] In one possible implementation, after using image samples of water pipes as input to a visual network model, the method further includes:

[0020] If it is determined that there are multiple leak locations in the image sample, the location result of the leak location with the highest location confidence, the location confidence, the first classification result of the leak type, and the first classification confidence are output.

[0021] In one possible implementation, the visual network model is trained using the following method:

[0022] Obtain training samples, which include sample images and labeled locations of leaks in the sample images, and types of leaks at the labeled locations of leaks in the sample images.

[0023] The sample image is input into a visual network model for feature extraction to obtain a feature image and an image feature vector. The feature image is then processed for leak detection and identification to obtain the location of the leak point. Finally, the image feature vector is classified to obtain the classification result of the image feature vector.

[0024] The obtained leak location results are compared with the labeled leak locations, and the obtained image feature vector classification results are compared with the leak types of the leak locations in the labeled sample images to determine the loss function value. Based on the loss function value, the network parameters of the visual network model are updated.

[0025] In one possible implementation, the sound network model is trained using the following method:

[0026] Obtain training samples, which include sample audio and the types of leaks in the labeled sample audio.

[0027] The sample sound is input into a sound network model for feature extraction to obtain a sound feature vector. The sound feature vector is then classified to obtain the classification result of the sound feature vector.

[0028] The classification results of the obtained sound feature vectors are compared with the types of leaks at the labeled locations in the sample sounds to determine the loss function value. Based on the loss function value, the network parameters of the sound network model are updated.

[0029] Secondly, this application provides a device for detecting water pipe leaks based on a combination of vision and sound, the device comprising:

[0030] The image processing module is configured to take an image sample of a water pipe as input to a visual network model to obtain a localization result and localization confidence level for characterizing the location of a leak in the image sample, and a first classification result and first classification confidence level for characterizing the type of leak in the image sample.

[0031] The sound processing module is configured to take the sound samples from the water pipes as input to the sound network model to obtain a second classification result and a second classification confidence level to characterize the types of leaks in the sound samples.

[0032] The detection result determination module is configured to, if it is determined that the location confidence meets the leak location requirement, determine whether the leak location is a leak point and the corresponding leak point category when it is determined to be a leak point, based on whether the larger value of the first category confidence and the second category confidence meets the leak point confidence requirement.

[0033] In one possible implementation, if the location confidence level is determined to meet the leak location requirement, then based on whether the larger of the first category confidence level and the second category confidence level meets the leak location confidence requirement, it is determined whether the leak location is a leak point and the corresponding leak point category when it is determined to be a leak point. The detection result determination module is configured as follows:

[0034] If it is determined that the location confidence is greater than the target location confidence, the first category confidence and the second category confidence are compared, and the larger value is selected as the actual category confidence.

[0035] If the confidence level of the actual category is greater than the confidence level of the target category, then the location of the leak corresponding to the positioning result is determined as the leak point, and the category corresponding to the confidence level of the actual category is taken as the category of the leak point; or,

[0036] If the confidence level of the actual category is not greater than the confidence level of the target category, then the location of the leak corresponding to the positioning result is determined to be not a leak.

[0037] In one possible implementation, the detection result determination module is further configured to:

[0038] If the location confidence level does not meet the requirements for the location of the leak, then the location of the leak corresponding to the location result is determined not to be a leak.

[0039] In one possible implementation, after using image samples of water pipes as input to the visual network model, the image processing module is configured to:

[0040] If it is determined that there are multiple leak locations in the image sample, the location result of the leak location with the highest location confidence, the location confidence, the first classification result of the leak type, and the first classification confidence are output.

[0041] In one possible implementation, the visual network model is trained using the following method:

[0042] Obtain training samples, which include sample images and labeled locations of leaks in the sample images, and types of leaks at the labeled locations of leaks in the sample images.

[0043] The sample image is input into a visual network model for feature extraction to obtain a feature image and an image feature vector. The feature image is then processed for leak detection and identification to obtain the location of the leak point. Finally, the image feature vector is classified to obtain the classification result of the image feature vector.

[0044] The obtained leak location results are compared with the labeled leak locations, and the obtained image feature vector classification results are compared with the leak types of the leak locations in the labeled sample images to determine the loss function value. Based on the loss function value, the network parameters of the visual network model are updated.

[0045] In one possible implementation, the sound network model is trained using the following method:

[0046] Obtain training samples, which include sample audio and the types of leaks in the labeled sample audio.

[0047] The sample sound is input into a sound network model for feature extraction to obtain a sound feature vector. The sound feature vector is then classified to obtain the classification result of the sound feature vector.

[0048] The classification results of the obtained sound feature vectors are compared with the types of leaks at the labeled locations in the sample sounds to determine the loss function value. Based on the loss function value, the network parameters of the sound network model are updated.

[0049] Thirdly, this application provides an electronic device, comprising:

[0050] Processor and memory;

[0051] The memory is used to store the processor-executable instructions;

[0052] The processor is configured to execute the instructions to implement a method for detecting water pipe leaks based on a combination of vision and sound, as provided in any of the first aspects of this application.

[0053] Fourthly, this application provides a computer-readable storage medium that, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform a method for detecting water pipe leaks based on a combination of vision and sound as described in any of the first aspects of this application.

[0054] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method for detecting water pipe leaks based on a combination of vision and sound as described in any of the claims in the first aspect of this application.

[0055] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0056] This application provides a method for detecting water pipe leaks based on a combination of vision and sound. During leak detection, both visual and sound recognition are performed simultaneously, shifting from a single data type to using multiple environmental data for identification. This improves the detection error tolerance and recognition accuracy. In situations with poor visual imaging, the assistance of sound data reduces the impact of lighting changes, lowering the recognition difficulty. Deep learning relies heavily on data; with a large variety and quantity of data, its reliability is better. The method uses visual recognition to locate and classify leaks, outputting location confidence and classification confidence. Sound recognition algorithms are used to classify leaks and output classification confidence. The location of the leak is determined by combining the location confidence and sound classification confidence. Then, the category information of the leak is determined by combining the visual and sound classification confidence, ultimately determining the location and type of the leak. Furthermore, the technical solution provided in this application is simple, has good versatility, and provides high-quality annotation.

[0057] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this application. Based on common knowledge in the art, the above preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this application. Attached Figure Description

[0058] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This application provides an illustration of an application scenario for a method for detecting water pipe leaks based on a combination of vision and sound.

[0060] Figure 2 This application provides an overall flowchart of a method for detecting water pipe leaks based on a combination of vision and sound.

[0061] Figure 3 A flowchart illustrating step 203 provided in an embodiment of this application;

[0062] Figure 4 A schematic diagram illustrating the process of training a visual network model provided in an embodiment of this application;

[0063] Figure 5 A schematic diagram illustrating the process of training a sound network model as provided in an embodiment of this application;

[0064] Figure 6A schematic diagram of a device for detecting water pipe leaks based on a combination of vision and sound is provided in an embodiment of this application;

[0065] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0067] Furthermore, in the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0068] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0069] The following explains the technical terms and technologies involved in this application:

[0070] Voiceprint recognition: Algorithms and techniques for classifying voices based on their characteristics. The voice network model used in this application classifies voice samples using this technique.

[0071] Target localization: Algorithms that use deep learning to locate targets in images. The visual network model used in this application employs this technique to locate the missing points in image samples.

[0072] Confidence level: Indicates the degree of credibility (probability) of the model's output.

[0073] The method for detecting water pipe leaks based on a combination of vision and sound, provided in this application, will be described below with reference to embodiments.

[0074] The inventive concept of this application can be summarized as follows: First, image samples of water pipes are used as input to a visual network model to obtain localization results and location confidence scores for characterizing the location of leaks in the image samples, as well as visual classification results and visual classification confidence scores for characterizing the type of leaks in the image samples. Then, sound samples of water pipes are used as input to a sound network model to obtain sound classification results and sound classification confidence scores for characterizing the type of leaks in the sound samples. The location of the leak is obtained by combining the location confidence score and the sound classification confidence score. Then, the category information of the leak is obtained by combining the visual leak classification confidence score and the sound classification confidence score, and finally, the location and type of the leak are obtained.

[0075] In summary, the method for detecting water pipe leaks based on joint vision and sound provided in this application combines visual and sound recognition during leak detection, transforming it from a single data type to using multiple environmental data for identification. This improves the detection error tolerance and recognition accuracy. In cases of poor visual imaging, the assistance of sound data reduces the impact of lighting changes, lowering the recognition difficulty. Deep learning relies heavily on data, and its reliability is better when there is a large variety and quantity of data. The method is simple, has good versatility, and high-quality annotation.

[0076] After introducing the main inventive concepts of the embodiments of this application, the following is a brief description of the application scenarios to which the technical solutions of the embodiments of this application are applicable. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0077] For ease of understanding, the following detailed description, in conjunction with the accompanying drawings, illustrates a method for detecting water pipe leaks based on a combination of visual and auditory methods, as provided in an embodiment of this application:

[0078] like Figure 1 The diagram illustrates an application scenario of a method for detecting water pipe leaks based on a combination of vision and sound, as provided in this embodiment of the application. The diagram includes: a network 10, a server 20, and a storage device 30. The server 20 acquires image and sound data of the water pipe via the network. Using the method provided in this embodiment, the server can detect and identify the image and sound data of the water pipe, and accurately output whether a leak exists in the water pipe and the corresponding leak category when it is determined to be a leak.

[0079] The description in this application focuses on a single server only. However, those skilled in the art should understand that the illustrated network 10, server 20, and memory 30 are intended to illustrate the operation of the terminal device, server, and memory involved in the technical solution of this application. The detailed description of a single server and memory is at least for ease of explanation and does not imply any limitation on the number, type, or location of servers. It should be noted that adding additional modules to or removing individual modules from the illustrated environment will not change the underlying concept of the exemplary embodiments of this application. Furthermore, although detailed descriptions are provided for ease of explanation... Figure 1 The diagram shows a bidirectional arrow from memory 30 to server 20, but those skilled in the art will understand that the sending and receiving of the aforementioned data also needs to be achieved through network 10.

[0080] It should be noted that the memory in the embodiments of this application can be, for example, a cache system, hard disk storage, memory storage, etc. Furthermore, the method for processing optically dispersed data proposed in this application is not only applicable to… Figure 1 The application scenarios shown can also be used in other possible application scenarios, and the embodiments of this application do not impose any limitations.

[0081] Based on the above description, this application provides a method for detecting water pipe leaks based on a combination of vision and sound, the overall flowchart of which is as follows: Figure 2 As shown, it may include the following:

[0082] In step 201, the image sample of the water pipe is used as the input of the visual network model to obtain the localization result and localization confidence of the leak point location in the image sample, as well as the first classification result and first classification confidence of the leak point type in the image sample.

[0083] In step 202, the sound samples from the water pipes are used as input to the sound network model to obtain a second classification result and a second classification confidence level to characterize the types of leaks in the sound samples.

[0084] In step 203, if the location confidence level meets the leak location requirements, then based on whether the larger of the first category confidence level and the second category confidence level meets the leak location confidence requirements, it is determined whether the leak location is a leak point and the corresponding leak point category when it is determined to be a leak point.

[0085] In one possible implementation, in step 203, if the location confidence level is determined to meet the leak location requirement, then based on whether the larger of the first category confidence level and the second category confidence level meets the leak location confidence requirement, it is determined whether the leak location is a leak point and the corresponding leak point category when it is determined to be a leak point. Figure 3 As shown, the process can be implemented as follows:

[0086] In step 301, if the determined location confidence is greater than the target location confidence, the first category confidence and the second category confidence are compared, and the larger value is selected as the actual category confidence.

[0087] In step 302, if the actual category confidence score is greater than the target category confidence score, then the location of the leak corresponding to the positioning result is determined as the leak point and the category corresponding to the actual category confidence score is taken as the category of the leak point.

[0088] In step 303, if the actual category confidence is not greater than the target category confidence, then the location of the leak corresponding to the location result is determined to be not a leak.

[0089] For example, in step 1, the target location confidence α is set to 0.5 and the target category confidence u is set to 0.6.

[0090] Step 2: Use the image samples of the water pipe as input to the visual network model. The output localization result is (x1, y1, x2, y2) and the localization confidence is 0.7. The image classification result (i.e. the first classification result) is z1 and the image category confidence (i.e. the first classification confidence) is 0.82.

[0091] Step 3: Use the sound sample from the water pipe as input to the sound network model, and output the sound category result (i.e., the second classification result) as z2, and the sound category confidence (i.e., the second classification confidence) as 0.91;

[0092] Step 4: Since the confidence level of the location result is greater than α (0.7>0.5), and the confidence levels of the image category and the sound category are compared (0.82<0.91), the sound category confidence level is selected as the actual category confidence level. According to 0.91>u, the location of the leak is determined to be a water leak point, and the location result is (x1,y1,x2,y2). The category of the water leak point is z2.

[0093] Furthermore, if the target category confidence value u is 1.0, then 0.91 > u, meaning the actual category confidence value is less than the target category confidence value, thus determining that the location of the leak corresponding to the positioning result is not a leak point.

[0094] In one possible implementation, if the location confidence does not meet the leak location requirements, then the leak location corresponding to the location result is determined not to be a leak.

[0095] For example, in step 1, the target location confidence α value in the embodiment of this application is 0.8, and the target category confidence u value is 0.75;

[0096] Step 2: Use the image samples of the water pipe as input to the visual network model. The output localization result is (x1, y1, x2, y2) and the localization confidence is 0.7. The image classification result (i.e. the first classification result) is z1 and the image category confidence (i.e. the first classification confidence) is 0.82.

[0097] Step 3: Use the sound sample from the water pipe as input to the sound network model, and output the sound category result (i.e., the second classification result) as z2, and the sound category confidence (i.e., the second classification confidence) as 0.91;

[0098] Step 4: Since the confidence level of the location result is less than α (0.7 < 0.8), it is determined that the leak location is not the leak point.

[0099] In one possible implementation, after using image samples of water pipes as input to a visual network model, the method further includes:

[0100] If multiple leak locations are identified in the image sample, the output will be the location result of the leak location with the highest location confidence, the location confidence, the first classification result of the leak type, and the first classification confidence.

[0101] For example, after the image sample of the water pipe is processed by the visual network model, three leak locations are identified. The confidence scores for each leak location are 0.5 for leak A, 0.6 for leak B, and 0.7 for leak C. Then, the output is the location result of leak C, the location confidence score, the first classification result of the leak type, and the first classification confidence score.

[0102] In one possible implementation, the process of training the visual network model in this application is as follows: Figure 4 As shown, it includes the following:

[0103] In step 401, training samples are obtained. The training samples include sample images and the locations of leaks in the labeled sample images and the types of leaks at the locations of leaks in the labeled sample images.

[0104] In step 402, the sample image is input into the visual network model for feature extraction processing to obtain a feature image and an image feature vector. The feature image is then processed for leak detection and identification to obtain the location result of the leak point. Finally, the image feature vector is classified to obtain the classification result of the image feature vector.

[0105] In step 403, the obtained location results of the leak points are compared with the labeled leak point locations, and the classification results of the obtained image feature vectors are compared with the leak point types of the leak point locations in the labeled sample images to determine the loss function value. Based on the loss function value, the network parameters of the visual network model are updated.

[0106] In one possible implementation, the process of training the sound network model in this application is as follows: Figure 5 As shown, it includes the following:

[0107] In step 501, training samples are obtained, which include sample audio and the types of leaks in the labeled sample audio.

[0108] In step 502, the sample sound is input into the sound network model for feature extraction to obtain a sound feature vector. The sound feature vector is then classified to obtain the classification result of the sound feature vector.

[0109] In step 503, the classification results of the obtained sound feature vectors are compared with the types of leaks at the labeled locations in the sample sounds to determine the loss function value. Based on the loss function value, the network parameters of the sound network model are updated.

[0110] In summary, the method for detecting water pipe leaks based on joint vision and sound provided in this application combines visual and sound recognition during leak detection, transforming the detection from a single data type to the use of multiple environmental data for identification. This improves the detection error tolerance and recognition accuracy. In cases of poor visual imaging, the assistance of sound data reduces the impact of changes in lighting, thus lowering the recognition difficulty. Deep learning is highly dependent on data, and its reliability is better when there is a large variety and quantity of data.

[0111] Based on the same inventive concept, embodiments of this application also provide a device for detecting water pipe leaks based on a combination of vision and sound, such as... Figure 6 As shown, the device 600 includes:

[0112] The image processing module 601 is configured to take an image sample of a water pipe as input to a visual network model to obtain a localization result and localization confidence level for characterizing the location of a leak in the image sample, and a first classification result and first classification confidence level for characterizing the type of leak in the image sample.

[0113] The sound processing module 602 is configured to take the sound samples of the water pipe as input to the sound network model to obtain a second classification result and a second classification confidence level to characterize the types of leaks in the sound samples.

[0114] The detection result determination module 603 is configured to, if it is determined that the location confidence meets the leak location requirement, determine whether the leak location is a leak point and the corresponding leak point category when it is determined to be a leak point, based on whether the larger value of the first category confidence and the second category confidence meets the leak point confidence requirement.

[0115] In one possible implementation, if the location confidence level is determined to meet the leak location requirement, then based on whether the larger of the first category confidence level and the second category confidence level meets the leak location confidence requirement, it is determined whether the leak location is a leak point and the corresponding leak point category when it is determined to be a leak point. The detection result determination module is configured as follows:

[0116] If it is determined that the location confidence is greater than the target location confidence, the first category confidence and the second category confidence are compared, and the larger value is selected as the actual category confidence.

[0117] If the confidence level of the actual category is greater than the confidence level of the target category, then the location of the leak corresponding to the positioning result is determined as the leak point, and the category corresponding to the confidence level of the actual category is taken as the category of the leak point; or,

[0118] If the confidence level of the actual category is not greater than the confidence level of the target category, then the location of the leak corresponding to the positioning result is determined to be not a leak.

[0119] In one possible implementation, the detection result determination module is further configured to:

[0120] If the location confidence level does not meet the requirements for the location of the leak, then the location of the leak corresponding to the location result is determined not to be a leak.

[0121] In one possible implementation, after using image samples of water pipes as input to the visual network model, the image processing module is configured to:

[0122] If it is determined that there are multiple leak locations in the image sample, the location result of the leak location with the highest location confidence, the location confidence, the first classification result of the leak type, and the first classification confidence are output.

[0123] In one possible implementation, the visual network model is trained using the following method:

[0124] Obtain training samples, which include sample images and labeled locations of leaks in the sample images, and types of leaks at the labeled locations of leaks in the sample images.

[0125] The sample image is input into a visual network model for feature extraction to obtain a feature image and an image feature vector. The feature image is then processed for leak detection and identification to obtain the location of the leak point. Finally, the image feature vector is classified to obtain the classification result of the image feature vector.

[0126] The obtained leak location results are compared with the labeled leak locations, and the obtained image feature vector classification results are compared with the leak types of the leak locations in the labeled sample images to determine the loss function value. Based on the loss function value, the network parameters of the visual network model are updated.

[0127] In one possible implementation, the sound network model is trained using the following method:

[0128] Obtain training samples, which include sample audio and the types of leaks in the labeled sample audio.

[0129] The sample sound is input into a sound network model for feature extraction to obtain a sound feature vector. The sound feature vector is then classified to obtain the classification result of the sound feature vector.

[0130] The classification results of the obtained sound feature vectors are compared with the types of leaks at the labeled locations in the sample sounds to determine the loss function value. Based on the loss function value, the network parameters of the sound network model are updated.

[0131] The following reference Figure 7 To describe an electronic device 130 according to this embodiment of the present application. Figure 7 The electronic device 130 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0132] like Figure 7 As shown, the electronic device 130 is presented in the form of a general-purpose electronic device. The components of the electronic device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).

[0133] Bus 133 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0134] The memory 132 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.

[0135] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0136] Electronic device 130 can also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 130, and / or with any device that enables electronic device 130 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 135. Furthermore, electronic device 130 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 136. As shown, network adapter 136 communicates with other modules used in electronic device 130 via bus 133. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0137] In an exemplary embodiment, this application also provides a computer-readable storage medium including instructions, such as a memory 132 including instructions, which can be executed by a processor 131 of an electronic device 130 to complete the above-described method for detecting water pipe leaks based on combined vision and sound. Optionally, the computer-readable storage medium can be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0138] In an exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor 131, implements the method for detecting water pipe leaks based on a combination of vision and sound as provided in this application.

[0139] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0141] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0142] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0143] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for detecting water pipe leaks based on a combination of vision and sound, characterized in that, The method includes: Using image samples of water pipes as input to a visual network model, we obtain localization results and localization confidence scores to characterize the location of leaks in the image samples, as well as first classification results and first category confidence scores to characterize the type of leaks in the image samples. Using the sound samples from the water pipes as input to the sound network model, a second classification result and a second category confidence score are obtained to characterize the types of leaks in the sound samples. If the location confidence level is determined to meet the leak location requirement, then based on whether the larger value of the first category confidence level and the second category confidence level meets the leak location confidence requirement, it is determined whether the leak location is a leak point and the corresponding leak point category when it is determined to be a leak point.

2. The method according to claim 1, characterized in that, If the location confidence level is determined to meet the leak location requirement, then based on whether the larger of the first category confidence level and the second category confidence level meets the leak location confidence requirement, it is determined whether the leak location is a leak point and the corresponding leak point category when it is determined to be a leak point, including: If it is determined that the location confidence is greater than the target location confidence, the first category confidence and the second category confidence are compared, and the larger value is selected as the actual category confidence. If the confidence level of the actual category is greater than the confidence level of the target category, then the location of the leak corresponding to the positioning result is determined as the leak point, and the category corresponding to the confidence level of the actual category is taken as the category of the leak point; or, If the confidence level of the actual category is not greater than the confidence level of the target category, then the location of the leak corresponding to the positioning result is determined to be not a leak.

3. The method according to claim 1, characterized in that, The method further includes: If the location confidence level does not meet the requirements for the location of the leak, then the location of the leak corresponding to the location result is determined not to be a leak.

4. The method according to claim 1, characterized in that, After using image samples of water pipes as input to the visual network model, the method further includes: If it is determined that there are multiple leak locations in the image sample, the location result of the leak location with the highest location confidence, the location confidence, the first classification result of the leak type, and the first category confidence are output.

5. The method according to claim 1, characterized in that, The visual network model was trained using the following method: Obtain training samples, which include sample images and labeled locations of leaks in the sample images, and types of leaks at the labeled locations of leaks in the sample images. The sample image is input into a visual network model for feature extraction to obtain a feature image and an image feature vector. The feature image is then processed for leak detection and identification to obtain the location of the leak point. Finally, the image feature vector is classified to obtain the classification result of the image feature vector. The obtained leak location results are compared with the labeled leak locations, and the obtained image feature vector classification results are compared with the leak types of the leak locations in the labeled sample images to determine the loss function value. Based on the loss function value, the network parameters of the visual network model are updated.

6. The method according to claim 1, characterized in that, The sound network model was trained using the following method: Obtain training samples, which include sample audio and the types of leaks in the labeled sample audio. The sample sound is input into a sound network model for feature extraction to obtain a sound feature vector. The sound feature vector is then classified to obtain the classification result of the sound feature vector. The classification results of the obtained sound feature vectors are compared with the types of leaks at the labeled locations in the sample sounds to determine the loss function value. Based on the loss function value, the network parameters of the sound network model are updated.

7. A device for detecting water pipe leaks based on a combination of vision and sound, characterized in that, The device includes: The image processing module is configured to take an image sample of a water pipe as input to a visual network model to obtain a localization result and localization confidence level for characterizing the location of a leak in the image sample, and a first classification result and first category confidence level for characterizing the type of leak in the image sample. The sound processing module is configured to take the sound samples of the water pipe as input to the sound network model to obtain a second classification result and a second category confidence level to characterize the types of leaks in the sound samples. The detection result determination module is configured to, if it is determined that the location confidence meets the leak location requirement, determine whether the leak location is a leak point and the corresponding leak point category when it is determined to be a leak point, based on whether the larger value of the first category confidence and the second category confidence meets the leak point confidence requirement.

8. The apparatus according to claim 7, characterized in that, If the location confidence level is determined to meet the leak location requirement, then based on whether the larger of the first category confidence level and the second category confidence level meets the leak location confidence requirement, it is determined whether the leak location is a leak point and the corresponding leak point category when it is determined to be a leak point. The detection result determination module is configured as follows: If it is determined that the location confidence is greater than the target location confidence, the first category confidence and the second category confidence are compared, and the larger value is selected as the actual category confidence. If the confidence level of the actual category is greater than the confidence level of the target category, then the location of the leak corresponding to the positioning result is determined as the leak point and the category corresponding to the confidence level of the actual category is taken as the category of the leak point. or, If the confidence level of the actual category is not greater than the confidence level of the target category, then the location of the leak corresponding to the positioning result is determined to be not a leak.

9. The apparatus according to claim 7, characterized in that, The detection result determination module is also configured to: If the location confidence level does not meet the requirements for the location of the leak, then the location of the leak corresponding to the location result is determined not to be a leak.

10. The apparatus according to claim 7, characterized in that, After using image samples of water pipes as input to the visual network model, the image processing module is configured as follows: If it is determined that there are multiple leak locations in the image sample, the location result of the leak location with the highest location confidence, the location confidence, the first classification result of the leak type, and the first category confidence are output.

11. The apparatus according to claim 7, characterized in that, The visual network model was trained using the following method: Obtain training samples, which include sample images and labeled locations of leaks in the sample images, and types of leaks at the labeled locations of leaks in the sample images. The sample image is input into a visual network model for feature extraction to obtain a feature image and an image feature vector. The feature image is then processed for leak detection and identification to obtain the location of the leak point. Finally, the image feature vector is classified to obtain the classification result of the image feature vector. The obtained leak location results are compared with the labeled leak locations, and the obtained image feature vector classification results are compared with the leak types of the leak locations in the labeled sample images to determine the loss function value. Based on the loss function value, the network parameters of the visual network model are updated.

12. The apparatus according to claim 7, characterized in that, The sound network model was trained using the following method: Obtain training samples, which include sample audio and the types of leaks in the labeled sample audio. The sample sound is input into a sound network model for feature extraction to obtain a sound feature vector. The sound feature vector is then classified to obtain the classification result of the sound feature vector. The classification results of the obtained sound feature vectors are compared with the types of leaks at the labeled locations in the sample sounds to determine the loss function value. Based on the loss function value, the network parameters of the sound network model are updated.

13. An electronic device, characterized in that, include: Processor and memory; The memory is used to store the processor-executable instructions; The processor is configured to execute the instructions to implement the method for detecting water pipe leaks based on a combination of vision and sound as described in any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method for detecting water pipe leaks based on a combination of vision and sound as described in any one of claims 1-6.

Images