A property response method and system based on multimedia voice and image recognition

Abstract

The present invention discloses a property response method and system based on multimedia voice and image recognition, which relates to the field of property scheduling technology, including starting the reporting function to input voice and image, and perform joint analysis; generating problem descriptions, and classifying problems; evaluating problem priorities, and generating work orders for scheduling. The property response method based on multimedia voice and image recognition provided by the present invention combines voice and images, provides multimodal data support, and effectively avoids information errors through consistency detection, thereby improving the accuracy of reporting information and the robustness of system processing. Through clear priority scoring rules and task merging mechanisms, reasonable resource allocation and improved processing efficiency are ensured. The present invention achieves better results in terms of accuracy, data diversity and efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of property scheduling, and in particular to a property response method and system based on multimedia voice and image recognition. Background Art

[0002] With the rapid development of smart communities, residents are placing higher demands on the convenience and efficiency of property management services. Traditional methods of reporting property issues, such as text descriptions and phone calls, have numerous drawbacks, including complex user operations, incomplete problem descriptions, and inefficient task assignments. These drawbacks severely impact user experience and the timeliness of problem resolution.

[0003] In recent years, multimedia speech recognition and image recognition technologies have made significant progress in the field of artificial intelligence. Speech recognition converts user voice input into structured text, while image recognition analyzes uploaded images and extracts key features to aid in problem description. Existing technologies, such as YOLOv5 and BERT, demonstrate superior performance in image object detection and text semantic extraction, respectively, and are widely used in intelligent interaction scenarios. However, these technologies are relatively limited in their application in traditional property management reporting systems, and multimodal information cannot be jointly analyzed, making it difficult to fully explore the correlation between speech and images.

[0004] Furthermore, property management issues are complex and diverse, encompassing issues such as plumbing and electrical repairs, public facility maintenance, and cleaning and sanitation. Existing systems have a relatively simplistic mechanism for problem classification and priority assessment, failing to comprehensively consider the urgency of the issue, the scope of impact, and the relevance of historical issues. This directly limits the accuracy and efficiency of problem resolution.

[0005] Therefore, there is an urgent need for a property response method that can effectively integrate multimedia voice and image recognition technologies to achieve accurate description of problems, dynamic priority assessment and intelligent task scheduling, so as to improve property management efficiency and user satisfaction. Summary of the Invention

[0006] In view of the above-mentioned problems, the present invention is proposed.

[0007] Therefore, the technical problem solved by the present invention is: the existing property problem response method has the problems of single data input, low processing efficiency, inaccurate problem description, and how to efficiently integrate multimodal information and realize intelligent scheduling.

[0008] To solve the above technical problems, the present invention provides the following technical solutions: a property response method based on multimedia voice and image recognition, including starting the reporting function to input voice and image, and perform joint analysis; generating problem descriptions and classifying problems; evaluating problem priorities and generating work orders for scheduling.

[0009] As a preferred embodiment of the property response method based on multimedia voice and image recognition described in the present invention, the method of activating the reporting function for voice and image input includes the user activating the reporting function module through the property APP or security robot, and the system enters the question input mode and displays the voice input and image upload entrances;

[0010] The user describes the question through voice, and the system activates the voice recognition module to convert the voice into text information:

[0011] Use the speech recognition model to transcribe user input in real time and extract core keywords;

[0012] The user uploads a picture related to the problem being described, and the system performs image recognition:

[0013] Use YOLOv5 to detect key areas in the image.

[0014] As a preferred solution of the property response method based on multimedia voice and image recognition of the present invention, wherein: the joint analysis includes extracting semantic features in voice and image based on a multimodal learning model;

[0015] Check the consistency of the two input contents. If there is a conflict, prompt the user to correct the input or add additional information.

[0016] Set V t is the text sequence vector obtained by speech recognition, generated by the semantic encoding model, I f The feature vector obtained by image recognition and the feature map of YOLO are obtained to calculate the matching degree between speech and image. The speech and image features are respectively distinguished by nonlinear mapping in high-dimensional features, which can be expressed as:

[0017]

[0018] in, For voice mapping, is the image map, W v and W i is the weight matrix, b v ,b i is the bias vector, tanh and ReLU are activation functions;

[0019] In the mapping space, speech and image features are fused through the weighted attention mechanism, which is expressed as:

[0020]

[0021] Among them, A ijis the feature interaction attention weight of speech and image, d is the feature dimension scaling factor, and N represents the dimension of the speech feature vector;

[0022]

[0023] Among them, σ is the bandwidth parameter of the kernel function, M represents the dimension of the image feature vector, S match is similarity;

[0024] If S match If the similarity is less than the preset value, the system prompts the user to reconfirm the input;

[0025] If the preset similarity value is met, conflict detection is performed and the keyword set K is extracted from the speech. v , extract the category label set K from the image recognition results i , detect intersection K common =K v ∩K i ;

[0026] If |K common |=0, prompting the user that the voice description is inconsistent with the image content.

[0027] As a preferred solution of the property response method based on multimedia voice and image recognition of the present invention, wherein: generating the problem description includes the system generating the problem type, specific description, and problem location based on the joint analysis results of voice and image;

[0028] Problem types include: when the core keywords appear as elevators, access control, and street lights, the problem type is identified as public facility failure; when the core keywords appear as water pipes, water leakage, tripping, and power outages, the problem type is identified as water and electricity maintenance; when the core keywords appear as garbage, odor, and accumulated water, the problem type is identified as cleanliness and hygiene; when the core keywords appear as smoke alarms, access control failures, and monitoring failures, the problem type is identified as security issues; when the core keywords appear as trees, green belts, and flower beds, the problem type is identified as greening maintenance;

[0029] The specific description includes: when the problem type is a public facility failure, the description template is set to the facility location, facility name, and failure description; when the problem type is water and electricity maintenance, the description template is set to the specific location and nature of the problem; when the problem type is cleaning and sanitation, the description template is set to the specific location and cleaning task; when the problem type is a security problem, the description template is set to the alarm device and failure status; when the problem type is green maintenance, the description template is set to the plant type and specific problem;

[0030] The question location is obtained through semantic extraction and user GPS, and voice extraction obtains the location by parsing the geographic location information in the voice.

[0031] As a preferred embodiment of the multimedia voice and image recognition-based property response method of the present invention, the problem classification includes directly synchronizing the problem type into a main category based on the core keywords of the user's question, and according to the subcategories under the main category, the subcategories of public facility failure include elevator failure, unit door failure, and street light failure; the subcategories of water and electricity maintenance include water pipe burst, leakage, tripping, water and power outage; the subcategories of cleaning and sanitation include garbage accumulation, odor removal, and water accumulation treatment; the subcategories of security issues include access control failure, false alarm, and abnormal monitoring images; and the subcategories of greening issues include fallen trees, plant pruning, and green belt cleaning.

[0032] Subcategories are matched based on core keywords. If the question involves multiple categories, the main category is selected based on the highest priority. When the priorities are the same, the category with the highest keyword frequency is selected as the main category.

[0033] As a preferred embodiment of the property response method based on multimedia voice and image recognition of the present invention, the assessment of the priority of the question includes calculating a priority score, which is expressed as:

[0034] S p =αP t +βR i +γH h

[0035] Among them, P t Score the problem type, 5 points for utility failure and water and electricity repair, 2 points for the remaining problems, and R i The score for the scope of influence is 1 point for a single household report, and 1 point is added for each additional user report, with a maximum of 5 points. h is the relevance score of historical issues. Issues that have appeared in the past are scored as 4 points, and those that appear for the first time are scored as 1 point. α, β, and γ are weight coefficients.

[0036] As a preferred solution of the property response method based on multimedia voice and image recognition described in the present invention, the generation of work orders for scheduling includes the system generating standardized work orders based on the problem description, including the problem type, problem description, problem location, priority, merging tasks for adjacent locations and the same problems, generating a comprehensive work order, and scheduling staff based on the work order.

[0037] Another object of the present invention is to provide a property response system based on multimedia voice and image recognition, which can solve the problems of incomplete information processing and low response efficiency in current property management technology through an intelligent response mechanism based on multimodal joint analysis.

[0038] As a preferred solution of the property response system based on multimedia voice and image recognition described in the present invention, it includes: a data analysis module, a problem classification module, and a scheduling module;

[0039] The path planning module is used to activate the reporting function to input voice and image and perform joint analysis;

[0040] The problem classification module is used to generate problem descriptions and classify problems;

[0041] The scheduling module is used to evaluate problem priorities and generate work orders for scheduling.

[0042] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a property response method based on multimedia voice and image recognition.

[0043] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a property response method based on multimedia voice and image recognition.

[0044] Beneficial Effects of the Presentation: The multimedia voice and image recognition-based property management response method provided by the present invention combines voice and image data, providing multimodal data support. Through consistency detection, it effectively avoids information errors, improves the accuracy of reported information and the robustness of system processing. Through clear priority scoring rules and task merging mechanisms, it ensures rational resource allocation and improves processing efficiency. This present invention achieves superior results in terms of accuracy, data diversity, and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 This is an overall flow chart of a property response method based on multimedia voice and image recognition provided by the first embodiment of the present invention. DETAILED DESCRIPTION

[0047] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0048] Example 1, with reference to Figure 1 , as an embodiment of the present invention, provides a property response method based on multimedia voice and image recognition, comprising:

[0049] S1: Start the reporting function to input voice and image, and perform joint analysis.

[0050] Furthermore, starting the reporting function for voice and image input includes the user starting the reporting function module through the property APP or security robot, the system entering the question input mode, and displaying the voice input and image upload entrances.

[0051] The user describes the question through voice, and the system activates the voice recognition module to convert the voice into text information:

[0052] Use the speech recognition model to transcribe user input in real time and extract core keywords.

[0053] The user uploads a picture related to the problem being described, and the system performs image recognition:

[0054] Use YOLOv5 to detect key areas in the image.

[0055] The characteristics of speech and images are significantly different. Speech features are usually time series with strong semantics but weak spatial information. Image features emphasize spatial distribution and have high-dimensional visual information. Mapping both to high-dimensional space and using shared feature representation is the mainstream method of existing multimodal learning. The present invention adapts the characteristics of the two modalities by using different activation functions (such as tanh and ReLU) to process speech and image features. At the same time, the nonlinear mapping enhances the discriminability of feature expression, making it easier to compare consistency in a shared space. This feature alignment method provides strong support for consistency detection.

[0056] It should be noted that joint parsing includes extracting semantic features from speech and images based on a multimodal learning model.

[0057] Check the consistency of the two input contents. If there is a conflict, prompt the user to correct the input or supplement the information.

[0058] Set V tis the text sequence vector obtained by speech recognition, generated by the semantic encoding model, I f The feature vector obtained by image recognition and the feature map of YOLO are obtained to calculate the matching degree between speech and image. The speech and image features are respectively distinguished by nonlinear mapping in high-dimensional features, which can be expressed as:

[0059]

[0060] in, For voice mapping, is the image map, W v and W i is the weight matrix, b v ,b i is the bias vector, tanh and ReLU are activation functions.

[0061] In the mapping space, speech and image features are fused through the weighted attention mechanism, which is expressed as:

[0062]

[0063] Among them, A ij is the feature interaction attention weight of speech and image, d is the feature dimension scaling factor, and N represents the dimension of the speech feature vector.

[0064]

[0065] Among them, σ is the bandwidth parameter of the kernel function, M represents the dimension of the image feature vector, S match For similarity.

[0066] If S match If the similarity is less than the preset value, the system prompts the user to reconfirm the input.

[0067] If the preset similarity value is met, conflict detection is performed and the keyword set K is extracted from the speech. v , extract the category label set K from the image recognition results i , detect intersection K common =K v ∩K i .

[0068] If |K common |=0, prompting the user that the voice description is inconsistent with the image content.

[0069] In real-world scenarios, voice descriptions and image content may differ or even conflict. For example, a user might describe an elevator outage but upload a photo of stairs. Traditional methods are unable to effectively address this type of modal conflict. This invention employs a dual mechanism of similarity scoring and keyword intersection detection, prompting users to make corrections promptly in the event of low similarity or keyword mismatches. This design not only improves the robustness of the system but also effectively reduces the impact of erroneous information transmission on subsequent problem resolution.

[0070] S2: Generate problem description and classify the problem.

[0071] Furthermore, the generation of the problem description includes generating the problem type, specific description, and problem location based on the joint analysis results of speech and image.

[0072] The problem types include: when the core keywords appear as elevators, access control, and street lights, the problem type is identified as public facility failure; when the core keywords appear as water pipes, water leakage, tripping, and power outages, the problem type is identified as water and electricity maintenance; when the core keywords appear as garbage, odor, and accumulated water, the problem type is identified as cleanliness and hygiene; when the core keywords appear as smoke alarms, access control failures, and monitoring failures, the problem type is identified as security issues; and when the core keywords appear as trees, green belts, and flower beds, the problem type is identified as greening maintenance.

[0073] The specific description includes when the problem type is public facility failure, the description template is set to the facility location, facility name, and fault description; when the problem type is water and electricity maintenance, the description template is set to the specific location and nature of the problem; when the problem type is cleaning and sanitation, the description template is set to the specific location and cleaning task; when the problem type is a security problem, the description template is set to the alarm equipment and fault status; when the problem type is greening maintenance, the description template is set to the plant type and specific problem.

[0074] The question location is obtained through semantic extraction and user GPS, and voice extraction obtains the location by parsing the geographic location information in the voice.

[0075] Problem descriptions are generated based on the results of joint voice and image analysis. Clear description templates are used to structure the problem type, specific description, and location. This standardized approach can address the problem of ambiguous and unclear descriptions in traditional property management reports. For example, when a user inputs "elevator out of service" via voice, combined with an uploaded elevator image, the system automatically generates a problem description: "elevator out of service, location: unit door, building 3." This generation approach not only reduces the user's description burden but also improves the completeness and accuracy of problem descriptions by matching the parsing model with the template, laying a solid foundation for subsequent problem classification and work order generation. Traditional systems often rely on free text input from users, which can lead to incomplete or inaccurate descriptions, hindering accurate problem classification and efficient processing. This invention uses templated problem description generation, combining specific problem types with fixed description logic. For example, a "alarm device + fault status" template is used for security issues, while a "plant species + specific problem" template is used for landscaping maintenance. This approach not only makes problem descriptions clear and intuitive, but also adapts to a variety of scenarios, improving the standardization of problem information.

[0076] The problem location is confirmed through a dual approach of semantic extraction and user GPS positioning, fully integrating the subjective description in the voice with the objective data provided by the device. For example, if a user describes "there's a strange smell in the stairwell," the system analyzes the keyword "stairwell" to locate the building area and, combined with GPS, further pinpoints the exact location of the stairwell. This method of location acquisition avoids the potential for positional deviations that could arise from relying solely on voice or GPS, ensuring the accuracy and completeness of location information.

[0077] It should be noted that the classification of problems includes directly synchronizing the problem type to the main category based on the core keywords of the user's question. According to the molecular categories under the main category, the subcategories of public facility failures include elevator failures, unit door failures, and street lights that are not on. The subcategories of water and electricity maintenance include burst pipes, leakages, tripping, and water and power outages. The subcategories of cleaning and sanitation include garbage accumulation, odor removal, and accumulated water treatment. The subcategories of security issues include access control failures, false alarms, and abnormal monitoring images. The subcategories of greening issues include fallen trees, plant pruning, and green belt cleaning.

[0078] Subcategories are matched based on core keywords. If the question involves multiple categories, the main category is selected based on the highest priority. When the priorities are the same, the category with the highest keyword frequency is selected as the main category.

[0079] The problem classification logic directly associates the core keywords extracted through speech parsing and image analysis with the problem type, and further subcategorizes them to enhance classification accuracy. For example, the keyword "water pipe leak" can be directly classified into the main category of "water and electricity maintenance" and the subcategory of "burst pipe." This master-slave structure not only reflects the hierarchical nature of classification but also refines problem attributes, ensuring that the problem classification results accurately reflect the essence of the problem. Existing problem classification often relies on single-modal input or manual judgment, making it difficult to quickly determine the problem type, especially when there are ambiguous problems with multiple categories. This invention uses priority rules and keyword frequency determination to effectively resolve the classification issue when problem categories overlap. For example, when a problem involves two categories, "street lights not working" and "garbage accumulation," the system selects the higher-priority "public facility failure" as the main category, ensuring that task assignments prioritize critical issues. This design effectively avoids classification conflicts, improving classification accuracy and scheduling efficiency.

[0080] S3: Evaluate the problem priority and generate a work order for scheduling.

[0081] Furthermore, evaluating the issue priority involves calculating a priority score, expressed as:

[0082] S p =αP t +βR i +γH h

[0083] Among them, P t Score the problem type, 5 points for utility failure and water and electricity repair, 2 points for the remaining problems, and R i The score for the scope of influence is 1 point for a single household report, and 1 point is added for each additional user report, with a maximum of 5 points. h is the relevance score of historical issues. Issues that have appeared in the past are scored as 4 points, and those that appear for the first time are scored as 1 point. α, β, and γ are weight coefficients.

[0084] It should be noted that generating work orders for scheduling includes the system generating standardized work orders based on the problem description, including the problem type, problem description, problem location, priority, merging tasks for adjacent locations and the same problems, generating comprehensive work orders, and scheduling staff based on the work orders.

[0085] Example 2, an embodiment of the present invention, provides a property response method based on multimedia voice and image recognition. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0086] First, use the property management system's mobile terminal (such as the property APP) and security robot to simulate users reporting problems.

[0087] Set common problem scenarios in advance, such as "elevator failure", "burst water pipe", and "garbage accumulation", and prepare voice descriptions and pictures related to these problems.

[0088] The system configures the YOLOv5 model for image target detection and combines it with the semantic coding model to extract speech keywords.

[0089] Table 1 Experimental data comparison table

[0090]

[0091]

[0092]

[0093] The tabular data shows that the similarity scores between speech and image were above the set threshold (e.g., 0.8) in all trials, and the system experienced no false positives or omissions. For example, in the third trial, the audio description of garbage accumulation matched the image of the trash can at a match score of 0.95, demonstrating that the system accurately captured the semantic consistency between speech and image. By jointly analyzing keywords and image labels, the system accurately categorized issues into main categories and subcategories. For example, in the first trial, the keyword "elevator.outage" combined with the detected elevator panel labels resulted in the classification of "public facility failure - elevator failure," consistent with manual judgment. This demonstrates the reliability and efficiency of the system's classification mechanism. Priority scoring fully considers issue type, impact scope, and historical relevance. For example, in the first trial, "elevator outage" received a priority score of 4.8 due to its impact on residents throughout the building and its frequent failures. In the sixth trial, "fallen trees" received a priority score of 3.5 due to its smaller impact area. This scoring mechanism ensures optimal resource allocation and prioritizes urgent tasks.

[0094] Example 3, an embodiment of the present invention, provides a property response system based on multimedia voice and image recognition, including a data analysis module, a problem classification module, and a scheduling module.

[0095] The route planning module is used to enable the reporting function, accept voice and image input, and perform joint analysis. The problem classification module is used to generate problem descriptions and classify problems. The scheduling module is used to assess problem priorities and generate work orders for scheduling.

[0096] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0097] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0098] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0099] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logical functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc. It should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention and are not intended to be limiting. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced with equivalents without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications should be encompassed by the claims of the present invention.

[0100] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A property response method based on multimedia voice and image recognition, characterized in that: include: Start the reporting function to input voice and images for joint analysis; Generate problem description and categorize the problem; Assess issue priorities and generate work orders for scheduling; The joint analysis includes extracting semantic features from speech and images based on a multimodal learning model; Check the consistency of the two input contents. If there is a conflict, prompt the user to correct the input or add additional information. set up is the text sequence vector obtained by speech recognition, generated by the semantic encoding model, The feature vector obtained by image recognition and the feature map of YOLO are obtained to calculate the matching degree between speech and image. The speech and image features are respectively distinguished by nonlinear mapping in high-dimensional features, which can be expressed as: ， ， in, For voice mapping, For image maps, and is the weight matrix, is the bias vector, tanh and ReLU are activation functions; In the mapping space, the speech and image features are fused through the weighted attention mechanism, which is expressed as: ， in, is the attention weight for the feature interaction between speech and image, is the feature dimension scaling factor, N represents the dimension of the speech feature vector; ， in, is the bandwidth parameter of the kernel function, represents the dimension of the image feature vector, is similarity; like If the similarity is less than the preset value, the system prompts the user to reconfirm the input; Meet the preset similarity value, perform conflict detection, and extract keyword sets from the speech , extract category label set for image recognition results , detect intersection ; like , prompting users that the voice description is inconsistent with the picture content.

2. The property response method based on multimedia voice and image recognition according to claim 1, characterized in that: The activation of the reporting function for voice and image input includes the user activating the reporting function module through the property APP or security robot, the system enters the question input mode, and displays the voice input and image upload entrances; The user describes the question through voice, and the system activates the voice recognition module to convert the voice into text information: Use the speech recognition model to transcribe user input in real time and extract core keywords; The user uploads a picture related to the problem being described, and the system performs image recognition: Use YOLOv5 to detect key areas in the image.

3. The property response method based on multimedia voice and image recognition according to claim 2, characterized in that: Generating the question description includes the system generating the question type, specific description, and question location based on the joint analysis results of the voice and image; Problem types include: when the core keywords appear as elevators, access control, and street lights, the problem type is identified as public facility failure; when the core keywords appear as water pipes, water leakage, tripping, and power outages, the problem type is identified as water and electricity maintenance; when the core keywords appear as garbage, odor, and accumulated water, the problem type is identified as cleanliness and hygiene; when the core keywords appear as smoke alarms, access control failures, and monitoring failures, the problem type is identified as security issues; when the core keywords appear as trees, green belts, and flower beds, the problem type is identified as greening maintenance; The specific description includes: when the problem type is a public facility failure, the description template is set to the facility location, facility name, and failure description; when the problem type is water and electricity maintenance, the description template is set to the specific location and nature of the problem; when the problem type is cleaning and sanitation, the description template is set to the specific location and cleaning task; when the problem type is a security problem, the description template is set to the alarm device and failure status; when the problem type is green maintenance, the description template is set to the plant type and specific problem; The question location is obtained through semantic extraction and user GPS, and voice extraction obtains the location by parsing the geographic location information in the voice.

4. The property response method based on multimedia voice and image recognition according to claim 3, characterized in that: The problem classification includes directly synchronizing the problem type to a main category based on the core keywords of the user's question, and according to the subcategories under the main category, the subcategories of public facility failure include elevator failure, unit door failure, and street light failure; the subcategories of water and electricity maintenance include water pipe burst, leakage, tripping, water and power outage; the subcategories of cleaning and sanitation include garbage accumulation, odor removal, and water accumulation treatment; the subcategories of security issues include access control failure, false alarms, and abnormal surveillance images; and the subcategories of greening issues include fallen trees, plant pruning, and green belt cleaning; Subcategories are matched based on core keywords. If the question involves multiple categories, the main category is selected based on the highest priority. When the priorities are the same, the category with the highest keyword frequency is selected as the main category.

5. The property response method based on multimedia voice and image recognition according to claim 4, characterized in that: Evaluating the problem priority includes calculating a priority score, expressed as: ， in, Score the problem type, 5 points for utility failures and water and electricity repairs, and 2 points for the remaining problems. For the impact range score, a single household report is worth 1 point, and each additional user report will increase by 1 point, with a maximum of 5 points. For the relevance score of historical questions, 4 points are given for questions that have appeared in the past and 1 point for the first time. , , is the weight coefficient.

6. The property response method based on multimedia voice and image recognition according to claim 5, characterized in that: The generation of work orders for scheduling includes the system generating standardized work orders based on the problem description, including the problem type, specific description, problem location, priority, merging tasks for adjacent locations and the same problem, generating a comprehensive work order, and scheduling staff based on the work order.

7. A system using the property response method based on multimedia voice and image recognition according to any one of claims 1 to 6, characterized in that: Including data analysis module, problem classification module, and scheduling module; The data analysis module is used to activate the reporting function to input voice and images and perform joint analysis; The problem classification module is used to generate problem descriptions and classify problems; The scheduling module is used to evaluate problem priorities and generate work orders for scheduling.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the property response method based on multimedia voice and image recognition according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the property response method based on multimedia voice and image recognition according to any one of claims 1 to 6 are implemented.

Images