Property response method and system based on multimedia voice image recognition

Abstract

The invention discloses a property response method and system based on multimedia voice image recognition, and relates to the technical field of property dispatching, and the method comprises the steps: starting an event reporting function to carry out voice and image input, and carrying out joint analysis; generating problem description, and classifying problems; evaluating the priority of the problem, and generating a work order for scheduling. According to the property response method based on multimedia voice image recognition, voice and images are combined, multi-modal data support is provided, information errors are effectively avoided through consistency detection, and the accuracy of report information and the robustness of system processing are improved. And through a clear priority scoring rule and a task merging mechanism, reasonable resource allocation is ensured, and the processing efficiency is improved. The method has better effects in the aspects of accuracy, data diversity and efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of property dispatching, and in particular to a property response method and system based on multimedia voice and image recognition. Background Art

[0002] With the rapid development of smart communities, residents have put forward higher requirements for the convenience and efficiency of property management services. Traditional property reporting methods, such as text descriptions and telephone communication, have many drawbacks, including complex user operations, incomplete problem descriptions, and low task assignment efficiency, which seriously affect user experience and the timeliness of problem handling.

[0003] In recent years, multimedia speech recognition technology and image recognition technology have made significant progress in the field of artificial intelligence. Among them, speech recognition technology can convert user voice input into structured text, while image recognition technology can analyze uploaded pictures and extract key features to assist in problem description. Existing technologies such as YOLOv5 and BERT algorithms have shown superior performance in image target detection and text semantic extraction, respectively, and are widely used in intelligent interaction scenarios. However, the application of these technologies in traditional property reporting systems is relatively single, and multimodal information cannot be jointly analyzed, resulting in difficulty in fully exploring the correlation between speech and images.

[0004] In addition, the types of property problems are complex and diverse, such as water and electricity maintenance, public facilities maintenance, cleaning and sanitation, etc. The existing system's mechanism for problem classification and priority assessment is relatively simple, and it is unable to comprehensively consider the urgency of the problem, the scope of impact, and the relevance of historical problems, which directly limits the accuracy and efficiency of problem handling.

[0005] Therefore, there is an urgent need for a property response method that can effectively integrate multimedia voice and image recognition technologies to achieve accurate description of problems, dynamic priority assessment and intelligent task scheduling, so as to improve property management efficiency and user satisfaction. Summary of the invention

[0006] In view of the above-mentioned problems, the present invention is proposed.

[0007] Therefore, the technical problem solved by the present invention is: the existing property problem response method has the problems of single data input, low processing efficiency, inaccurate problem description, and how to efficiently integrate multimodal information and realize intelligent scheduling.

[0008] To solve the above technical problems, the present invention provides the following technical solutions: a property response method based on multimedia voice and image recognition, including starting the reporting function to input voice and image, and perform joint analysis; generating problem descriptions and classifying the problems; evaluating problem priorities and generating work orders for scheduling.

[0009] As a preferred solution of the property response method based on multimedia voice and image recognition described in the present invention, wherein: the startup of the reporting function for voice and image input includes the user starting the reporting function module through the property APP or security robot, the system enters the question input mode, and displays the voice input and image upload entrances;

[0010] The user describes the question through voice, and the system activates the voice recognition module to convert the voice into text information:

[0011] Use the speech recognition model to transcribe user input in real time and extract core keywords;

[0012] The user uploads pictures related to the described problem, and the system performs image recognition:

[0013] Use YOLOv5 to detect key areas in the image.

[0014] As a preferred solution of the property response method based on multimedia voice and image recognition described in the present invention, wherein: the joint analysis includes extracting semantic features in voice and image based on a multimodal learning model;

[0015] Check the consistency of the two input contents. If there is a conflict, prompt the user to correct the input or supplement the information.

[0016] Set V t is the text sequence vector obtained by speech recognition, generated by the semantic encoding model, I f The feature vector obtained by image recognition, the feature map of YOLO is obtained, the matching degree between speech and image is calculated, and the speech and image features are distinguished in high-dimensional features through nonlinear mapping, which is expressed as:

[0017]

[0018] in, For voice mapping, is the image map, W v and W i is the weight matrix, b v ,b i is the bias vector, tanh and ReLU are activation functions;

[0019] In the mapping space, the speech and image features are fused through the weighted attention mechanism, which is expressed as:

[0020]

[0021] Among them, A ijis the feature interaction attention weight of speech and image, d is the feature dimension scaling factor, and N represents the dimension of the speech feature vector;

[0022]

[0023] Among them, σ is the bandwidth parameter of the kernel function, M represents the dimension of the image feature vector, S match for similarity;

[0024] If S match If it is less than the preset similarity value, the system prompts the user to reconfirm the input content;

[0025] If the preset similarity value is met, conflict detection is performed and the keyword set K is extracted from the speech. v , extract the category label set K from the image recognition results i , detect the intersection K common =K v ∩K i ;

[0026] If |K common |=0, prompting the user that the voice description is inconsistent with the image content.

[0027] As a preferred solution of the property response method based on multimedia voice and image recognition described in the present invention, wherein: the generating of the problem description includes the system generating the problem type, specific description, and problem location based on the joint analysis results of the voice and image;

[0028] The problem types include: when the core keywords appear as elevator, access control, and street light, the problem type is public facility failure; when the core keywords appear as water pipe, water leakage, tripping, and power outage, the problem type is water and electricity maintenance; when the core keywords appear as garbage, odor, and accumulated water, the problem type is cleaning and hygiene; when the core keywords appear as smoke alarm, access control failure, and monitoring failure, the problem type is security; when the core keywords appear as trees, green belts, and flower beds, the problem type is greening maintenance;

[0029] The specific description includes: when the problem type is a public facility failure, the description template is set to the facility location, facility name, and failure description; when the problem type is water and electricity maintenance, the description template is set to the specific location and nature of the problem; when the problem type is cleaning and sanitation, the description template is set to the specific location and cleaning tasks; when the problem type is a security problem, the description template is set to the alarm device and failure status; when the problem type is greening maintenance, the description template is set to the plant type and specific problem;

[0030] The question location is obtained through semantic extraction and user GPS, and voice extraction obtains the location by parsing the geographic location information in the voice.

[0031] As a preferred solution of the property response method based on multimedia voice and image recognition described in the present invention, wherein: the classification of the questions includes directly synchronizing the type of the questions to which the user belongs as the main category according to the core keywords of the questions raised by the user, and according to the molecular categories under the main category, the subcategories of public facility failures include elevator failures, unit door failures, and street lamp failures; the subcategories of water and electricity maintenance include water pipe bursts, leakage, tripping, water and power outages; the subcategories of cleaning and sanitation include garbage accumulation, odor removal, and water accumulation treatment; the subcategories of security issues include access control failures, false alarms, and abnormal monitoring images; and the subcategories of greening issues include tree lodging, plant pruning, and green belt cleaning;

[0032] Subcategories are matched based on core keywords. If the question involves multiple categories, the main category is selected based on the highest priority. When the priorities are the same, the category with the highest keyword frequency is selected as the main category.

[0033] As a preferred solution of the property response method based on multimedia voice and image recognition described in the present invention, wherein: the assessment question priority includes calculating the priority score, which is expressed as:

[0034] S p =αP t +βR i +γH h

[0035] Among them, P t Score the problem type, 5 points for utility failures and water and electricity repairs, 2 points for the remaining problems, and R i The score for the scope of influence is 1 point for a single household report, and 1 point is added for each additional user report, with a maximum of 5 points. h is the relevance score of historical issues. Issues that have appeared in the past are scored as 4 points, and those that appear for the first time are scored as 1 point. α, β, and γ are weight coefficients.

[0036] As a preferred solution of the property response method based on multimedia voice and image recognition described in the present invention, the generation of work orders for scheduling includes the system generating standardized work orders based on the problem description, including the problem type, problem description, problem location, priority, merging tasks for adjacent locations and the same problems, generating comprehensive work orders, and scheduling staff based on the work orders.

[0037] Another object of the present invention is to provide a property response system based on multimedia voice and image recognition, which can solve the problems of incomplete information processing and low response efficiency in current property management technology through an intelligent response mechanism based on multimodal joint analysis.

[0038] As a preferred solution of the property response system based on multimedia voice and image recognition described in the present invention, it includes: a data analysis module, a problem classification module, and a scheduling module;

[0039] The path planning module is used to start the reporting function to input voice and images and perform joint analysis;

[0040] The problem classification module is used to generate a problem description and classify the problem;

[0041] The scheduling module is used to evaluate the problem priority and generate work orders for scheduling.

[0042] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a property response method based on multimedia voice and image recognition.

[0043] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a property response method based on multimedia voice and image recognition.

[0044] Beneficial effects of the present invention: The property response method based on multimedia voice and image recognition provided by the present invention combines voice and image, provides multimodal data support, effectively avoids information errors through consistency detection, improves the accuracy of reported information and the robustness of system processing. Through clear priority scoring rules and task merging mechanisms, it ensures reasonable resource allocation and improved processing efficiency. The present invention achieves better results in terms of accuracy, data diversity and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0046] Figure 1 An overall flow chart of a property response method based on multimedia voice and image recognition provided for the first embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0048] Example 1, reference Figure 1 , as an embodiment of the present invention, provides a property response method based on multimedia voice image recognition, comprising:

[0049] S1: Start the reporting function to input voice and images for joint analysis.

[0050] Furthermore, starting the reporting function for voice and image input includes the user starting the reporting function module through the property APP or security robot, the system entering the question input mode, and displaying the voice input and image upload entrances.

[0051] The user describes the question through voice, and the system activates the voice recognition module to convert the voice into text information:

[0052] Use the speech recognition model to transcribe user input in real time and extract core keywords.

[0053] The user uploads pictures related to the described problem, and the system performs image recognition:

[0054] Use YOLOv5 to detect key areas in the image.

[0055] The characteristics of speech and images are significantly different. Speech features are usually time series with strong semantics but weak spatial information. Image features emphasize spatial distribution and have high-dimensional visual information. Mapping both to high-dimensional space and using shared feature representation is the mainstream method of existing multimodal learning. The present invention adapts the characteristics of the two modalities by using different activation functions (such as tanh and ReLU) to process speech and image features. At the same time, nonlinear mapping enhances the distinguishability of feature expression, making it easier to compare consistency in a shared space. This feature alignment method provides strong support for consistency detection.

[0056] It should be noted that the joint parsing includes extracting semantic features from speech and images based on a multimodal learning model.

[0057] Check the consistency of the two input contents. If there is a conflict, prompt the user to correct the input or supplement the information.

[0058] Set V tis the text sequence vector obtained by speech recognition, generated by the semantic encoding model, I f The feature vector obtained by image recognition, the feature map of YOLO is obtained, the matching degree between speech and image is calculated, and the speech and image features are distinguished in high-dimensional features through nonlinear mapping, which is expressed as:

[0059]

[0060] in, For voice mapping, is the image map, W v and W i is the weight matrix, b v ,b i is the bias vector, tanh and ReLU are the activation functions.

[0061] In the mapping space, the speech and image features are fused through the weighted attention mechanism, which is expressed as:

[0062]

[0063] Among them, A ij is the feature interaction attention weight of speech and image, d is the feature dimension scaling factor, and N represents the dimension of the speech feature vector.

[0064]

[0065] Among them, σ is the bandwidth parameter of the kernel function, M represents the dimension of the image feature vector, S match For similarity.

[0066] If S match If the similarity is less than the preset value, the system prompts the user to reconfirm the input.

[0067] If the preset similarity value is met, conflict detection is performed and the keyword set K is extracted from the speech. v , extract the category label set K from the image recognition results i , detect the intersection K common =K v ∩K i .

[0068] If |K common |=0, prompting the user that the voice description is inconsistent with the image content.

[0069] There may be differences or even conflicts in the expression of voice description and image content in real scenes. For example, the user describes "elevator failure" by voice, but the uploaded picture is a photo of stairs. Traditional methods cannot effectively handle such modal conflict problems. The present invention adopts a dual mechanism of similarity scoring and keyword intersection detection, which can prompt users to make corrections in time when the similarity is low or the keywords do not match. This design not only improves the robustness of the system, but also effectively reduces the impact of erroneous information transmission on subsequent problem handling.

[0070] S2: Generate problem description and classify the problem.

[0071] Furthermore, the generation of the question description includes generating the question type, specific description, and question location based on the joint analysis results of speech and image.

[0072] The problem types include: when the core keywords appear as elevator, access control, and street light, the problem type is identified as public facility failure; when the core keywords appear as water pipes, water leakage, tripping, and power outage, the problem type is identified as water and electricity maintenance; when the core keywords appear as garbage, odor, and accumulated water, the problem type is identified as cleanliness and hygiene; when the core keywords appear as smoke alarm, access control failure, and monitoring failure, the problem type is identified as security problem; when the core keywords appear as trees, green belts, and flower beds, the problem type is identified as greening maintenance.

[0073] The specific description includes when the problem type is public facility failure, the description template is set to the facility location, facility name, and fault description; when the problem type is water and electricity maintenance, the description template is set to the specific location and nature of the problem; when the problem type is cleaning and sanitation, the description template is set to the specific location and cleaning tasks; when the problem type is a security problem, the description template is set to the alarm equipment and fault status; when the problem type is greening maintenance, the description template is set to the plant type and specific problem.

[0074] The question location is obtained through semantic extraction and user GPS, and voice extraction obtains the location by parsing the geographic location information in the voice.

[0075] The problem description is generated based on the joint analysis results of voice and image, and the problem type, specific description and problem location are structured through a clear description template. This standardized processing method can solve the problem of vague problem description and unclear semantics in traditional property reporting. For example, when the user inputs "elevator out of service" through voice, combined with the uploaded elevator picture, the system can automatically generate a problem description of "elevator out of service, location: unit door of Building 3". This generation method not only reduces the description burden of the user, but also improves the completeness and expression accuracy of the problem description through parsing model and template matching, laying a solid foundation for subsequent problem classification and work order generation. Traditional systems rely on users' free text input, which is prone to incomplete or biased descriptions, affecting the accurate classification and processing efficiency of problems. The present invention generates problem descriptions through templates, combining specific problem types with fixed description logic, such as "alarm equipment + fault status" templates for security problems, and "plant species + specific problems" templates for greening maintenance. This method not only makes the problem description clear and intuitive, but also adapts to a variety of scenarios, improving the standardization of reporting information.

[0076] The problem location is confirmed through the dual means of semantic extraction and user GPS positioning, which fully combines the subjective description in the voice with the objective data provided by the device. For example, if the user describes "there is a strange smell in the stairwell" in voice, the system locates the building range by parsing the keyword "stairwell", and further accurately locates the specific geographical location of the stairwell by combining GPS. This location acquisition method avoids the location deviation that may be caused by relying solely on voice or GPS, ensuring the accuracy and completeness of location information.

[0077] It should be noted that the classification of problems includes directly synchronizing the type of problems to the main category according to the core keywords of the questions raised by users, and according to the molecular categories under the main category, the subcategories of public facility failures include elevator failures, unit door failures, and street lights not lighting up; the subcategories of water and electricity maintenance include burst pipes, leakages, tripping, and water and power outages; the subcategories of cleaning and sanitation include garbage accumulation, odor removal, and water accumulation treatment; the subcategories of security issues include access control failures, false alarms, and abnormal surveillance images; the subcategories of greening issues include fallen trees, plant pruning, and green belt cleaning.

[0078] Subcategories are matched based on core keywords. If the question involves multiple categories, the main category is selected based on the highest priority. When the priorities are the same, the category with the highest keyword frequency is selected as the main category.

[0079] The problem classification logic directly associates the problem type based on the core keywords extracted by speech analysis and image analysis, and enhances the classification accuracy through further subdivision of subcategories. For example, "water pipe leakage" as a keyword can be directly classified into the main category of "water and electricity maintenance", and specifically belongs to the subcategory of "water pipe burst". This master-slave structure design not only reflects the hierarchy of classification, but also refines the problem attributes to ensure that the problem classification results can accurately reflect the essence of the problem. In the prior art, problem classification often relies on single modal input or manual judgment, and it is difficult to quickly determine the problem type, especially when there are multiple categories of fuzzy problems, which are prone to ambiguity. The present invention reasonably solves the attribution problem when the problem categories overlap through priority rules and keyword frequency determination methods. For example, when the problem involves two categories, "street lights are not on" and "garbage accumulation", the system uses the high-priority "public facility failure" as the main category to ensure that the task assignment gives priority to key problems. This design can effectively avoid classification conflicts and improve the accuracy of classification and the rationality of scheduling.

[0080] S3: Evaluate the problem priority and generate a work order for scheduling.

[0081] Furthermore, evaluating the issue priority involves calculating a priority score, expressed as:

[0082] S p =αP t +βR i +γH h

[0083] Among them, P t Score the problem type, 5 points for utility failures and water and electricity repairs, 2 points for the remaining problems, and R i The score for the scope of influence is 1 point for a single household report, and 1 point is added for each additional user report, with a maximum of 5 points. h is the relevance score of historical issues. Issues that have appeared in the past are scored as 4 points, and those that appear for the first time are scored as 1 point. α, β, and γ are weight coefficients.

[0084] It should be noted that generating work orders for scheduling includes the system generating standardized work orders based on the problem description, including the problem type, problem description, problem location, priority, merging tasks for adjacent locations and the same problems, generating comprehensive work orders, and scheduling staff based on the work orders.

[0085] Example 2, an embodiment of the present invention, provides a property response method based on multimedia voice and image recognition. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0086] First, use the property management system's mobile terminal (such as the property APP) and security robot to simulate users reporting problems.

[0087] Set common problem scenarios in advance, such as "elevator failure", "burst water pipe", and "garbage accumulation", and prepare voice descriptions and pictures related to these problems.

[0088] The system configures the YOLOv5 model for image target detection and combines it with the semantic coding model to extract speech keywords.

[0089] Table 1 Experimental data comparison table

[0090]

[0091]

[0092]

[0093] The table data shows that the similarity scores of speech and image in all experiments were higher than the set threshold (such as 0.8), and the system did not have any false alarms or omissions. For example, in the third experiment, the matching degree between the speech description of garbage accumulation and the image of the trash can was 0.95, indicating that the system can accurately capture the semantic consistency of speech and image. The system accurately classifies problems into main categories and subcategories through the joint analysis of keywords and image labels. For example, in the first experiment, the keyword "elevator. Out of service" combined with the detected elevator panel label was classified as "public facility failure-elevator failure", which is consistent with the manual judgment result. This shows that the classification mechanism of the system is reliable and efficient. The priority scoring fully considers the problem type, impact range and historical relevance. For example, in the first experiment, "elevator outage" involved the residents of the entire building and frequent failures, and the priority score reached 4.8 points, while in the sixth experiment, "tree fall" had a priority score of 3.5 points due to its smaller impact range. This scoring mechanism ensures the rationality of resource allocation and the priority processing of urgent tasks.

[0094] Embodiment 3, an embodiment of the present invention, provides a property response system based on multimedia voice and image recognition, including a data analysis module, a problem classification module, and a scheduling module.

[0095] Among them, the path planning module is used to start the reporting function to input voice and images for joint analysis. The problem classification module is used to generate problem descriptions and classify problems. The scheduling module is used to evaluate problem priorities and generate work orders for scheduling.

[0096] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0097] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0098] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0099] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

[0100] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A property response method based on multimedia voice and image recognition, characterized in that: include: Start the reporting function to input voice and images for joint analysis; Generate problem description and categorize the problem; Assess issue priorities and generate work orders for scheduling.

2. The property response method based on multimedia voice and image recognition as claimed in claim 1, characterized in that: The initiation of the reporting function for voice and image input includes the user initiating the reporting function module through the property APP or the security robot, the system entering the question input mode, and displaying the voice input and image upload entrances; The user describes the question through voice, and the system activates the voice recognition module to convert the voice into text information: Use the speech recognition model to transcribe user input in real time and extract core keywords; The user uploads pictures related to the described problem, and the system performs image recognition: Use YOLOv5 to detect key areas in the image.

3. The property response method based on multimedia voice and image recognition as claimed in claim 2, characterized in that: The joint analysis includes extracting semantic features from speech and images based on a multimodal learning model; Check the consistency of the two input contents. If there is a conflict, prompt the user to correct the input or supplement the information. Set V t is the text sequence vector obtained by speech recognition, generated by the semantic encoding model, I f The feature vector obtained by image recognition, the feature map of YOLO is obtained, the matching degree between speech and image is calculated, and the speech and image features are distinguished in high-dimensional features through nonlinear mapping, which is expressed as: in, For voice mapping, is the image map, W v and W i is the weight matrix, b v ,b i is the bias vector, tanh and ReLU are activation functions; In the mapping space, the speech and image features are fused through the weighted attention mechanism, which is expressed as: Among them, A ij is the feature interaction attention weight of speech and image, d is the feature dimension scaling factor, and N represents the dimension of the speech feature vector; Among them, σ is the bandwidth parameter of the kernel function, M represents the dimension of the image feature vector, S match for similarity; If S match If it is less than the preset similarity value, the system prompts the user to reconfirm the input content; If the preset similarity value is met, conflict detection is performed and the keyword set K is extracted from the speech. v , extract the category label set K from the image recognition results i , detect the intersection K common =K v ∩K i ; If |K common |=0, prompting the user that the voice description is inconsistent with the image content.

4. The property response method based on multimedia voice and image recognition as claimed in claim 3, characterized in that: Generating the question description includes the system generating the question type, specific description, and question location based on the joint analysis results of the speech and image; The problem types include: when the core keywords appear as elevator, access control, and street light, the problem type is public facility failure; when the core keywords appear as water pipe, water leakage, tripping, and power outage, the problem type is water and electricity maintenance; when the core keywords appear as garbage, odor, and accumulated water, the problem type is cleaning and hygiene; when the core keywords appear as smoke alarm, access control failure, and monitoring failure, the problem type is security; when the core keywords appear as trees, green belts, and flower beds, the problem type is greening maintenance; The specific description includes: when the problem type is a public facility failure, the description template is set to the facility location, facility name, and failure description; when the problem type is water and electricity maintenance, the description template is set to the specific location and nature of the problem; when the problem type is cleaning and sanitation, the description template is set to the specific location and cleaning tasks; when the problem type is a security problem, the description template is set to the alarm device and failure status; when the problem type is greening maintenance, the description template is set to the plant type and specific problem; The question location is obtained through semantic extraction and user GPS, and voice extraction obtains the location by parsing the geographic location information in the voice.

5. The property response method based on multimedia voice and image recognition as claimed in claim 4, characterized in that: The classification of the questions includes directly synchronizing the type of questions to which the user belongs according to the core keywords of the questions raised by the user as the main category, and according to the molecular categories under the main category, the subcategories of public facility failures include elevator failures, unit door failures, and street lamp failures; the subcategories of water and electricity maintenance include water pipe bursts, leakages, tripping, and water and power outages; the subcategories of cleaning and sanitation include garbage accumulation, odor removal, and water accumulation treatment; the subcategories of security issues include access control failures, false alarms, and abnormal monitoring images; and the subcategories of greening issues include tree lodging, plant pruning, and green belt cleaning; Subcategories are matched based on core keywords. If the question involves multiple categories, the main category is selected based on the highest priority. When the priorities are the same, the category with the highest keyword frequency is selected as the main category.

6. The property response method based on multimedia voice and image recognition as claimed in claim 5, characterized in that: Evaluating the problem priority includes calculating a priority score, expressed as: S p =αP t +βR i +γH h Among them, P t Score the problem type, 5 points for utility failures and water and electricity repairs, 2 points for the remaining problems, and R i The score for the scope of influence is 1 point for a single household report, and 1 point is added for each additional user report, with a maximum of 5 points. h is the relevance score of historical issues. Issues that have appeared in the past are scored as 4 points, and those that appear for the first time are scored as 1 point. α, β, and γ are weight coefficients.

7. The property response method based on multimedia voice and image recognition as claimed in claim 6, characterized in that: The generation of work orders for scheduling includes the system generating a standardized work order based on the problem description, including the problem type, problem description, problem location, priority, merging tasks for adjacent locations and the same problem, generating a comprehensive work order, and scheduling staff based on the work order.

8. A system using the property response method based on multimedia voice and image recognition as claimed in any one of claims 1 to 7, characterized in that: Includes data analysis module, problem classification module, and scheduling module; The path planning module is used to start the reporting function to input voice and images and perform joint analysis; The problem classification module is used to generate a problem description and classify the problem; The scheduling module is used to evaluate the problem priority and generate work orders for scheduling.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the property response method based on multimedia voice and image recognition described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the property response method based on multimedia voice and image recognition described in any one of claims 1 to 7 are implemented.

Images