House video processing method and device, storage medium and program product

Abstract

The embodiment of the invention provides a house video processing method and device, a storage medium and a program product. In the embodiment of the invention, frame-by-frame scene semantics identification is carried out on a video stream of a target house by taking video frames as granularity, and key frame extraction is carried out based on the scene semantics to which each identified video frame belongs and the probability of the identified video frame belonging to the scene semantics; further, by taking the video clip as granularity, respectively sliding from any key frame to two directions far away from the key frame, and identifying whether the video clip falling into the sliding window during each sliding and the key frame belong to the same scene semantics or not; and when the specified number of video clips and the key frame are continuously identified not to belong to the same scene semantics, the sliding is stopped, so that the start frame and the end frame of the scene semantics to which the key frame belongs can be accurately determined, and the accuracy of identifying the scene semantics corresponding to each space region in the house video is improved; and the recognition accuracy of the video frames corresponding to the scene semantics is improved.

Description

Technical Field

[0001] The present application relates to the field of video processing, and in particular to a house video processing method, device, storage medium and program product. Background Art

[0002] During the online house viewing process, users can select the house they want to know about, and then view the video of the house online. The video includes videos of each room in the house, which makes it easier for users to view and understand the house more intuitively and realistically. It not only meets the needs of users for online house viewing, but also saves the time cost of offline house viewing. In order to facilitate users to understand the house information more quickly and comprehensively, during the process of shooting the house video, the house provider can record audio explanation information for each space area such as the master bedroom and bathroom while shooting, so as to provide voice introduction to the information of each space area. However, the cost and difficulty of recording voice explanation information while shooting is extremely high, and the house provider who shoots the house video is also required to have a high ability to explain the house.

[0003] Therefore, a solution has emerged to configure audio explanations for property videos. That is, after obtaining a property video shot in advance, the scene is technically identified in the property video to identify the various spatial areas in the property, and then audio explanations are configured for each spatial area. This method can reduce the implementation cost and difficulty of configuring audio explanations for property videos, and is not limited by the broker's ability to explain. However, if the scene recognition results for each spatial area in the property video are not accurate enough, the audio explanation and the video will be out of sync. Therefore, how to accurately perform scene recognition on property videos is a technical problem that needs to be solved urgently. Summary of the invention

[0004] Multiple aspects of the present application provide a house video processing method, device, storage medium and program product to improve the accuracy of scene recognition for house videos.

[0005] The embodiment of the present application provides a house video processing method, comprising: receiving a video stream obtained by shooting a target house, the video stream comprising multiple video frames, the target house comprising multiple spatial regions, each spatial region having corresponding scene semantics; inputting the multiple video frames into a scene recognition model to perform scene semantics recognition at the granularity of video frames, and obtaining the scene semantics to which each video frame belongs and the probability of it belonging to the scene semantics; according to the scene semantics to which each video frame belongs, determining continuous video frames belonging to the same scene semantics; according to the probability of each video frame belonging to the scene semantics in the continuous video frames belonging to the same scene semantics, determining the continuous video frames belonging to the same scene semantics. The key frame is determined in the example; for each key frame, a sliding window is used to slide in two directions away from the key frame in sequence starting from the key frame, and the video segment that falls into the sliding window each time is input into the scene recognition model to identify the probability that each video segment and the key frame belong to the same scene semantics with the video segment as the granularity, and the sliding is stopped when a specified number of video segments are continuously identified that do not belong to the same scene semantics as the key frame according to the probability that each video segment and the key frame belong to the same scene semantics; according to the position of the sliding window when it stops sliding in two directions, the start frame and the end frame corresponding to the scene semantics to which the key frame belongs are determined.

[0006] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is coupled to the memory and is used to execute the computer program to implement the steps in each method provided in the embodiment of the present application.

[0007] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps in the above method.

[0008] An embodiment of the present application also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the processor is enabled to implement the steps in the above method embodiment.

[0009] In an embodiment of the present application, the scene semantics of the video stream of the target house are identified frame by frame with the video frame as the granularity, and key frames are extracted based on the scene semantics to which each identified video frame belongs and the probability of it belonging to the scene semantics; further, with the video clip as the granularity, sliding is performed in two directions away from any key frame respectively, and for each video clip that falls into the sliding window, it is identified whether it belongs to the same scene semantics as the key frame; and the sliding is stopped when a specified number of video clips are continuously identified that do not belong to the same scene semantics as the key frame, thereby accurately determining the start frame and end frame of the scene semantics to which the key frame belongs, thereby improving the accuracy of identifying the scene semantics corresponding to each spatial area in the house video, and improving the accuracy of identifying the video frames corresponding to each scene semantics. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0011] Figure 1 A schematic flow chart of a method for processing house videos provided as an exemplary embodiment of the present application;

[0012] Figure 2a-2c This is a schematic diagram of the output of a scene recognition model in another exemplary embodiment of the present application;

[0013] Figure 3 A flowchart of another house video processing method provided as an exemplary embodiment of the present application;

[0014] Figure 4 A schematic structural diagram of an electronic device is provided as yet another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0016] It should be noted that, in the case where the embodiments of the present application involve user information, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) are in compliance with relevant laws and standards.

[0017] In order to solve the technical problem of low accuracy in scene recognition for house videos, in an embodiment of the present application, the scene semantics of the video stream of the target house are recognized frame by frame with the video frame as the granularity, and key frames are extracted based on the scene semantics to which each recognized video frame belongs and the probability of it belonging to the scene semantics; further, with the video clip as the granularity, sliding is performed in two directions away from any key frame respectively, and the video clip that falls into the sliding window each time is identified as belonging to the same scene semantics as the key frame; and the sliding is stopped when a specified number of video clips are continuously identified that do not belong to the same scene semantics as the key frame, thereby accurately determining the start frame and end frame of the scene semantics to which the key frame belongs, thereby improving the accuracy of scene semantics recognition corresponding to each spatial area in the house video, and improving the accuracy of video frame recognition corresponding to each scene semantics.

[0018] Furthermore, when continuously identifying whether a specified number of video clips have the same semantics as the scene to which the key frame belongs, a contradiction-proof model is introduced to perform consistency verification on the recognition results of the scene recognition model to correct inaccurate recognition results. In this process, object information is extracted from the video clips that are continuously identified to be different from the scene to which the key frame belongs; according to the contradiction-proof rule between the object information and the scene semantics to which the key frame belongs, a scene consistency verification is performed on the specified number of video clips. If the scene semantics to which the specified number of video clips belong is the same as the scene semantics to which the key frame belongs, the sliding window continues to slide; if the scene semantics to which the specified number of video clips belong is different from the scene semantics to which the key frame belongs, the sliding window stops sliding.

[0019] Furthermore, the concept of target scene semantics is introduced. The target scene semantics refers to the scene semantics including multiple sub-scene semantics. For the case where there are multiple key frames belonging to the target scene semantics, each key frame can determine a video segment, and multiple key frames correspond to multiple video segments. These video segments belong to the target scene semantics. In an embodiment of the present application, lighting features, field of view features and area features are identified based on a multimodal recognition model to divide the multiple video segments corresponding to the target scene semantics into sub-scene semantics, so as to obtain a sub-scene semantics corresponding to each video segment.

[0020] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0021] Figure 1 The following is a flow chart of a method for processing house video provided by an exemplary embodiment of the present application. Figure 1 As shown, the method includes:

[0022] S101: receiving a video stream obtained by shooting a target house, where the video stream includes multiple video frames, and the target house includes multiple spatial regions, each of which has corresponding scene semantics;

[0023] S102: Inputting multiple video frames into a scene recognition model to perform scene semantics recognition at the video frame granularity, and obtaining the scene semantics to which each video frame belongs and the probability of it belonging to the scene semantics;

[0024] S103: determining continuous video frames belonging to the same scene semantics according to the scene semantics to which each video frame belongs; determining key frames from the continuous video frames belonging to the same scene semantics according to the probability that each video frame in the continuous video frames belonging to the same scene semantics belongs to the scene semantics;

[0025] S104: for each key frame, slide the sliding window from the key frame to two directions away from the key frame respectively, input the video segments that fall into the sliding window each time into the scene recognition model, identify the probability that each video segment and the key frame belong to the same scene semantics based on the video segment granularity, and stop sliding when a specified number of video segments are continuously identified that do not belong to the same scene semantics as the key frame according to the probability that each video segment and the key frame belong to the same scene semantics;

[0026] S105: Determine the start frame and the end frame corresponding to the scene semantics to which the key frame belongs according to the position where the sliding window stops sliding in two directions.

[0027] In the embodiment of the present application, the target house includes multiple spatial areas, and different types of spatial areas correspond to different scene semantics. The corresponding scene semantics can be defined according to the function of the spatial area. For example, according to the function of the spatial area, it can be divided into scene semantics such as bedroom, bathroom, kitchen and living room. Among them, there can be multiple spatial areas in the target house that belong to the target scene semantics, and the target scene semantics can be but not limited to bedroom, bathroom and kitchen, etc. For example, the target house can include multiple spatial areas that belong to the scene semantics of bathroom.

[0028] In this embodiment, a corresponding video stream can be obtained by shooting the target house, and the video stream includes multiple video frames sorted in time sequence on the time axis; wherein the multiple video frames are obtained by shooting different spatial areas of the target house, and the multiple video frames can cover multiple spatial areas, for example, some video frames are obtained by shooting spatial area A, and some video frames are obtained by shooting spatial area B, and so on. Among them, the video frames obtained by shooting the same spatial area may be continuous or discontinuous. For example, bedroom A is shot twice, and 10 video frames are obtained by the first shooting. Then, other spatial areas such as the living room and balcony are shot, and then bedroom A is shot again to obtain 30 video frames. It can be seen that the video frames shot for bedroom A are not continuous, and the video frames obtained by the first shooting and the video frames obtained by the second shooting are separated by video frames corresponding to other spatial areas.

[0029] In this embodiment, the content provider can provide the video stream obtained by shooting the target house to the computer device as the video stream for subsequent scene semantic recognition. There is no restriction on the content provider, for example, it can be a real estate agent; or it can also be a landlord; or it can also be other relevant personnel, etc. There is no restriction on the type of computer device, and the computer device can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone or an IOT (Internet of Things) device, or it can be a server device such as a conventional server, a cloud server or a server array.

[0030] Furthermore, the multiple video frames included in the video stream are input into the scene recognition model to perform scene semantics recognition at the granularity of the video frame. The scene recognition model is pre-trained using sample video streams of some houses, and the sample video streams include multiple sample video frames. The scene semantics to which these sample video frames belong can cover various scene semantics in the house, such as the living room, bedroom, bathroom, and kitchen, etc. Furthermore, the scene recognition model can be trained using the sample video frames in the sample video stream to obtain a scene recognition model. The scene recognition model can be used to recognize the scene semantics of the video frames included in the video stream. For example, it can be recognized that the scene semantics to which the video frame belongs is the living room, bedroom, bathroom, or kitchen, etc. The scene recognition model can be, but is not limited to, a MAE (Masked AutoEncoders) model.

[0031] Among them, for any input video frame, the scene recognition model will predict the probability that the video frame may belong to each scene semantics. For example, it can predict the probability that the video frame may belong to the scene semantics of the living room, bedroom, bathroom and kitchen, etc., and select the first target probability that meets the first probability condition from the probabilities that the video frame may belong to each scene semantics, and take the scene semantics corresponding to the first target probability as the scene semantics to which the video frame belongs. For example, the first probability condition can be to select the highest probability from the probabilities that the video frame may belong to each scene semantics. The highest probability can be regarded as an example of the first target probability, but is not limited to this; the scene semantics corresponding to the highest probability is taken as the scene semantics to which the video frame belongs; and then, the scene recognition model outputs the scene semantics to which the video frame belongs and the probability of it belonging to the scene semantics.

[0032] like Figure 2a As shown, it is an example of a scene recognition model outputting the scene semantics to which a video frame belongs and the probability of it belonging to the scene semantics, see Figure 2a , the horizontal axis is the duration of the video stream, and the vertical axis is the probability of the scene semantics to which the video frame belongs. Figure 2a In the probability range, 0 means that it is impossible to belong to the scene semantics, 1 means that it belongs to the scene semantics, and between 0 and 1, it means the possibility of belonging to the scene semantics. From 0 to 1, the possibility gradually increases. It should be understood that Figure 2a The value range of this probability is only an example and does not limit this embodiment. Figure 2a Different colors represent different scene semantics. Figure 2a The scene semantics include "other 1", "other 2", "bedroom", "bathroom", "kitchen" and "living room". Figure 2b-Figure 2c The scene semantics in can be referred to the definition here and will not be repeated in subsequent embodiments.

[0033] Further, according to the scene semantics to which each video frame belongs, the continuous video frames belonging to the same scene semantics are determined; according to the probability that each video frame in the continuous video frames belonging to the same scene semantics belongs to the scene semantics, key frames are respectively determined from the continuous video frames belonging to the same scene semantics; in this process, from the continuous video frames belonging to the same scene semantics, the video frames with the second target probability whose probability of belonging to the scene semantics meets the second probability condition are selected as the key frames of the scene semantics; wherein the second probability condition can be to select the video frame with the highest probability of belonging to the scene semantics from the continuous video frames of the same scene semantics, but is not limited thereto. It should be understood that the first probability condition and the second probability condition can be the same or different, and there is no limitation on this.

[0034] Continue with the above Figure 2a Example, Figure 2b An example of a key frame is given, which is key frame 1, key frame 2 and key frame 3 determined for the scene semantics of bedroom. For the scene semantics of bedroom, three continuous video frames are involved, and key frames are extracted for each continuous video frame to obtain key frame 1, key frame 2 and key frame 3. It can be seen that there may be multiple continuous video frames belonging to the same scene semantics, and the key frames of the scene semantics may be located in different segments of continuous video frames.

[0035] Among them, any key frame is used as a reference point to determine the starting frame position and the ending frame position of the scene semantics to which the key frame belongs, and according to the starting frame position and the ending frame position corresponding to the key frame, the starting frame and the ending frame corresponding to the scene semantics to which the key frame belongs are determined, and then according to the starting frame and the ending frame, a video segment belonging to the scene semantics can be determined. Among them, the same scene semantics can have one or more key frames, and each key frame can be used to determine a video segment of the scene semantics to which it belongs, and the video segment is the same as the scene semantics to which the key frame belongs.

[0036] In this embodiment, the same sliding window processing is performed for each key frame, and the following description is carried out by taking any key frame as an example. Taking any key frame as a reference point, the sliding window is used to slide in two directions away from the key frame in sequence from the key frame; wherein, one of the two directions is the time direction of the time stamp earlier than the key frame, and the other is the time direction of the time stamp later than the key frame, and there is no limitation on the order of sliding in the two directions; the video clips that fall into the sliding window each time are input into the scene recognition model, and the probability of each video clip and the key frame belonging to the same scene semantics is identified with the video clip as the granularity; for each video clip, the scene recognition model will predict the probability that the video clip and the key frame belong to the same scene semantics, and then the video clip and the key frame can be identified according to the probability that the video clip and the key frame belong to the same scene semantics. For example, if the scene semantics of the key frame is bedroom, the scene recognition model can predict the probability that the video clip belongs to bedroom, and then the scene semantics of the video clip can be judged according to the probability that the video clip belongs to bedroom.

[0037] In an optional implementation, for each video clip, the scene recognition model predicts the probability that the video clip may belong to each scene semantics, for example, predicts the probability that the video clip may belong to the scene semantics of the living room, bedroom, bathroom and kitchen, and selects the highest probability from the probabilities that the video clip may belong to each scene semantics, and uses the scene semantics corresponding to the highest probability as the scene semantics of the video clip; then, determines whether the scene semantics of the video clip is the same as that of the key frame; if not, obtains the probability that the video clip and the key frame belong to the same scene semantics, and determines whether the video clip and the key frame belong to the same scene semantics based on the probability.

[0038] In which, with the video clip as the granularity, in the process of sequentially performing scene semantics recognition on the video clips falling into the sliding window, the video clips adjacent to the key frame and having the same scene semantics as the key frame are searched from the key frame. In this process, a probability threshold can be set. If the probability that the video clip and the key frame belong to the same scene semantics is greater than or equal to the probability threshold, it can be considered that the video clip and the key frame belong to the same scene semantics; conversely, if the probability that the video clip and the key frame belong to the same scene semantics is less than the probability threshold, it can be considered that the video clip and the key frame do not belong to the same scene semantics.

[0039] In the process of performing sliding window processing from a key frame to find a video segment with the same scene semantics as the key frame, there may be a sudden change in the scene semantics of a few consecutive video segments in the sliding window, that is, the scene semantics of a few consecutive video segments are different from the scene semantics of the key frame, wherein the multiple video segments that fall into the sliding window in sequence during the continuous sliding process are continuous video segments. In other words, continuous video segments refer to multiple video frequency bands that are immediately connected in time sequence. However, a few continuous video segments with sudden scene semantics are not the boundaries of the key frame in the true sense, because when the video segments with sudden scene semantics continue to slide away from the key frame, video segments with the same scene semantics as the key frame may be found. If these video segments with sudden scene semantics are used as the "boundary" of the key frame, video segments with the same scene semantics outside the "boundary" may be lost.

[0040] Based on this, in order to eliminate the interference caused by the sudden change of the recognition results in a few continuous sliding windows in a short period of time, in this embodiment, a specified number is predetermined, and the specified number can be determined according to an empirical value. When the video clips in less than the specified number of continuous sliding windows are different from the scene to which the key frame belongs, it is considered that the scene semantics of the video clips in less than the specified number of continuous sliding windows have mutated. These video clips with mutated scene semantics are not the boundaries of the key frame in the true sense. Therefore, the sliding window processing can continue to identify the video clips with the same scene semantics as the key frame. When it is continuously identified that the scene semantics of the specified number of video clips are different from the scene semantics of the key frame, it can be considered that the boundary of the key frame is found, and then the sliding window can stop sliding to stop the recognition operation of the video clip.

[0041] In this embodiment, the sliding window starts from any key frame and slides in two directions away from the key frame in sequence; wherein, one direction of the two directions is a time direction earlier than the timestamp of the key frame, and the other direction is a time direction later than the timestamp of the key frame; each direction uses the key frame as a reference point, and the sliding window starts from the key frame and slides in that direction away from the key frame in sequence until the boundary of the key frame on the timeline in that direction is found, and the boundary is the time position on the timeline that does not belong to the same scene semantics as the key frame; when the boundary is found in two directions, the sliding window stops sliding in both directions. Furthermore, according to the time position when the sliding window stops sliding in two directions, the start frame and the end frame corresponding to the scene semantics to which the key frame belongs are determined, wherein the start frame can be determined by the start position, and the start frame position is the farthest time position of the sliding window sliding in the time direction earlier than the timestamp of the key frame; correspondingly, the end frame can be determined by the end frame position, and the end frame position is the farthest time position of the sliding window sliding in the time direction later than the timestamp of the key frame; according to the start frame and the end frame, the video segment corresponding to the scene semantics to which the key frame belongs is determined, and the scene semantics of the video segment are the same as the scene semantics to which the key frame belongs.

[0042] Continue with the above Figure 2a For example, Figure 2c As shown, Figure 2c yes Figure 2a From the effect diagram after the above-mentioned house video processing method, we can see that Figure 2a The jitter and burr of the video have been removed. These burrs and jitters are the video frames that interfere with the scene semantic recognition of the video clip. For example, the burr can be a video frame that does not belong to the scene semantics of a certain video clip, and the jitter can be a video frame where the scene is wrongly cut due to the shaking of the mobile phone caused by human factors during video stream shooting. Figure 2c These glitches and jitters can be removed, and the start time and end time of the video segment corresponding to each scene semantics can be clearly seen, wherein the start time and end time are determined according to the start frame position and the end frame position.

[0043] In an embodiment of the present application, the scene semantics of the video stream of the target house are identified frame by frame with the video frame as the granularity, and key frames are extracted based on the scene semantics to which each identified video frame belongs and the probability of it belonging to the scene semantics; further, with the video clip as the granularity, sliding is performed in two directions away from any key frame respectively, and for each video clip that falls into the sliding window, it is identified whether it belongs to the same scene semantics as the key frame; and the sliding is stopped when a specified number of video clips are continuously identified that do not belong to the same scene semantics as the key frame, thereby accurately determining the start frame and end frame of the scene semantics to which the key frame belongs, thereby improving the accuracy of identifying the scene semantics corresponding to each spatial area in the house video, and improving the accuracy of identifying the video frames corresponding to each scene semantics.

[0044] Figure 3 A flowchart of another house video processing method provided by an exemplary embodiment of the present application is shown below. Figure 3 As shown, the process involves a terminal device 31 of a content provider and a computer device 30 that executes a house video processing method; wherein the terminal device 31 of the content provider and the computer device 30 can be the same device or different devices, and there is no restriction on this. In addition, there is no restriction on the deployment location of the scene recognition model 32, for example, it can be deployed on the computer device 30; or, it can also be deployed on the terminal device 31 of the content provider. Figure 3 In the figure, the terminal device 31 of the content provider and the computer device 30 are taken as different devices, and the scene recognition model 32 is deployed on the computer device 30 for example, but it is not limited to this.

[0045] In this embodiment, the content provider uploads the video stream obtained by shooting the target house to the computer device 30; after receiving the video stream, the computer device 30 inputs the multiple video frames included in the video stream into the scene recognition model 32, such as Figure 3 As shown in S301 , the scene recognition model 32 recognizes scene semantics at the granularity of video frames, and obtains the scene semantics to which each video frame belongs and the probability of it belonging to the scene semantics.

[0046] Optionally, during the process of recognizing scene semantics of video frames, the scene recognition model performs feature extraction on any video frame to obtain feature information of the video frame; performs scene semantics prediction on the video frame to obtain the probability that the video frame may belong to each scene semantics; selects a first target probability that meets a first probability condition from the probabilities that the video frame may belong to each scene semantics, and uses the scene semantics corresponding to the first target probability and the first target probability as the scene semantics to which the video frame belongs and the probability that it belongs to the scene semantics, respectively.

[0047] In an optional embodiment, before inputting multiple video frames into the scene recognition model, the scene semantic scope can also be determined in advance, and the determination method includes: receiving the floor plan of the target house; determining the scene semantic scope of the target house based on the floor plan of the target house; inputting the scene semantic scope into the scene recognition model as a constraint condition for scene semantic recognition; accordingly, when predicting the scene semantics of any video frame to obtain the probability that the video frame may belong to each scene semantics, it includes: within the scene semantic scope, predicting the scene semantics of any video frame to obtain the probability that the video frame may belong to each scene semantics. Since a limited scene semantic scope is set, it is ensured that the output result of the scene recognition model is limited to the scene semantic scope, thereby improving the recognition efficiency and accuracy of the scene semantics and the video frames corresponding to the scene semantics.

[0048] Further, if Figure 3 As shown in S302, according to the scene semantics to which each video frame belongs, continuous video frames belonging to the same scene semantics are determined; according to the probability that each video frame in the continuous video frames belonging to the same scene semantics belongs to the scene semantics, key frames are respectively determined from the continuous video frames belonging to the same scene semantics; the same scene semantics may include at least one key frame, and the details of this part can be referred to the aforementioned embodiment, which will not be repeated here.

[0049] In an optional embodiment, before using the sliding window to slide in two directions away from any key frame in sequence, it also includes: obtaining parameter information of the video stream, the parameter information includes the duration and / or frame rate of the video stream; further, determining the width of the sliding window according to the duration and / or frame rate of the video stream; wherein the duration and / or frame rate of the video stream is positively correlated with the width of the sliding window, and the longer the duration of the video stream, the slower the scene switching is during the process of shooting the target house, which means that the amount of information contained in the video frame per unit time is more limited, because the width of the sliding window can be set larger; the higher the frame rate, the more video frames are contained per unit time, so the width of the sliding window also needs to be set larger to improve the overall recognition efficiency of the video stream. Table 1 is given below, which is a comparison table of the duration of a video stream and the width of the sliding window.

[0050] Table 1:

[0051] Video duration / minute Sliding Window / Second [0,2] 0.05s (2,5] 0.15s (5,10] 0.25s (10,+) 0.50s

[0052] As shown in Table 1, if the duration of the video stream is ≥0 minutes and ≤2 minutes, the width of the sliding window can be 0.05 seconds; if the duration of the video stream is >2 minutes and ≤5 minutes, the width of the sliding window can be 0.15 seconds; Table 1 also includes the width of the sliding window corresponding to other video durations, which will not be repeated here. Considering that the sliding window has a preset width, whether it is the starting frame position or the ending frame position, the position is determined by the number of times the sliding window slides in the corresponding direction and the preset width of the sliding window. For example, assuming that the timestamp of a key frame is T seconds, the preset width of the sliding window is △t, △t = 2 seconds; the sliding window stops sliding after sliding N times in the time direction earlier than the timestamp of the key frame, N ≥ 1, and N is a natural number; then the starting frame position = TN × △t; accordingly, the ending frame position can refer to the calculation method of the starting frame position, which will not be repeated here.

[0053] Furthermore, if Figure 3 As shown in S303, for each key frame, a sliding window is used to slide in two directions away from the key frame with a preset width, and the video segments that fall into the sliding window each time are input into the scene recognition model to identify the probability that each video segment and the key frame belong to the same scene semantics with the video segment as the granularity. Optionally, the sliding window has a preset width, and the preset width can be determined according to the parameter information of the video stream, and the determination method can refer to the above embodiment; further, in the process of scene semantics recognition with the video segment as the granularity of the scene recognition model, the video segment that falls into the sliding window each time is input into the scene recognition model, and feature extraction is performed on each video frame in the video segment to obtain feature information of each video frame in the video segment, wherein the number of video frames included in the video segment is positively correlated with the preset width of the sliding window; further, feature aggregation is performed on the feature information of each video frame in the video segment to obtain aggregated features; scene semantics are predicted for the aggregated features to obtain the probability that the video segment and the key frame belong to the same scene semantics, so as to determine whether the video segment and the key frame belong to the same scene semantics.

[0054] In this embodiment, scene semantics recognition is performed at the granularity of video segments starting from any key frame. Compared with the solution of performing scene semantics recognition frame by frame, it can not only significantly improve the recognition efficiency, but also reduce the impact of scene semantic mutations of a single video frame on the scene semantic recognition of the entire video segment, thereby improving the accuracy of scene semantics and the recognition of video frames corresponding to scene semantics.

[0055] Optionally, in the process of continuously identifying a specified number of video clips that do not belong to the same scene semantics as any key frame based on the probability that each video clip belongs to the same scene semantics as any key frame, on the one hand, a set probability threshold can be corresponded for the scene semantics to which any key frame belongs, and different scene semantics can correspond to the same or different set probability thresholds; further, based on the probability that each video clip belongs to the same scene semantics as the key frame, video clips with a probability less than the set probability threshold can be identified; on the other hand, statistics are performed on video clips that are continuously identified and whose probability is less than the set probability threshold, and when the count value used to count the number of video clips does not reach the specified number, the count value is increased by 1 each time one is identified until the specified number is reached; if the probability of identifying that the video clip and the key frame belong to the same scene semantics is greater than the set probability threshold, and the count value is not zero, the count value can be cleared to zero, thereby ensuring that the counted count value is a video clip within the continuous sliding window.

[0056] In this embodiment, in the process of continuously identifying that the scene semantics of a specified number of video clips are different from the scene semantics of the key frame, such as Figure 3 As shown in S304, the scene semantics to which a specified number of video clips belong are checked for consistency using a disproven model and an object detection model to avoid misjudgment of the scene recognition model and improve the recognition rate of the scene recognition model. The specified number of video clips are fed into the object detection model, and at least one item information appearing in the specified number of video clips is obtained through the object detection model, so as to determine whether the at least one item information contains target item information that matches the scene semantics to which the key frame belongs; if the at least one item information does not contain the target item information, the sliding window is stopped, otherwise, if the at least one item information contains the target item information, the sliding window can continue to slide. Figure 3 Steps S3041 and S3042 in describe how to perform consistency check.

[0057] In an alternative embodiment, if Figure 3 As shown in step S3041 in the example, a specified number of video clips are input into the object detection model to detect object information, and at least one object information appearing in the specified number of video clips is obtained; at least one object information and the scene semantics to which the key frame belongs are input into the disproof model, and different scene semantics correspond to their own disproof rules, such as Figure 3As shown in step S3042 in , the proof-by-contradiction model verifies whether at least one item information includes target item information that is adapted to the scene semantics to which the key frame belongs, based on the proof-by-contradiction rule existing between the item information and the scene semantics; the target item information is adapted to the scene semantics to which the key frame belongs, which means that the target item information does not violate facts or common sense under the scene semantics, and can serve as a marker of the scene semantics, and is item information unique to the scene semantics. For example, if the scene semantics to which the key frame belongs is a bathroom, the target item information may be a toilet; if the scene semantics to which the key frame belongs is a kitchen, the target item information may be a range hood.

[0058] Wherein, the counter-proof rule may include target item information; or, it may also include other item information except the target item information. In the case where the counter-proof rule includes the target item information, the counter-proof rule is used in a positive proof manner, that is, to verify whether at least one item information includes the target item information set in the counter-proof rule. In the case of inclusion, it means that at least one item information includes the target item information that is adapted to the key frame scene semantics, otherwise, it means that at least one item information does not include the target item information that is adapted to the scene semantics to which the key frame belongs; in the case where the counter-proof rule does not include the target item information, the counter-proof rule is used in a reverse proof manner, that is, to verify whether at least one item information falls within the scope of other item information except the target item information included in the counter-proof rule; if at least one item information falls within the scope of other item information included in the counter-proof rule, it means that at least one item information does not include the target item information that is adapted to the scene semantics to which the key frame belongs, otherwise, it means that at least one item information includes the target item information that is adapted to the scene semantics to which the key frame belongs. Whether it is a positive proof or a reverse proof, in this embodiment, it is to verify whether at least one item information includes the target item information that is adapted to the scene semantics to which the key frame belongs.

[0059] In this embodiment, when at least one of the item information does not include the target item information, it is determined that the scene semantics to which the specified number of video clips are continuously identified is the same scene semantics to which the key frame does not belong; when at least one of the item information includes the target item information, it is determined that the scene semantics to which the specified number of video clips are continuously identified is the same scene semantics to which the key frame belongs. Further, the verification result is fed back to the scene recognition model so that the scene recognition model can determine whether to continue the sliding window processing. The sliding window processing can be implemented with reference to the above step S303; further, if Figure 3As shown in step S305 in , if the verification result is the same, it means that the scene semantics of the specified number of video clips to which the key frame belongs are the same, and the sliding window processing can be continued; if the verification result is different, it means that the specified number of video clips to which the key frame belongs do not belong to the same scene semantics, and the sliding window processing can be stopped. In an example, for example, if the scene semantics of the key frame is bathroom, the target item information adapted to the scene semantics of the bathroom can be a toilet; then it can be verified whether at least one item information includes a toilet, if not, it can be determined that the specified number of video clips to which the key frame belongs do not belong to the scene semantics of the bathroom; if included, it can be determined that the specified number of video clips to which the key frame belongs belong to the scene semantics of the bathroom; for another example, if the scene semantics of the key frame is kitchen, the target item information adapted to the scene semantics of the kitchen can be a range hood; then it can be verified whether at least one item information includes a range hood, if not, it can be determined that the specified number of video clips to which the key frame belongs do not belong to the kitchen; if included, it can be determined that the specified number of video clips to which the key frame belongs belong to the kitchen.

[0060] In the embodiments of the present application, the concept of target scene semantics is introduced. The target scene semantics refers to the scene semantics including sub-scene semantics. For example, if the target scene semantics is a bedroom, it may include sub-scene semantics such as a master bedroom and a second bedroom. For another example, if the target scene semantics is a balcony, it may include sub-scene semantics such as a living room balcony, a master bedroom balcony, and a second bedroom balcony. Among them, there are multiple key frames belonging to the target scene semantics, each key frame has a corresponding start frame and an end frame, and the corresponding video segment can be determined according to the start frame and the end frame. Multiple key frames can determine multiple video segments, and these video segments belong to the target scene semantics, and each video segment has a corresponding sub-scene semantics.

[0061] First, in order to distinguish multiple video segments under the target scene semantics, it is proposed to distinguish multiple video segments by marking. For example, for the target scene semantics of bedroom, when the video segment of bedroom is recognized for the first time, it is marked as bedroom A; further, when the video segment of bedroom is recognized for the second time, it is marked as bedroom B, so as to realize the division of bedroom A and bedroom B. However, bedroom A and bedroom B are only simply distinguished by marking. Although it can be known that both belong to the target scene semantics of bedroom, it is not possible to identify which of bedroom A and bedroom B is the master bedroom and which is the secondary bedroom.

[0062] Therefore, in this embodiment, in order to realize the division of sub-scene semantics for multiple video segments of target scene semantics, this embodiment also provides an optional implementation method, in which a multimodal recognition model is used to divide the target scene semantics into sub-scene semantics. Among them, the multimodal recognition model is pre-trained using sample video streams of some houses, and the sample video stream includes multiple sample video frames, and these sample video frames include sample video frames of different sub-scene semantics under each target scene semantics; the multimodal recognition model is trained using sample video frames in the sample video stream to obtain a multimodal recognition model, and the multimodal recognition model can perform lighting feature recognition, field of view feature recognition, and area size recognition on video frames to realize further division of the target scene semantics.

[0063] In this embodiment, according to the scene semantics to which each key frame belongs, it is determined whether there are multiple key frames that all belong to the target scene semantics, where the target scene semantics refers to the scene semantics including multiple sub-scene semantics; if the judgment result is yes, multiple video segments corresponding to the target scene semantics are determined, each video segment is a continuous video frame determined according to the start frame and the end frame corresponding to the scene semantics to which a key frame belongs; the multiple video segments of the target scene semantics are input into the multimodal recognition model to perform sub-scene semantics recognition on the video segments of the target scene semantics, and obtain the sub-scene semantics corresponding to each video segment; in this process, the multimodal recognition model is used to perform at least one of the following operations:

[0064] Operation 1: Perform lighting feature recognition for each video segment to obtain lighting feature information contained in each video segment, including but not limited to light intensity, light source direction, and light source quantity. Optionally, the time information and light source direction can be combined to distinguish between natural light sources and artificial light sources to further determine the lighting conditions of the bedroom. For example, the master bedroom generally has better lighting than the second bedroom. When it is recognized that bedroom A has better lighting than bedroom B, bedroom A can be considered to be the master bedroom and bedroom B is the second bedroom.

[0065] Operation 2: Perform field of view feature recognition on each video segment to obtain field of view feature information contained in each video segment, the field of view feature information including but not limited to the degree of openness and field of view direction. For example, the master bedroom generally has a better field of view than the secondary bedroom. When it is recognized that bedroom A has a better field of view than bedroom B, bedroom A can be considered to be the master bedroom and bedroom B is the secondary bedroom.

[0066] Operation 3: Perform area feature recognition on each video segment to obtain area feature information contained in each video segment, and the area feature information may include area size. For example, the master bedroom generally has a larger area than the secondary bedroom. When it is recognized that bedroom A has a better area than bedroom B, bedroom A can be considered to be the master bedroom and bedroom B is the secondary bedroom.

[0067] For each video segment, the sub-scene semantics to which the video segment belongs is identified based on at least one of lighting feature information, field of view feature information and area feature information contained in the video segment; wherein there is at least one video segment belonging to the same sub-scene semantics.

[0068] Further, if Figure 3 As shown in S306, according to the position of the sliding window when it stops sliding in two directions, the start frame and the end frame corresponding to the scene semantics to which the key frame belongs are determined. The start frame can be determined according to the start frame position, and the start frame position is the farthest time position of the sliding window sliding in the time direction earlier than the timestamp of the key frame; correspondingly, the end frame can be determined according to the end frame position, and the end frame position is the farthest time position of the sliding window sliding in the time direction later than the timestamp of the key frame; according to the start frame and the end frame, the video segment corresponding to the scene semantics to which the key frame belongs is determined, and the scene semantics of the video segment are the same as the scene semantics to which the key frame belongs. Optionally, the scene semantics recognition result is output in the form of a structured description file. For example, the output structure is as follows (unit / second):

[0069] {Start time: 0.0; End time: 30.3; Scene semantics: "Living room"};

[0070] {Start time: 30.3; End time: 75.6, Scene semantics: "Bedroom"};

[0071] {Start time: 75.6; End time: 95.0; Scene semantics: "Bathroom"};

[0072] {Start time: 95.0: End time: 133.6, Scene semantics: "kitchen"}.

[0073] As shown in the above structure, the video segment corresponding to each scene semantics has a corresponding start time and end time, which correspond to the start frame position and the end frame position respectively. For example, for the living room, the start time is 0.0, the end time is 30.3, and the duration is 30.3 seconds. The video frames during this period belong to the living room; the subsequent scene semantics such as bedroom, bathroom and kitchen are similar and will not be repeated here.

[0074] In this embodiment, based on the recognition result of the output scene semantics of the above-mentioned scene recognition model, the generation model can generate the audio explanation information matching it. Wherein, the generation model can determine the corresponding video segment of each scene semantics according to the recognition result of the scene semantics, that is, the scene semantics to which the key frame belongs and its corresponding start frame and end frame, and then, based on the scene semantics to which each video segment belongs and the start frame and end frame of each video segment, configure the audio explanation information adapted thereto for each video segment, so as to obtain the target house video with the audio explanation information, wherein the video segment and the adapted audio explanation information have the same start frame and end frame.

[0075] Optionally, for each video segment, identify whether the scene semantics to which the video segment belongs is the target scene semantics, where the target scene semantics refers to the scene semantics including multiple sub-scene semantics; if the judgment result is yes, obtain the sub-scene semantics to which the video segment belongs, and based on the sub-scene semantics to which the video segment belongs and the start frame and end frame of the video segment, configure the video segment with audio explanation information that is adapted thereto, so as to obtain the target house video with the audio explanation information.

[0076] Optionally, the generation model can also receive text information, which is text information that describes the video segment corresponding to the scene semantics and can be provided by the content provider. Based on the text information, each scene semantics and the video segment belonging to the scene semantics, corresponding audio explanation information is generated. The audio content in the audio explanation information is generated based on the text information. The audio explanation information has the same start time and end time as at least one video segment corresponding to the scene semantics.

[0077] In an optional embodiment, the model can be constructed based on each scene semantics and its corresponding at least one video segment to generate a three-dimensional house model of the target house. The process includes: obtaining initial data from at least one video segment corresponding to the same scene semantics; the initial data is 2D house type data corresponding to the scene semantics; then, extracting available plane information from the 2D house type data, and processing the plane information to obtain three-dimensional spatial data of the spatial area corresponding to the scene semantics, thereby generating a three-dimensional spatial model corresponding to each spatial area in the target house based on the three-dimensional spatial data; integrating multiple three-dimensional spatial models corresponding to multiple spatial areas in the target house to generate an initial three-dimensional house model of the target house; then, based on the image information in at least one video segment, mapping corresponding textures and materials to different spatial areas of the initial three-dimensional house model to obtain a target three-dimensional house model of the target house.

[0078] Optionally, the audio explanation information of the video segment used to describe the spatial area and the three-dimensional spatial model corresponding to the spatial area can be combined to obtain a three-dimensional spatial model with audio explanation information; then, the three-dimensional spatial model with audio explanation information is integrated and other subsequent operations are performed to obtain a target three-dimensional house model of the target house. The corresponding spatial area in the target house has audio explanation information, so that when the user browses the corresponding area in the target house model, the audio explanation information can be played, so that the user can better understand the details of the house.

[0079] The detailed implementation and beneficial effects of each step in the method of this embodiment have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0080] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 103 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0081] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0082] Figure 4 FIG. 1 is a schematic diagram of a structure of an electronic device provided by another exemplary embodiment of the present application. Figure 4 As shown, the device includes: a memory 44 and a processor 45.

[0083] The memory 44 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0084] The memory 44 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0085] The processor 45 is coupled to the memory 44 and is used to execute the computer program in the memory 44, so as to: receive a video stream obtained by shooting a target house, the video stream includes multiple video frames, the target house includes multiple spatial regions, and each spatial region has corresponding scene semantics; input the multiple video frames into a scene recognition model to perform scene semantics recognition at the granularity of video frames, and obtain the scene semantics to which each video frame belongs and the probability of belonging to the scene semantics; according to the scene semantics to which each video frame belongs, determine the continuous video frames belonging to the same scene semantics; according to the probability that each video frame in the continuous video frames belonging to the same scene semantics belongs to the scene semantics, determine the continuous video frames belonging to the same scene semantics. The method comprises the following steps: determining a key frame from continuous video frames having a certain meaning; for each key frame, using a sliding window to slide in two directions away from the key frame in sequence starting from the key frame, inputting a video segment that falls into the sliding window each time into the scene recognition model, identifying the probability that each video segment and the key frame belong to the same scene semantics with the video segment as the granularity, and stopping the sliding when a specified number of video segments are continuously identified that do not belong to the same scene semantics as the key frame according to the probability that each video segment and the key frame belong to the same scene semantics; and determining a start frame and an end frame corresponding to the scene semantics to which the key frame belongs according to the position of the sliding window when it stops sliding in two directions.

[0086] In an optional embodiment, the processor 45 inputs the multiple video frames into a scene recognition model to perform scene semantics recognition at the granularity of video frames to obtain the scene semantics to which each video frame belongs and the probability of it belonging to the scene semantics. Specifically, it is used to: input the multiple video frames into the scene recognition model, perform feature extraction on any video frame to obtain feature information of any video frame; predict the scene semantics of any video frame to obtain the probability that any video frame may belong to each scene semantics; select a first target probability that meets a first probability condition from the probabilities that any video frame may belong to each scene semantics, and use the scene semantics corresponding to the first target probability and the first target probability as the corresponding scene semantics to which any video frame belongs and the probability that it belongs to the scene semantics.

[0087] In an optional embodiment, before inputting the multiple video frames into the scene recognition model, the processor 45 is further used to: receive the floor plan of the target house; determine the scene semantic range of the target house based on the floor plan of the target house; input the scene semantic range into the scene recognition model as a constraint condition for scene semantic recognition; accordingly, perform scene semantic prediction on any of the video frames to obtain the probability that any of the video frames may belong to each scene semantics, including: within the scene semantic range, perform scene semantic prediction on any of the video frames to obtain the probability that any of the video frames may belong to each scene semantics.

[0088] In an optional embodiment, before using the sliding window to slide in two directions away from the key frame in sequence starting from the key frame, the processor 45 is used to: obtain parameter information of the video stream, the parameter information including the duration and / or frame rate of the video stream; determine the width of the sliding window according to the duration and / or frame rate of the video stream; wherein the duration and / or frame rate of the video stream is positively correlated with the width of the sliding window.

[0089] In an optional embodiment, when the processor 45 inputs the video segment that falls into the sliding window each time into the scene recognition model to identify the probability that each video segment and the key frame belong to the same scene semantics at the granularity of the video segment, it is specifically used to: input the video segment that falls into the sliding window each time into the scene recognition model, perform feature extraction on each video frame in the video segment respectively, and obtain feature information of each video frame in the video segment; perform feature aggregation on the feature information of each video frame in the video segment to obtain aggregated features; and predict the probability that the video segment and the key frame belong to the same scene semantics based on the aggregated features.

[0090] In an optional embodiment, when the processor 45 determines the start frame and the end frame corresponding to the scene semantics to which the key frame belongs based on the position of the sliding window when it stops sliding in two directions, it is specifically used to: when sliding in the time direction earlier than the timestamp of the key frame, the position of the sliding window farthest from the key frame when it stops sliding is used as the start frame corresponding to the scene semantics to which the key frame belongs; when sliding in the time direction later than the timestamp of the key frame, the position of the sliding window farthest from the key frame when it stops sliding is used as the end frame corresponding to the scene semantics to which the key frame belongs.

[0091] In an optional embodiment, when the processor 45 continuously identifies that a specified number of video clips and the key frame do not belong to the same scene semantics based on the probability that each video clip and the key frame belong to the same scene semantics, the processor 45 is specifically used to: based on the probability that each video clip and the key frame belong to the same scene semantics, if a specified number of video clips with a probability less than a set probability threshold are continuously identified; then the specified number of video clips are input into an object detection model for object detection to obtain at least one item information appearing in the specified number of video clips; the at least one item information and the scene semantics to which the key frame belongs are input into a contradiction-proof model, and based on the contradiction-proof rule between the item information and the scene semantics, verify whether the at least one item information includes target item information that is compatible with the scene semantics to which the key frame belongs; and when the at least one item information does not include the target item information, determine that the specified number of video clips and the key frame do not belong to the same scene semantics.

[0092] In an optional embodiment, the processor 45 is further used to: when the at least one object information includes target object information, determine that the specified number of video clips and the key frame belong to the same scene semantics, and continue to slide the sliding window.

[0093] In an optional embodiment, when the processor 45 continuously identifies a specified number of video clips with a probability less than a set probability threshold based on the probability that each video clip and the key frame belong to the same scene semantics, the processor 45 is specifically used to: for each video clip, determine whether the probability that the video clip and the key frame belong to the same scene semantics is less than the set probability threshold; if the judgment result is less than the set probability threshold, and the count value used to count the number of video clips does not reach the specified number, then add 1 to the count value until it reaches the specified number; if the judgment result is greater than or equal to the set probability threshold, and the count value is not zero, then clear the count value.

[0094] In an optional embodiment, the processor 45 is also used to: determine whether there are multiple key frames belonging to the target scene semantics based on the scene semantics to which each key frame belongs, wherein the target scene semantics refers to the scene semantics including multiple sub-scene semantics; if the judgment result is yes, determine multiple video segments corresponding to the target scene semantics, each video segment is a continuous video frame determined based on the start frame and the end frame corresponding to the scene semantics to which a key frame belongs; input the multiple video segments into a multimodal recognition model, and use the multimodal recognition model to perform at least one of the following operations: perform lighting feature recognition on each video segment to obtain lighting feature information contained in each video segment; perform field of view feature recognition on each video segment to obtain field of view feature information contained in each video segment; perform area size recognition on each video segment to obtain area feature information contained in each video segment; for each video segment, identify the sub-scene semantics to which the video segment belongs based on at least one of the lighting feature information, field of view feature information and area feature information contained in the video segment; wherein there is at least one video segment belonging to the same sub-scene semantics.

[0095] In an optional embodiment, the processor 45 is further used to: determine the video segment corresponding to the scene semantics of each key frame according to the start frame and the end frame corresponding to the scene semantics of each key frame; and configure audio explanation information adapted to each video segment based on the scene semantics to which each video segment belongs and the start frame and the end frame of each video segment, so as to obtain a target house video with audio explanation information.

[0096] In an optional embodiment, when the processor 45 configures audio explanation information adapted to each video segment based on the scene semantics to which each video segment belongs and the start frame and the end frame of each video segment to obtain a target house video with audio explanation information, it is specifically used to: for each video segment, identify whether the scene semantics to which the video segment belongs is the target scene semantics, and the target scene semantics refers to the scene semantics including multiple sub-scene semantics; if the judgment result is yes, obtain the sub-scene semantics to which the video segment belongs, and based on the sub-scene semantics to which the video segment belongs and the start frame and the end frame of the video segment, configure the audio explanation information adapted to the video segment to obtain the target house video with audio explanation information.

[0097] Further, if Figure 4 As shown, the electronic device also includes: a communication component 46, a display 47, a power component 48, an audio component 49 and other components. Figure 4 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 4 In addition, Figure 4The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the working node. The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT device, or a server device such as a conventional server, a cloud server, or a server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 4 If the working node of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Figure 4 Components within the dashed box.

[0098] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the processor is enabled to implement the steps in the above method.

[0099] An embodiment of the present application also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the processor is enabled to implement the steps in the above method embodiment.

[0100] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0101] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0102] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0103] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0104] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0105] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) containing computer-usable program code.

[0106] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0107] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0109] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0110] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0111] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0112] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0113] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A housing video processing method, characterized in that: include: Receiving a video stream obtained by shooting a target house, the video stream including a plurality of video frames, the target house including a plurality of spatial regions, each of which has corresponding scene semantics; Inputting the multiple video frames into a scene recognition model to perform scene semantics recognition at the video frame granularity, and obtaining the scene semantics to which each video frame belongs and the probability of it belonging to the scene semantics; According to the scene semantics to which each video frame belongs, continuous video frames belonging to the same scene semantics are determined; according to the probability that each video frame in the continuous video frames belonging to the same scene semantics belongs to the scene semantics, key frames are determined from the continuous video frames belonging to the same scene semantics; For each key frame, a sliding window is used to slide in two directions away from the key frame in sequence starting from the key frame, and the video segments that fall into the sliding window each time are input into the scene recognition model to identify the probability that each video segment and the key frame belong to the same scene semantics based on the video segment granularity, and the sliding is stopped when a specified number of video segments are continuously identified as not belonging to the same scene semantics as the key frame according to the probability that each video segment and the key frame belong to the same scene semantics; According to the position of the sliding window when it stops sliding in two directions, the starting frame and the ending frame corresponding to the scene semantics to which the key frame belongs are determined.

2. The method according to claim 1, characterized in that Inputting the multiple video frames into a scene recognition model to perform scene semantics recognition at the video frame granularity, and obtaining the scene semantics to which each video frame belongs and the probability of it belonging to the scene semantics, including: Inputting the multiple video frames into a scene recognition model, and performing feature extraction on any video frame to obtain feature information of any video frame; Predicting scene semantics for any of the video frames to obtain probabilities that the any of the video frames may belong to each scene semantics; A first target probability that meets a first probability condition is selected from the probabilities that any of the video frames may belong to each of the scene semantics, and the scene semantics corresponding to the first target probability and the first target probability are respectively used as the scene semantics to which the corresponding any of the video frames belongs and the probability that it belongs to the scene semantics.

3. The method according to claim 2, characterized in that Before inputting the plurality of video frames into the scene recognition model, the method further includes: Receiving a floor plan of the target house; Determining a scene semantic range of the target house based on the floor plan of the target house; Inputting the scene semantic range into the scene recognition model as a constraint condition for scene semantic recognition; Accordingly, the scene semantics are predicted for any video frame to obtain the probability that any video frame may belong to each scene semantics, including: Within the scope of the scene semantics, the scene semantics of any video frame are predicted to obtain the probability that the any video frame may belong to each scene semantics.

4. The method according to claim 1, characterized in that: Before using the sliding window to sequentially slide in two directions away from the key frame starting from the key frame, the method further includes: Acquire parameter information of the video stream, wherein the parameter information includes duration and / or frame rate of the video stream; Determining the width of the sliding window according to the duration and / or frame rate of the video stream; The duration and / or frame rate of the video stream is positively correlated with the width of the sliding window.

5. The method according to claim 1, characterized in that Inputting the video segments that fall into the sliding window each time into the scene recognition model to identify the probability that each video segment and the key frame belong to the same scene semantics with the video segment as the granularity, including: Inputting the video clips that fall into the sliding window each time into the scene recognition model, performing feature extraction on each video frame in the video clip, and obtaining feature information of each video frame in the video clip; Performing feature aggregation on feature information of each video frame in the video clip to obtain aggregated features; The probability that the video segment and the key frame belong to the same scene semantics is predicted based on the aggregated features.

6. The method according to claim 1, characterized in that Determining the start frame and the end frame corresponding to the scene semantics to which the key frame belongs according to the position where the sliding window stops sliding in two directions, including: When sliding in the time direction earlier than the timestamp of the key frame, the position of the sliding window farthest from the key frame when the sliding window stops sliding is used as the starting frame corresponding to the semantics of the scene to which the key frame belongs; When sliding in the time direction later than the timestamp of the key frame, the position of the sliding window farthest from the key frame when the sliding window stops sliding is used as the end frame corresponding to the scene semantics to which the key frame belongs.

7. The method according to claim 1, characterized in that According to the probability that each video segment and the key frame belong to the same scene semantics, continuously identifying a specified number of video segments and the key frame that do not belong to the same scene semantics, including: According to the probability that each video clip and the key frame belong to the same scene semantics, if a specified number of video clips with a probability less than a set probability threshold are continuously identified, the specified number of video clips are input into the object detection model for object detection to obtain at least one item information appearing in the specified number of video clips; Inputting the at least one item information and the scene semantics to which the key frame belongs into a disproof model, and verifying whether the at least one item information includes target item information that is compatible with the scene semantics to which the key frame belongs according to a disproof rule between the item information and the scene semantics; In a case where the at least one item information does not include target item information, it is determined that the specified number of video clips and the key frame do not belong to the same scene semantics.

8. The method according to claim 7, characterized in that Also includes: In a case where the at least one item information includes target item information, it is determined that the specified number of video clips and the key frame belong to the same scene semantics, and the sliding operation on the sliding window continues.

9. The method according to claim 7, characterized in that: According to the probability that each video segment and the key frame belong to the same scene semantics, a specified number of video segments whose probability is less than a set probability threshold are continuously identified, including: For each video segment, determining whether the probability that the video segment and the key frame belong to the same scene semantics is less than a set probability threshold; If the judgment result is less than the set probability threshold, and the count value used to count the number of video clips has not reached the specified number, then the count value is increased by 1 until it reaches the specified number; If the judgment result is greater than or equal to the set probability threshold, and the count value is not zero, the count value is cleared.

10. The method according to any one of claims 1 to 9, characterized in that: Also includes: According to the scene semantics to which each key frame belongs, determining whether there are multiple key frames that all belong to the target scene semantics, wherein the target scene semantics refers to the scene semantics including multiple sub-scene semantics; If the judgment result is yes, determine multiple video segments corresponding to the target scene semantics, each video segment is a continuous video frame determined according to a start frame and an end frame corresponding to the scene semantics to which a key frame belongs; Input the multiple video segments into a multimodal recognition model, and use the multimodal recognition model to perform at least one of the following operations: Perform lighting feature recognition on each video segment to obtain lighting feature information contained in each video segment; Performing visual field feature recognition on each video segment to obtain visual field feature information contained in each video segment; Identify the area size of each video segment to obtain area feature information contained in each video segment; For each video segment, the sub-scene semantics to which the video segment belongs is identified according to at least one of lighting feature information, field of view feature information and area feature information contained in the video segment; wherein there is at least one video segment belonging to the same sub-scene semantics.

11. The method according to any one of claims 1 to 9, characterized in that: Also includes: Determine, according to the starting frame and the ending frame corresponding to the scene semantics to which each key frame belongs, the video segment corresponding to the scene semantics to which each key frame belongs; Based on the scene semantics to which each video segment belongs and the start frame and the end frame of each video segment, each video segment is configured with audio explanation information adapted thereto, so as to obtain a target house video with audio explanation information.

12. The method according to claim 11, characterized in that Based on the scene semantics of each video segment and the start frame and end frame of each video segment, each video segment is configured with audio explanation information adapted thereto, so as to obtain a target house video with audio explanation information, including: For each video segment, identifying whether the scene semantics to which the video segment belongs is a target scene semantics, where the target scene semantics refers to a scene semantics including a plurality of sub-scene semantics; If the judgment result is yes, the sub-scene semantics to which the video segment belongs is obtained, and based on the sub-scene semantics to which the video segment belongs and the start frame and the end frame of the video segment, audio explanation information adapted to the video segment is configured to obtain a target house video with audio explanation information.

13. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method according to any one of claims 1 to 12.

14. A computer-readable storage medium storing a computer program / instruction, characterized in that: When the computer program / instructions are executed by a processor, the processor is enabled to implement the steps of the method according to any one of claims 1 to 12.

15. A computer program product, characterized in that include: A computer program / instruction, when executed by a processor, causes the processor to implement the steps of the method according to any one of claims 1 to 12.

Images