VR content problem detection method, device, equipment and storage medium

Abstract

The present application provides a VR content problem detection method, device, equipment and storage medium, which can be used in the field of virtual reality technology. The method includes: obtaining an input target VR image to be detected and at least one corresponding target review category; the target VR image to be detected is a plane VR image generated by a preset projection algorithm; inputting the target VR image into at least one trained problem detection model corresponding to the target review category to perform problem detection on the target VR image corresponding to the target review category; obtaining the problem detection result output by at least one trained problem detection model, and determining whether the target VR image has passed the review of the corresponding target review category based on the problem detection result. Missed detection caused by limited viewing angle is avoided, and there is no need to convert the plane VR image into a VR image with a spherical field of view, which further reduces the computing power overhead of the processing process, significantly improves the processing speed, and improves the problem detection efficiency.

Description

Technical Field

[0001] The present application relates to the field of virtual reality technology, and in particular to a method, device, equipment and storage medium for detecting problems in VR content. Background Art

[0002] During the production process of VR content, it is necessary to use a projection transformation algorithm to map the spherical field of view into a flat image to generate a VR content file that includes the flat image; before the VR content is released, it is necessary to detect problems in the VR content to determine whether the VR content has passed the review and can be released.

[0003] At present, after receiving the VR content file to be tested, it is usually necessary to perform projection transformation on the VR content file to restore it to a spherical field of view, and display the VR content in the spherical field of view so that the auditor can observe whether there are any problems in the VR content. Manual observation of whether there are any problems in VR content is inefficient, consumes a lot of time resources, and manual review may lead to missed detection problems due to limited viewing angles. Summary of the invention

[0004] The present application provides a VR content problem detection method, apparatus, device and storage medium to solve the problem that manual observation of whether there are problems in VR content is inefficient, consumes a lot of time resources, and manual review may lead to missed detection due to limited viewing angle.

[0005] In a first aspect, the present application provides a method for detecting VR content problems, comprising:

[0006] Obtaining an input target VR image to be detected and at least one corresponding target audit category; the target VR image to be detected is a planar VR image generated by projecting a stereoscopic field of view image through a preset projection algorithm;

[0007] Input the target VR image into at least one trained problem detection model corresponding to the target audit category, and use at least one trained problem detection model to detect problems corresponding to the target audit category on the target VR image; the trained problem detection model is pre-trained using corresponding training samples;

[0008] Obtain a problem detection result output by at least one trained problem detection model, and determine whether a target VR image has passed the review of a corresponding target review category based on the problem detection result.

[0009] Optionally, the target VR image is a VR image frame in a VR video stream to be detected, and the step of obtaining the input target VR image to be detected includes:

[0010] The input VR video stream file to be detected is obtained, and the VR video stream file to be detected is decoded to obtain a corresponding VR image frame sequence; the VR video stream file to be detected is a planar VR video stream file generated by projecting a stereoscopic field of view image through a preset projection algorithm; VR image frames in the VR image frame sequence are extracted according to a preset image frame extraction algorithm; and the extracted VR image frames are sequentially determined as target VR images to be detected.

[0011] Optionally, extracting VR image frames from a VR image frame sequence according to a preset image frame extraction algorithm includes:

[0012] VR image frames in a VR image frame sequence are extracted using a frame-by-frame extraction algorithm; or, VR image frames in a VR image frame sequence are extracted using a preset ratio extraction algorithm; or, VR image frames in a VR image frame sequence are extracted using a preset time interval or a preset frame interval extraction algorithm.

[0013] Optionally, the at least one target review category includes at least one of the following categories: content quality review, technical quality review, and content compliance review; the step of inputting the target VR image into at least one trained problem detection model corresponding to the target review category, and using at least one trained problem detection model to perform problem detection on the target VR image corresponding to the target review category includes:

[0014] If it is determined that the target review category includes content quality review, the target VR image is input into the trained content quality review problem detection model, so that the trained content quality review problem detection model is used to detect the target VR image for problems corresponding to the content quality; the corresponding content quality problems include: subtitle mismatch, action mismatch, and composition mismatch; if it is determined that the target review category includes technical quality review, the target VR image is input into the trained technical quality review problem detection model, so that the trained technical quality review problem detection model is used to detect the target VR image for problems corresponding to the technical quality; the corresponding technical quality problems include: color cast problems and excessive brightness problems; if it is determined that the target review category includes content compliance review, the target VR image is input into the trained content compliance review problem detection model, so that the trained content compliance review problem detection model is used to detect the target VR image for problems corresponding to the content compliance; the corresponding content compliance problems include: containing illegal information and containing illegal information.

[0015] Optionally, using corresponding training samples to pre-train a corresponding problem detection model includes:

[0016] A first training sample including problems corresponding to content quality is input into a content quality review problem detection model, and the content quality review problem detection model is trained to obtain a trained content quality review problem detection model; the first training sample includes a first VR image sample and corresponding content quality problem annotations; a second training sample including problems corresponding to technical quality is input into a technical quality review problem detection model, and the technical quality review problem detection model is trained to obtain a trained technical quality review problem detection model, and the second training sample includes a second VR image sample and technical quality problem annotations; a third training sample including problems corresponding to content compliance is input into a content compliance review problem detection model, and the content compliance review problem detection model is trained to obtain a trained content compliance review problem detection model, and the third training sample includes a third VR image sample and annotations corresponding to the content compliance problem type.

[0017] Optionally, before using the corresponding training samples to pre-train the corresponding problem detection model, the method includes:

[0018] Acquire training samples and marked problem areas and marked problem labels in the training samples; the training samples include: VR images that are not detected by the trained problem detection model but have problems, and VR images that are detected by the trained problem detection model and have problem combination areas that are not manually marked; the preset projection algorithm is the training sample of the Equirectangular projection algorithm, and the corresponding marked problem areas include: boundary image areas and combined areas relative to boundary image areas; determine the audit category corresponding to the training sample based on the marked problem labels.

[0019] Optionally, determining whether the target VR image has passed the review of the corresponding target review category based on the problem detection result includes:

[0020] If it is determined that the problem detection results output by a trained problem detection model include: marked problem areas and corresponding problem labels, then it is determined that the target VR image has not passed the review of the corresponding review category of the trained problem detection model; if it is determined that the problem detection results output by a trained problem detection model do not include: marked problem areas and corresponding problem labels, then it is determined that the target VR image has passed the review of the corresponding review category of the trained problem detection model.

[0021] In a second aspect, the present application provides a VR content problem detection device, comprising:

[0022] An acquisition module, used to acquire an input target VR image to be detected and at least one corresponding target audit category; the target VR image to be detected is a planar VR image generated by projecting a stereoscopic field of view image through a preset projection algorithm;

[0023] A detection module, used to input the target VR image into at least one trained problem detection model corresponding to the target audit category, and use at least one trained problem detection model to perform problem detection on the target VR image corresponding to the target audit category; the trained problem detection model is pre-trained using corresponding training samples;

[0024] The determination module is used to obtain a problem detection result output by at least one trained problem detection model, and determine whether the target VR image has passed the review of the corresponding target review category based on the problem detection result.

[0025] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory and a transceiver communicatively connected to the processor;

[0026] The memory stores computer-executable instructions; the transceiver is used to send and receive data;

[0027] The processor executes the computer-executable instructions stored in the memory to implement the VR content problem detection method described in any one of the above aspects.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the VR content problem detection method described in any of the above aspects.

[0029] In a fifth aspect, the present application provides a computer program product, comprising computer execution instructions, which, when executed by a processor, implement the VR content problem detection method described in any of the above aspects.

[0030] The VR content problem detection method, device, equipment and storage medium provided by the present application obtain the input target VR image to be detected and at least one corresponding target audit category; the target VR image to be detected is a plane VR image generated by projecting a stereoscopic field of view image through a preset projection algorithm; the target VR image is input into at least one trained problem detection model corresponding to the target audit category, and at least one trained problem detection model is used to detect the problem corresponding to the target audit category on the target VR image; the trained problem detection model is pre-trained using the corresponding training sample; the problem detection result output by at least one trained problem detection model is obtained, and based on the problem detection result, it is determined whether the target VR image has passed the audit of the corresponding target audit category. Using the trained problem detection model to detect problems in the target VR image can be free from the view angle limit, avoid missed detection caused by the view angle limit, and improve the detection efficiency and reduce the time resources consumed. In addition, the problem detection can be directly performed on the plane VR image generated by projecting the stereoscopic field of view image through the preset projection algorithm, without converting the plane VR image into a VR image with a spherical field of view, further reducing the computing power overhead of the processing process, significantly improving the processing speed, and improving the problem detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0032] Figure 1 A schematic diagram of a network architecture of this application;

[0033] Figure 2 A flow chart of a method for detecting VR content problems provided in an embodiment of the present application;

[0034] Figure 3 A flow chart of a VR content problem detection method provided in an embodiment of the present application;

[0035] Figure 4 A flow chart of a method for training a problem detection model provided in an embodiment of the present application;

[0036] Figure 5 Schematic diagram of a planar VR image generated using the Equirectangular projection algorithm;

[0037] Figure 6 A schematic diagram of the structure of a VR content problem detection device provided in an embodiment of the present application;

[0038] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0039] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0040] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0041] The terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. In the description of the following embodiments, "multiple" means more than two, unless otherwise clearly and specifically defined.

[0042] First, the prior art involved in the present invention is described and analyzed in detail.

[0043] During the production process of VR content, it is necessary to use a projection transformation algorithm to map the spherical field of view into a flat image to generate a VR content file that includes the flat image; before the VR content is released, it is necessary to detect problems in the VR content to determine whether the VR content has passed the review and can be released.

[0044] At present, after receiving the VR content file to be tested, it is usually necessary to perform projection transformation on the VR content file to restore it to a spherical field of view, and display the VR content in the spherical field of view so that the auditor can observe whether there are any problems in the VR content. Manual observation of whether there are any problems in VR content is inefficient, consumes a lot of time resources, and manual review may lead to missed detection problems due to limited viewing angles.

[0045] The inventors found in their research that artificial intelligence models can be trained in advance, and artificial intelligence AI can be used to detect problems in VR content files, which is not limited by viewing angles, avoids missed detection caused by limited viewing angles, and can improve detection efficiency and reduce time resources consumed. Therefore, the present application provides a VR content problem detection method, which obtains an input target VR image to be detected and at least one corresponding target review category; the target VR image to be detected is a planar VR image generated by projecting a stereoscopic field of view image through a preset projection algorithm; the target VR image is input into at least one trained problem detection model corresponding to the target review category, and at least one trained problem detection model is used to perform problem detection on the target VR image corresponding to the target review category; the trained problem detection model is pre-trained using corresponding training samples; the problem detection result output by at least one trained problem detection model is obtained, and based on the problem detection result, it is determined whether the target VR image has passed the review of the corresponding target review category.

[0046] The VR content problem detection method, device, equipment and storage medium provided in this application are suitable for situations where it is necessary to use a traditional video system to evaluate, detect, and supervise pre-made VR content, including but not limited to content detection, technical detection, content supervision, etc.; including but not limited to use in pre-broadcast review, content transaction detection, content warehousing detection, live real-time monitoring, broadcast content supervision and other links.

[0047] Figure 1 A schematic diagram of a network architecture of this application is shown in FIG. Figure 1 As shown, it includes: a user terminal 1, an electronic device 2, and the electronic device 2 includes a VR content problem detection device; the user can send the VR content to be reviewed and the target review category to the electronic device 2 through the user terminal 1, and the VR content to be reviewed includes the target VR image to be detected; after receiving the VR content to be reviewed and the target review category, the electronic device 2 can extract the target VR image from the VR content to be reviewed, and execute the VR content problem detection method provided by the present application, use at least one trained problem detection model corresponding to the target review category to perform problem detection on the target VR image corresponding to the target review category, and obtain the output problem detection result to determine whether the target VR image has passed the review of the corresponding target review category.

[0048] It should be understood that the VR content that can be processed by the VR content problem detection method of the present application includes but is not limited to VR videos and static images produced by shooting, CG generation, or a combination of the two, VR content with shallow interactive operations, and VR content generated in real time by a rendering engine, etc.; including but not limited to 360°, 180° videos / images with binocular parallax, and single-eye and binocular panoramic videos / images without parallax, etc.; including but not limited to VR content generated using projection algorithms such as Equirectagular and Cube_Map. The VR content problem detection method provided in the present application can process prefabricated VR content including but not limited to VR video content corresponding to various resolutions, image compression algorithms (MPEG-2, H.264, H.265, AVS, AV-1, etc.), frames (1:1, 4:3, 16:9, 16:10, etc.), file storage formats (TS, AVI, etc.), distribution service methods (file distribution, streaming services such as broadcast, multicast, unicast, etc.), file encryption methods, viewing methods (head-mounted devices, TVs, computers, mobile terminals, etc.), and usage methods (consumer services, industry vertical applications, etc.).

[0049] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0050] It should be noted that the VR content problem detection method, device, equipment and storage medium of the present application can be used in the field of virtual reality technology, and can also be used in any field outside the field of virtual reality technology. The application field of the VR content problem detection method, device, equipment and storage medium of the present application is not limited.

[0051] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0052] Figure 2This is a flow chart of a VR content problem detection method provided in an embodiment of the present application. The embodiment of the present application provides a VR content problem detection method to address the problem that manual observation of whether there are problems in VR content is inefficient, consumes a lot of time resources, and manual review may result in missed detection due to limited viewing angles. The method in this embodiment is applied to a VR content problem detection device, and the VR content problem detection device can be located in an electronic device. Among them, the electronic device can be a digital computer representing various forms. Such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.

[0053] like Figure 2 As shown, the specific steps of this method are as follows:

[0054] Step S101: obtaining an input target VR image to be detected and at least one corresponding target review category.

[0055] The target VR image to be detected is a planar VR image generated by projecting a stereoscopic field of view image through a preset projection algorithm. The target review category is the category for reviewing the target VR image.

[0056] In the embodiment of the present application, the VR image may be a static VR image or a frame image in a VR video.

[0057] In the embodiment of the present application, after obtaining the input target VR image to be detected, at least one corresponding target review category can be obtained. Specifically, the user can select the target review category for detecting the target VR image on the user terminal and send it to the VR content problem detection device in the electronic device.

[0058] Step S102: input the target VR image into at least one trained problem detection model corresponding to the target audit category, and use at least one trained problem detection model to perform problem detection corresponding to the target audit category on the target VR image.

[0059] The trained problem detection model is pre-trained using corresponding training samples. One trained problem detection model corresponds to one audit category.

[0060] In the embodiment of the present application, the problem detection model can be any applicable artificial intelligence model. The problem detection model can have image processing and recognition functions.

[0061] Specifically, after determining the target audit category, the target VR image is input into the trained problem detection model corresponding to the target audit category, and problem detection corresponding to each target audit category is performed respectively.

[0062] Step S103: Obtain a problem detection result output by at least one trained problem detection model, and determine whether the target VR image has passed the review of the corresponding target review category based on the problem detection result.

[0063] In the embodiment of the present application, based on the problem detection result, it can be determined whether the target VR image has problems corresponding to each target review category. If there are problems corresponding to the target review category, it can be determined that the review of the target review category has not been passed, and the failed target review category and the corresponding problems can be sent to the user terminal so that the user can correct the problems corresponding to the target review category; if it is determined that there are no problems corresponding to the target review category, it can be determined that the review of the target review category has been passed, and the target VR image can be released.

[0064] The VR content problem detection method provided by the present application obtains an input target VR image to be detected and at least one corresponding target audit category; the target VR image to be detected is a plane VR image generated by projecting a stereoscopic field of view image through a preset projection algorithm; the target VR image is input into at least one trained problem detection model corresponding to the target audit category, and at least one trained problem detection model is used to detect problems corresponding to the target audit category on the target VR image; the trained problem detection model is pre-trained using corresponding training samples; the problem detection result output by at least one trained problem detection model is obtained, and based on the problem detection result, it is determined whether the target VR image has passed the audit of the corresponding target audit category. Using the trained problem detection model to detect problems in the target VR image can be free from the view angle limit, avoid missed detection caused by the view angle limit, improve detection efficiency, and reduce the time resources consumed. In addition, the problem detection can be directly performed on the plane VR image generated by projecting the stereoscopic field of view image through the preset projection algorithm, without converting the plane VR image into a VR image with a spherical field of view, further reducing the computing power overhead of the processing process, significantly improving the processing speed, and improving the problem detection efficiency.

[0065] The VR content problem detection method provided in the above embodiment is described below with reference to a specific example. Figure 3 A flow chart of a VR content problem detection method provided in an embodiment of the present application. In the embodiment of the present application, the target VR image is a VR image frame in a VR video stream to be detected, such as Figure 3 As shown, the following steps are included:

[0066] Step S201: Obtain an input VR video stream file to be detected, and decode the VR video stream file to be detected to obtain a corresponding VR image frame sequence.

[0067] Among them, the VR video stream file to be detected is a planar VR video stream file generated by projecting a stereoscopic field of view image through a preset projection algorithm.

[0068] Step S202: extracting VR image frames from a VR image frame sequence according to a preset image frame extraction algorithm.

[0069] Optionally, extracting VR image frames in a VR image frame sequence according to a preset image frame extraction algorithm may include: extracting VR image frames in a VR image frame sequence according to a frame-by-frame extraction algorithm; or, extracting VR image frames in a VR image frame sequence according to a preset ratio extraction algorithm; or, extracting VR image frames in a VR image frame sequence according to a preset time interval or a preset frame interval extraction algorithm.

[0070] The embodiments of the present application do not limit the preset image frame extraction algorithm used, which can be determined according to specific needs. For example, VR image frames in the VR image frame sequence can be extracted frame by frame, at a preset ratio, at a preset time interval, or at a preset frame interval.

[0071] Specifically, the user may pre-select a preset image frame extraction algorithm in the user terminal, and determine parameters in the image frame extraction algorithm, for example, adjusting a preset ratio, adjusting a preset time interval, and adjusting a preset frame interval.

[0072] In the embodiment of the present application, VR image frames in a VR image frame sequence are extracted according to a frame-by-frame extraction algorithm; or, VR image frames in a VR image frame sequence are extracted according to a preset ratio extraction algorithm; or, VR image frames in a VR image frame sequence are extracted according to a preset time interval or a preset frame interval extraction algorithm. The preset image frame extraction algorithm can be selected as required, and the preset ratio, or the preset time interval, or the preset frame interval can be adjusted, which can improve the detection efficiency while ensuring the accuracy of detecting VR video stream problems, and improve the flexibility and applicability of extracting VR image frames in a VR image frame sequence.

[0073] Step S203: sequentially determine the extracted VR image frames as target VR images to be detected.

[0074] In the embodiment of the present application, after extracting the VR image frames in the VR image frame sequence, the extracted VR image frames are sequentially determined as target VR images to be detected, so as to execute step S204 and subsequent steps to perform problem detection on the target VR images determined to be detected.

[0075] In the embodiment of the present application, the input VR video stream file to be detected is obtained, and the VR video stream file to be detected is decoded to obtain the corresponding VR image frame sequence; the VR video stream file to be detected is a planar VR video stream file generated by projecting the stereoscopic field of view image through a preset projection algorithm; the VR image frames in the VR image frame sequence are extracted according to the preset image frame extraction algorithm; the extracted VR image frames are sequentially determined as the target VR images to be detected. By extracting the target VR image to be detected from the VR video stream to be detected, the detection of the VR video stream can be realized, the efficiency of problem detection of the VR video stream can be improved, and the missed detection of the VR video stream caused by the limited viewing angle can be avoided.

[0076] Step S204: obtaining at least one target audit category corresponding to the target VR image to be detected.

[0077] Step S205: If it is determined that the target audit category includes content quality audit, the target VR image is input into the trained content quality audit problem detection model, so as to use the trained content quality audit problem detection model to detect problems corresponding to the content quality of the target VR image.

[0078] In the embodiment of the present application, the content quality corresponding problems may include: inappropriate subtitles, inappropriate actions, inappropriate composition and other content quality problems.

[0079] Among them, the content quality review problem detection model is a problem detection model that detects problems corresponding to the content quality of VR images.

[0080] Step S206: If it is determined that the target audit category includes technical quality audit, the target VR image is input into the trained technical quality audit problem detection model, so as to use the trained technical quality audit problem detection model to perform technical quality problem detection on the target VR image.

[0081] In the embodiment of the present application, the problems corresponding to technical quality may include: color cast problems, excessive brightness problems and other technical quality problems.

[0082] Among them, the technical quality audit problem detection model is a problem detection model for detecting problems corresponding to the technical quality of VR images.

[0083] Step S207: If it is determined that the target review category includes content compliance review, the target VR image is input into the trained content compliance review problem detection model, so as to use the trained content compliance review problem detection model to perform content compliance problem detection on the target VR image.

[0084] In the embodiment of the present application, content compliance corresponding issues may include: content compliance issues such as containing illegal information and containing illegal information.

[0085] Among them, the content compliance review problem detection model is a problem detection model that detects problems corresponding to content compliance of VR images.

[0086] Specifically, the user can determine whether to conduct content quality review, technical quality review, and content compliance review on the target VR image by setting the target review category. Exemplarily, the target review category may include: content quality review and technical quality review, and the target VR image is input into the trained content quality review problem detection model and the trained technical quality review problem detection model to conduct content quality review and technical quality review on the target VR image.

[0087] In an embodiment of the present application, the audit categories may include: content quality audit, technical quality audit, and content compliance audit. Using corresponding problem detection models to detect content quality problems, technical quality problems, and content compliance problems respectively can improve the accuracy of detecting problems corresponding to the three types of audit categories. And the problem detection corresponding to content quality, the problem detection corresponding to technical quality, and the problem detection corresponding to content compliance can be performed in parallel to improve detection efficiency. And users can select the target audit category without having to perform problem detection on unnecessary audit categories.

[0088] Step S208: store the output problem detection result and determine whether it passes the review.

[0089] Specifically, whether the problem detection result has passed the review can be determined by determining whether the output problem detection result includes: the marked problem area and the corresponding problem label. If it is determined that the problem detection result output by a trained problem detection model includes: the marked problem area and the corresponding problem label, it is determined that the target VR image has not passed the review of the corresponding review category of the trained problem detection model; if it is determined that the problem detection result output by a trained problem detection model does not include: the marked problem area and the corresponding problem label, it is determined that the target VR image has passed the review of the corresponding review category of the trained problem detection model.

[0090] If it is determined that the target VR image has not passed the review of any target review category, then it is determined that the target VR image has not passed the review.

[0091] In the embodiment of the present application, the trained problem detection model can output the target VR image, as well as the problem area annotated in the target VR image and the corresponding problem label. Specifically, the problem area annotated by generating a detection box can be output.

[0092] Whether the review of the corresponding review category has been passed can be determined by whether the marked problem area and the corresponding problem label are output, so that the user can determine the reason for failure to pass the review and modify the target VR image so that the modified target VR image can pass the review.

[0093] Step S209: Determine whether to traverse the extracted VR image frames.

[0094] Step S210: Determine whether all extracted VR image frames have passed the review.

[0095] Step S211: If all the extracted VR image frames pass the review, it is determined that the input VR video stream file to be detected can be released.

[0096] Optionally, it is also possible to select the VR image frames that have been extracted and reviewed, perform projection transformation on the selected VR image frames to restore them into a spherical field of view, and display the selected VR image frames of the spherical field of view for manual review. If it is determined that the selected VR image frames have problems that have not been detected by the trained problem detection model, the extracted VR image frames are determined as VR images that have not been detected by the trained problem detection model but have problems, and are added to the training samples to train the problem detection model corresponding to the undetected problems.

[0097] Step S212: If all extracted VR image frames fail to pass the review, the review problem detection results of the failed VR image frames are sent to the corresponding user terminal.

[0098] Optionally, the user can determine whether the problem area marked in the VR image frame that failed the review is a combination of multiple areas. If it is determined to be a group combination of multiple areas, and the combination of areas is not marked in the training sample, then the VR image frame that failed the review can be determined as a VR image detected by the trained problem detection model and containing unlabeled problem combination areas. The VR image frame and the labeled problem areas and corresponding problem labels can be added to the training samples corresponding to the labeled problems, and the corresponding training samples can be used to further train the corresponding problem detection model.

[0099] The following is a specific example to illustrate the method of training the problem detection model. Figure 4 As shown, the following steps are included:

[0100] Step S301: Obtain training samples and the marked problem areas and marked problem labels in the training samples.

[0101] The training samples include: VR images that have problems but are not detected by the trained problem detection model, and VR images that have problem combination areas that are detected by the trained problem detection model and are not manually labeled.

[0102] In an embodiment of the present application, after problem detection is performed on the target VR image, VR images that have problems but are not detected by the trained problem detection model, as well as VR images that have been detected by the trained problem detection model and have problem combination areas that are not manually labeled, can be added to the corresponding training samples, thereby improving the analysis and judgment capabilities of the trained problem detection model and increasing the accuracy of detecting problems.

[0103] Optionally, the preset projection algorithm is a training sample of an Equirectangular projection algorithm, and the corresponding annotated problem area includes: a boundary image area and a combined area relative to the boundary image area.

[0104] It should be understood that due to the closed-loop characteristics of the boundaries of VR images, there is continuity of image information between the four boundaries of a frame of planar VR image, and a certain algorithm must be used to ensure the integrity and traversal of the information.

[0105] Figure 5 This is a schematic diagram of a planar VR image generated using the Equirectangular projection algorithm. Figure 5 As shown, if the trained problem detection model detects image area A, there will be no missed detection due to missing information; if the trained problem detection model detects boundary image area B and boundary image area C, due to the continuity of the image, the effective information may be partially in the relative boundary image area B# and boundary image area C#, then only the boundary image area B or boundary image area C may be detected for possible information omission. Therefore, when annotating the training samples whose preset projection algorithm is the Equirectangular projection algorithm, if the problem area is in the boundary image area, the combined area of ​​the boundary image area and the relative boundary image area is annotated as the problem area at the same time, for example, the annotated problem areas are: boundary image area B+boundary image area B#, boundary image area C+boundary image area C#.

[0106] In an embodiment of the present application, the preset projection algorithm is a training sample of the Equirectangular projection algorithm, and the corresponding annotated problem area includes: a boundary image area and a combined area of ​​a relative boundary image area, which can avoid missed judgments caused by information loss due to the boundary closed-loop characteristics of the planar VR image.

[0107] In an embodiment of the present application, when obtaining training samples, different sampling window shapes, sizes, movement methods and compensation methods can be set to correspond according to different frame resolutions to be detected, sampling accuracy requirements required by the problem detection model, processing speed requirements, and different projection algorithms and other limitations. Each sampling window is not necessarily completely consistent.

[0108] Step S302: Determine the audit category corresponding to the training sample based on the annotated question label.

[0109] Specifically, after determining the review category corresponding to the training sample based on the annotated question label, the training sample can be determined to be the first training sample, the second training sample, or the third training sample based on the review category corresponding to the training sample.

[0110] Step S303: input the first training sample including the problems corresponding to the content quality into the content quality review problem detection model, train the content quality review problem detection model, and obtain a trained content quality review problem detection model.

[0111] The first training sample includes a first VR image sample and a corresponding content quality problem annotation. The first VR image sample has a content quality problem.

[0112] Step S304: input the second training sample including the corresponding technical quality problems into the technical quality audit problem detection model, train the technical quality audit problem detection model, and obtain a trained technical quality audit problem detection model.

[0113] The second training sample includes a second VR image sample and a technical quality problem annotation. The second VR image sample has a technical quality problem.

[0114] Step S305: input the third training sample including the corresponding issues of content compliance into the content compliance review issue detection model, train the content compliance review issue detection model, and obtain a trained content compliance review issue detection model.

[0115] The third training sample includes a third VR image sample and a label corresponding to the content compliance problem type. There is a content compliance problem in the third VR image sample.

[0116] In an embodiment of the present application, different training samples are used to respectively train a content quality review problem detection model, a technical quality review problem detection model, and a content compliance review problem detection model. This can improve the recognition accuracy of the problem detection model for the corresponding problem. For example, it can avoid missing detection of a certain type of review problem when there are multiple types of review problems.

[0117] Figure 6 This is a structural diagram of the VR content problem detection device provided in the embodiment of the present application. The VR content problem detection device provided in the embodiment of the present application can execute the processing flow provided in the VR content problem detection method embodiment. Figure 6 As shown, the VR content problem detection device 50 includes: an acquisition module 501, a detection module 502, and a determination module 503.

[0118] Specifically, the acquisition module 501 is used to acquire an input target VR image to be detected and at least one corresponding target audit category; the target VR image to be detected is a planar VR image generated by projecting a stereoscopic field of view image through a preset projection algorithm;

[0119] The detection module 502 is used to input the target VR image into at least one trained problem detection model corresponding to the target audit category, and use the at least one trained problem detection model to detect problems corresponding to the target audit category on the target VR image; the trained problem detection model is pre-trained using corresponding training samples;

[0120] The determination module 503 is used to obtain the problem detection result output by at least one trained problem detection model, and determine whether the target VR image has passed the review of the corresponding target review category based on the problem detection result.

[0121] The device provided in the embodiments of the present application can be specifically used to execute the method embodiments provided in the above embodiments, and the specific functions will not be repeated here.

[0122] Optionally, the acquisition module 501 is specifically used to: obtain an input VR video stream file to be detected, and decode the VR video stream file to be detected to obtain a corresponding VR image frame sequence; the VR video stream file to be detected is a planar VR video stream file generated by projecting a stereoscopic field of view image through a preset projection algorithm; extract VR image frames in the VR image frame sequence according to a preset image frame extraction algorithm; and sequentially determine the extracted VR image frames as target VR images to be detected.

[0123] Optionally, the acquisition module 501 is specifically used to: extract VR image frames in a VR image frame sequence according to a frame-by-frame extraction algorithm; or, extract VR image frames in a VR image frame sequence according to a preset ratio extraction algorithm; or, extract VR image frames in a VR image frame sequence according to a preset time interval or a preset frame interval extraction algorithm.

[0124] Optionally, the at least one target review category includes at least one of the following categories: content quality review, technical quality review, and content compliance review; the detection module 502 is specifically used to:

[0125] If it is determined that the target review category includes content quality review, the target VR image is input into the trained content quality review problem detection model, so that the trained content quality review problem detection model is used to detect the target VR image for problems corresponding to content quality; the corresponding content quality problems include: subtitle mismatch, action mismatch, and composition mismatch; if it is determined that the target review category includes technical quality review, the target VR image is input into the trained technical quality review problem detection model, so that the trained technical quality review problem detection model is used to detect the target VR image for problems corresponding to technical quality; the corresponding technical quality problems include: color cast problems and excessive brightness problems; if it is determined that the target review category includes content compliance review, the target VR image is input into the trained content compliance review problem detection model, so that the trained content compliance review problem detection model is used to detect the target VR image for problems corresponding to content compliance; the corresponding content compliance problems include: containing illegal information and containing illegal information.

[0126] Optionally, the VR content problem detection device 50 further includes: a training module; the training module is used to:

[0127] The first training sample including the corresponding problems of content quality is input into the content quality review problem detection model, and the content quality review problem detection model is trained to obtain the trained content quality review problem detection model; the first training sample includes the first VR image sample and the corresponding content quality problem annotation; the second training sample including the corresponding problems of technical quality is input into the technical quality review problem detection model, and the technical quality review problem detection model is trained to obtain the trained technical quality review problem detection model, and the second training sample includes the second VR image sample and the technical quality problem annotation; the third training sample including the corresponding problems of content compliance is input into the content compliance review problem detection model, and the content compliance review problem detection model is trained to obtain the trained content compliance review problem detection model, and the third training sample includes the third VR image sample and the annotation corresponding to the content compliance problem type.

[0128] Optionally, the training module is also used to: obtain training samples and marked problem areas and marked problem labels in the training samples; the training samples include: VR images that have not been detected by the trained problem detection model but have problems, and VR images that have been detected by the trained problem detection model and have combined problem areas that have not been manually marked; the preset projection algorithm is the Equirectangular projection algorithm for training samples, and the corresponding marked problem areas include: boundary image areas and combined areas relative to boundary image areas; and determine the audit category corresponding to the training samples based on the marked problem labels.

[0129] Optionally, the determination module 503 is specifically used to: if it is determined that the problem detection result output by a trained problem detection model includes: a marked problem area and a corresponding problem label, then it is determined that the target VR image has not passed the review of the corresponding review category of the trained problem detection model; if it is determined that the problem detection result output by a trained problem detection model does not include: a marked problem area and a corresponding problem label, then it is determined that the target VR image has passed the review of the corresponding review category of the trained problem detection model.

[0130] The device provided in the embodiment of the present application can be specifically used to execute the above method embodiment, and the specific functions will not be repeated here.

[0131] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 7 As shown, the present application also provides an electronic device 60, including: a processor 601, and a memory 602 and a transceiver 603 that are communicatively connected to the processor 601. The memory 602 stores computer-executable instructions; the transceiver 603 is used to send and receive data; the processor 601 executes the computer-executable instructions stored in the memory 602 to implement the method provided in any embodiment of the present application.

[0132] Specifically, the program may include a program code, and the program code includes a computer-executable instruction. The memory 602 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. Among them, the computer-executable instruction is stored in the memory 602 and is configured to be executed by the processor 601 to implement the method provided in any embodiment of the present application. The relevant description can be understood by referring to the relevant description and effect corresponding to the steps in the accompanying drawings, and no further elaboration is made here.

[0133] In the embodiment of the present application, the memory 602 and the processor 601 are connected via a bus. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0134] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method provided by any embodiment of the present application.

[0135] An embodiment of the present application also provides a computer program product, including computer execution instructions, which implement the method provided by any embodiment of the present application when executed by a processor.

[0136] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0137] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0138] In addition, each functional module in each embodiment of the present application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0139] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable full-path trajectory fusion device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0140] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0141] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0142] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0143] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for detecting VR content problems, characterized in that: include: Obtain an input target VR image to be detected and at least one corresponding target review category for detecting the target VR image, wherein the target review category is selected by a user on a user terminal; the target VR image to be detected is a planar VR image generated by projecting a stereoscopic field of view image through a preset projection algorithm, and there is continuity of image information between four boundaries of the planar VR image; the target review category includes at least one of the following: content quality review, technical quality review, and content compliance review; Input the target VR image into at least one trained problem detection model corresponding to at least one target audit category, and use at least one trained problem detection model to detect problems corresponding to the target audit category on the target VR image; the trained problem detection model is pre-trained using corresponding training samples; Obtaining a problem detection result output by at least one trained problem detection model, and determining whether the target VR image has passed the review of the corresponding target review category according to whether the output problem detection result includes a marked problem area and a corresponding problem label; Before pre-training the corresponding problem detection model using the corresponding training samples, the method further includes: Obtaining training samples and marked problem areas and marked problem labels in the training samples; The training samples include: VR images that have problems but are not detected by the trained problem detection model, and VR images that have problem combination areas that are detected by the trained problem detection model and are not manually marked; The preset projection algorithm is a training sample of the Equirectangular projection algorithm, and the corresponding annotated problem areas include: a boundary image area and a combined area relative to the boundary image area; Determine the audit category corresponding to the training sample based on the annotated question labels; The step of inputting the target VR image into at least one trained problem detection model corresponding to the target audit category, and using at least one trained problem detection model to perform problem detection corresponding to the target audit category on the target VR image includes: If it is determined that the target audit category includes content quality audit, the target VR image is input into the trained content quality audit problem detection model, so as to detect the problems corresponding to the content quality of the target VR image using the trained content quality audit problem detection model; the problems corresponding to the content quality include: inappropriate subtitles, inappropriate actions, and inappropriate composition; If it is determined that the target audit category includes a technical quality audit, the target VR image is input into a trained technical quality audit problem detection model, so as to detect problems corresponding to the technical quality of the target VR image using the trained technical quality audit problem detection model; the problems corresponding to the technical quality include: color cast problems and excessive brightness problems; If it is determined that the target review category includes content compliance review, the target VR image is input into the trained content compliance review problem detection model, so as to use the trained content compliance review problem detection model to perform content compliance problem detection on the target VR image; content compliance corresponding problems include: containing illegal information and containing illegal information.

2. The method according to claim 1, characterized in that The target VR image is a VR image frame in a VR video stream to be detected, and the step of obtaining the input target VR image to be detected includes: Obtain an input VR video stream file to be detected, and decode the VR video stream file to be detected to obtain a corresponding VR image frame sequence; the VR video stream file to be detected is a planar VR video stream file generated by projecting a stereoscopic field of view image through a preset projection algorithm; Extracting VR image frames from a VR image frame sequence according to a preset image frame extraction algorithm; The extracted VR image frames are sequentially determined as target VR images to be detected.

3. The method according to claim 2, characterized in that The step of extracting VR image frames from a VR image frame sequence according to a preset image frame extraction algorithm includes: Extracting VR image frames from a VR image frame sequence according to a frame-by-frame extraction algorithm; Or, extracting VR image frames from a VR image frame sequence according to a preset ratio extraction algorithm; Or, extract the VR image frames in the VR image frame sequence according to a preset time interval or a preset frame interval extraction algorithm.

4. The method according to claim 1, characterized in that Use the corresponding training samples to pre-train the corresponding problem detection model, including: Inputting a first training sample including content quality corresponding problems into a content quality review problem detection model, training the content quality review problem detection model to obtain a trained content quality review problem detection model; the first training sample includes a first VR image sample and a corresponding content quality problem annotation; Inputting a second training sample including the technical quality corresponding problem into a technical quality audit problem detection model, training the technical quality audit problem detection model to obtain a trained technical quality audit problem detection model, wherein the second training sample includes a second VR image sample and a technical quality problem annotation; A third training sample including corresponding content compliance issues is input into a content compliance review issue detection model, and the content compliance review issue detection model is trained to obtain a trained content compliance review issue detection model, wherein the third training sample includes a third VR image sample and annotations corresponding to the content compliance issue type.

5. The method according to claim 1, characterized in that Determining whether the target VR image has passed the review of the corresponding target review category based on the problem detection result includes: If it is determined that the problem detection result output by a trained problem detection model includes: a marked problem area and a corresponding problem label, then it is determined that the target VR image has not passed the review of the corresponding review category of the trained problem detection model; If it is determined that the problem detection results output by a trained problem detection model do not include: the marked problem area and the corresponding problem label, then it is determined that the target VR image has passed the review of the corresponding review category of the trained problem detection model.

6. A VR content problem detection device, characterized in that: include: An acquisition module is used to acquire an input target VR image to be detected and at least one corresponding target review category for detecting the target VR image, wherein the target review category is selected by a user on a user terminal; the target VR image to be detected is a planar VR image generated by projecting a stereoscopic field of view image through a preset projection algorithm, and there is continuity of image information between four boundaries of the planar VR image; the target review category includes at least one of the following: content quality review, technical quality review, and content compliance review; A detection module, used to input the target VR image into at least one trained problem detection model corresponding to at least one target audit category, and use at least one trained problem detection model to perform problem detection on the target VR image corresponding to the target audit category; the trained problem detection model is pre-trained using corresponding training samples; A determination module is used to obtain a problem detection result output by at least one trained problem detection model, and determine whether the target VR image has passed the review of the corresponding target review category according to whether the output problem detection result includes a marked problem area and a corresponding problem label; The device also includes a training module, Before the corresponding problem detection model is pre-trained using the corresponding training sample, it is also used to obtain the training sample and the problem area and the marked problem label in the training sample. The training sample includes: VR images that have not been detected by the trained problem detection model but have problems, VR images that have been detected by the trained problem detection model and have problem combination areas that have not been manually marked, the preset projection algorithm is the training sample of the Equirectangular projection algorithm, and the corresponding marked problem area includes: a boundary image area and a combination area of ​​a relative boundary image area; the audit category corresponding to the training sample is determined based on the marked problem label; The detection module is specifically used for, if it is determined that the target audit category includes content quality audit, inputting the target VR image into a trained content quality audit problem detection model, so as to use the trained content quality audit problem detection model to detect problems corresponding to the content quality of the target VR image; problems corresponding to content quality include: subtitle mismatch, action mismatch, and composition mismatch; if it is determined that the target audit category includes technical quality audit, then inputting the target VR image into a trained technical quality audit problem detection model, so as to use the trained technical quality audit problem detection model to detect problems corresponding to the technical quality of the target VR image; problems corresponding to technical quality include: color cast problem and excessive brightness problem; if it is determined that the target audit category includes content compliance audit, then inputting the target VR image into a trained content compliance audit problem detection model, so as to use the trained content compliance audit problem detection model to detect problems corresponding to content compliance of the target VR image; problems corresponding to content compliance include: containing illegal information and containing illegal information.

7. An electronic device, characterized in that: include: A processor, and a memory and a transceiver communicatively connected to the processor; The memory stores computer-executable instructions; the transceiver is used to send and receive data; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 5 when executed by a processor.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images