Object detection method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present disclosure disclose an object detection method and device, electronic equipment and a storage medium, wherein the method comprises: obtaining a current camera attribute corresponding to a current perspective of a user and a panoramic mask graph corresponding to a current point; regions in the panoramic mask graph corresponding to different objects in a panoramic graph of the current point are respectively different colors; generating a rendering image corresponding to the current perspective according to the current camera attribute and the panoramic mask graph; detecting color information included in the rendering image; and determining an object observed under the current perspective according to the color information. Embodiments of the present disclosure identify objects through color detection, greatly reducing the object identification calculation complexity and improving real-time performance.

Description

Technical Field

[0001] This disclosure relates to three-dimensional panoramic technology, and in particular to an object detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] In 3D scenarios such as VR (Virtual Reality) space tours, including VR house viewing and VR sightseeing, it is usually necessary to determine whether the user sees a preset object from their current perspective, and then trigger subsequent interactions or other operations based on the result. Related technologies typically perform object recognition on panoramic images; however, object recognition based on panoramic images has high computational complexity and poor real-time performance. Summary of the Invention

[0003] This disclosure provides an object detection method, apparatus, electronic device, and storage medium to reduce the computational complexity of object recognition in three-dimensional scenes and effectively improve real-time performance.

[0004] One aspect of this disclosure provides an object detection method, including:

[0005] Obtain the current camera attributes corresponding to the user's current viewpoint and the panoramic mask image corresponding to the current point; the regions in the panoramic mask image corresponding to different objects in the panoramic image of the current point are of different colors;

[0006] Based on the current camera attributes and the panoramic mask image, generate a rendered image corresponding to the current viewpoint;

[0007] Detect the color information included in the rendered image;

[0008] Based on the color information, determine the object observed from the current viewpoint.

[0009] In one embodiment of this disclosure, generating a rendered image corresponding to the current viewpoint based on the current camera attributes and the panoramic mask image includes:

[0010] Obtain the panoramic sphere model in the reference coordinate system corresponding to the panoramic mask image and the mapping relationship between each coordinate point of the panoramic sphere model and the pixels of the panoramic mask image; the center point of the panoramic sphere model is the camera position corresponding to the current viewpoint;

[0011] Based on the current camera attributes, project each target coordinate point of the panoramic sphere model within the field of view of the current viewpoint onto the rendering canvas corresponding to the current viewpoint to obtain the projected pixels corresponding to each target coordinate point.

[0012] Based on the panoramic mask and the mapping relationship, determine the pixel value of the pixel in the panoramic mask corresponding to each of the target coordinate points;

[0013] Based on the pixel values ​​corresponding to each of the target coordinate points, determine the pixel values ​​corresponding to each of the projected pixels.

[0014] The rendered image is determined based on the pixel values ​​corresponding to each of the projected pixels.

[0015] In one embodiment of this disclosure, detecting the color information included in the rendered image includes:

[0016] The color types included in the rendered image are determined based on the pixel values ​​of each pixel in the rendered image.

[0017] In one embodiment of this disclosure, detecting the color information included in the rendered image includes:

[0018] The rendered image is downsampled to obtain a target image at the target resolution;

[0019] The color types included are determined based on the pixel values ​​of each pixel in the target image.

[0020] In one embodiment of this disclosure, determining the object observed from the current viewpoint based on the color information includes:

[0021] Based on the color information, including the color types and the mapping relationship between the color types and objects, the object observed from the current viewpoint is determined.

[0022] Another aspect of this disclosure provides an object detection device, comprising:

[0023] The acquisition module is used to acquire the current camera attributes corresponding to the user's current viewpoint and the panoramic mask image corresponding to the current point; the regions in the panoramic mask image corresponding to different objects in the panoramic image of the current point are of different colors.

[0024] The first processing module is used to generate a rendered image corresponding to the current viewpoint based on the current camera attributes and the panoramic mask image.

[0025] The second processing module is used to detect the color information included in the rendered image;

[0026] The third processing module is used to determine the object observed from the current viewpoint based on the color information.

[0027] In one embodiment of this disclosure, the first processing module is specifically used for:

[0028] Obtain the panoramic sphere model in the reference coordinate system corresponding to the panoramic mask image and the mapping relationship between each coordinate point of the panoramic sphere model and the pixels of the panoramic mask image; the center point of the panoramic sphere model is the camera position corresponding to the current viewpoint;

[0029] Based on the current camera attributes, project each target coordinate point of the panoramic sphere model within the field of view of the current viewpoint onto the rendering canvas corresponding to the current viewpoint to obtain the projected pixels corresponding to each target coordinate point.

[0030] Based on the panoramic mask and the mapping relationship, determine the pixel value of the pixel in the panoramic mask corresponding to each of the target coordinate points;

[0031] Based on the pixel values ​​corresponding to each of the target coordinate points, determine the pixel values ​​corresponding to each of the projected pixels.

[0032] The rendered image is determined based on the pixel values ​​corresponding to each of the projected pixels.

[0033] In one embodiment of this disclosure, the second processing module is specifically used for:

[0034] The color types included in the rendered image are determined based on the pixel values ​​of each pixel in the rendered image.

[0035] In one embodiment of this disclosure, the second processing module is specifically used for:

[0036] The rendered image is downsampled to obtain a target image at the target resolution;

[0037] The color types included are determined based on the pixel values ​​of each pixel in the target image.

[0038] In one embodiment of this disclosure, the third processing module is specifically used for:

[0039] Based on the color information, including the color types and the mapping relationship between the color types and objects, the object observed from the current viewpoint is determined.

[0040] According to another aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the object detection method described in any of the above embodiments of the present disclosure.

[0041] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0042] Memory, used to store computer program products;

[0043] A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the object detection method described in any of the above embodiments of the present disclosure.

[0044] According to another aspect of the present disclosure, a computer program product is provided, including computer program instructions, characterized in that, when executed by a processor, the computer program instructions implement the object detection method described in any of the above embodiments of the present disclosure.

[0045] The object detection method, apparatus, electronic device, and storage medium disclosed herein can acquire the user's current viewpoint in real time during VR viewing by setting a panoramic mask image corresponding to the point location. Based on the current camera attributes corresponding to the user's current viewpoint and the panoramic mask image corresponding to the user's current point location, a rendered image corresponding to the current viewpoint is generated. Then, the color information included in the rendered image is detected. Since different objects in the panoramic mask image are set with different colors, the object observed in the current viewpoint can be determined according to the color type. This enables rapid identification of objects that can be seen from the user's current viewpoint, greatly reduces the computational complexity of object recognition, effectively improves real-time performance, and solves the problems of high computational complexity and poor real-time performance of existing technologies for object recognition using panoramic images.

[0046] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0047] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0048] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0049] Figure 1 This is a schematic flowchart of an exemplary embodiment of the present disclosure of an object detection method;

[0050] Figure 2 This is a flowchart illustrating an object detection method provided in another exemplary embodiment of this disclosure;

[0051] Figure 3 This is a flowchart illustrating an object detection method provided in yet another exemplary embodiment of this disclosure;

[0052] Figure 4 This is a schematic diagram of the structure of an object detection device provided in an exemplary embodiment of the present disclosure;

[0053] Figure 5 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0054] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0055] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0056] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0057] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0058] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0059] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0060] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0061] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0062] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0063] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0064] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0065] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0066] This disclosure outlines

[0067] In the process of developing this disclosure, the inventors discovered that in 3D scenarios such as VR (Virtual Reality) space tours, including VR house viewing and VR sightseeing, it is usually necessary to determine whether the user sees a preset object from their current perspective, and then trigger subsequent interactions or other operations based on the determination result. In related technologies, object recognition is usually performed on panoramic images, but object recognition based on panoramic images has high computational complexity and poor real-time performance.

[0068] Exemplary Overview

[0069] In VR spatial exploration scenarios such as VR house viewing and VR scenic spot tours, when it is necessary to detect whether a preset object is observed from the user's current perspective, the object detection method disclosed herein can be used to obtain the current camera attributes corresponding to the user's current perspective and the panoramic mask image corresponding to the user's current position. This panoramic mask image includes the regions corresponding to each object in the panoramic image of the current position. Regions corresponding to different objects are rendered in different colors, and regions without objects are rendered in a background color different from the color corresponding to the object, such as black. Based on the current camera attributes and the panoramic mask image, a rendered image corresponding to the current perspective is generated. This rendered image is the rendered image corresponding to the region of the panoramic mask image within the field of view of the current perspective. By detecting the rendered image, the color information included in the rendered image can be obtained, and then the objects observed from the current perspective can be determined based on the color information. Since different objects correspond to different colors, it is possible to quickly identify which objects can be observed from the current perspective based on the color type, or to quickly determine whether a preset object can be observed from the current perspective, thereby triggering subsequent interactions or other operations. For example, when a user is touring a tourist attraction in VR, for objects within the scene that require detailed information, the system can display that object's detailed information or show the object's different viewpoints when it detects that the object is visible from the user's current perspective. The specific application of object detection can be configured according to actual needs, and this disclosure does not impose any limitations.

[0070] Exemplary methods

[0071] Figure 1 This is a schematic flowchart of an exemplary embodiment of the present disclosure of an object detection method. This object detection method can be applied to electronic devices, such as mobile phones, tablets, PCs, and other terminal devices. The method includes the following steps:

[0072] Step 201: Obtain the current camera attributes corresponding to the user's current viewpoint and the panoramic mask image corresponding to the current point.

[0073] In the panoramic mask, the regions corresponding to different objects in the panoramic image at the current point are each a different color. Specifically, the panoramic mask includes the regions corresponding to each object in the panoramic image at the current point. Different regions in the panoramic mask are rendered with different colors depending on the object; for example, object A is red, object B is blue, object C is yellow, and so on. The parts other than the objects can be set to a background color different from the object color, such as black. Specific colors can be set according to actual needs and are not limited to the aforementioned settings. The user's current viewpoint is determined based on the user's actions on the terminal device. For example, when a user is using a mobile phone for VR tours, they can enter that point by clicking on a marker in the VR 3D scene and view the corresponding panoramic scene. By swiping on the screen, the viewpoint at the current point changes, allowing the user to view scene content from different perspectives. The specific principles of VR tours will not be elaborated further. Current camera attributes can include the current camera's position, pose, field of view (FOV), clipping plane, aspect ratio, etc. Based on the current camera attributes, the projection matrix from the 3D spatial coordinate points to the rendering canvas can be determined. The coordinate system of the rendering canvas is a normalized coordinate system.

[0074] Step 202: Generate a rendered image corresponding to the current viewpoint based on the current camera attributes and the panoramic mask.

[0075] The rendered image corresponding to the current viewpoint is obtained by rendering based on the pixel values ​​of the panoramic mask image in the area corresponding to the current viewpoint. Therefore, the color information included in the rendered image is consistent with that included in the panoramic mask image area within the current viewpoint range.

[0076] In some optional embodiments, a mapping relationship between the pixels of the panoramic mask and 3D spatial points in the reference coordinate system of the 3D scene can be pre-established to facilitate the rendering of the current viewpoint image based on the panoramic mask, ensuring that the rendered image corresponds to the panoramic mask area within the current viewpoint range. The reference coordinate system can be a pre-established world coordinate system with a preset position in the 3D scene as the origin; the specific reference coordinate system can be set according to actual needs. The mapping relationship between the pixels of the panoramic mask and 3D spatial points can be set according to actual needs. For example, based on the association relationship between the pixels of the panoramic image and 3D spatial points corresponding to each point, some 3D spatial points for association with the panoramic mask can be pre-sampled. Based on the correspondence between each 3D spatial point and the pixels of the panoramic image, and the correspondence between the panoramic image and the pixels of the panoramic mask, a mapping relationship between each 3D spatial point and the panoramic mask can be established.

[0077] In some optional embodiments, the resolution of the panoramic mask can be the same as or different from that of the panoramic image. The correspondence between the objects in the panoramic image and the color areas of the panoramic mask can be achieved by establishing a mapping relationship between the pixels of the panoramic mask and the pixels of the panoramic image. No specific limitation is imposed.

[0078] In some optional embodiments, the mapping relationship between the pixels of the panoramic mask and the points in 3D space can also be achieved by establishing a panoramic sphere model in the 3D scene. Specifically, a 3D mesh model of the sphere can be established, including the vertices of the sphere's surface (called 3D mesh vertices). The panoramic mask is then applied to the sphere as a texture map, or an association relationship can be established between the pixels of the panoramic mask and the vertices of the sphere's 3D mesh. The center point of the sphere is set at the camera position corresponding to the point. By rotating the sphere, the object area of ​​the panoramic mask is mapped to the corresponding object in the 3D scene. The coordinates of the 3D mesh vertices of the sphere in the reference coordinate system are recorded in the corresponding state, obtaining the panoramic sphere model in the reference coordinate system. This ensures that when viewing the panoramic sphere model or the panoramic mask on the model from any angle at that point, the color area seen coincides with the corresponding object, achieving the purpose of detecting observed objects through the color information of the rendered image from the panoramic mask. It should be noted that the panoramic sphere model is only used for generating the rendered image for object detection and does not affect the user's browsing of the panoramic scene. For example, when providing a viewing experience to the user, only the 3D scene can be rendered, without rendering the panoramic sphere model, and the rendering result can be displayed. Conversely, when performing object detection, only the panoramic sphere model can be rendered, without rendering the 3D scene, and the rendering result cannot be displayed. In practical applications, a viewing camera with consistent attributes and an off-screen rendering camera can be established for rendering the user's viewing experience and the object detection image, respectively.

[0079] Step 203: Detect the color information included in the rendered image.

[0080] The detection of color information in a rendered image can be determined by the color values ​​contained in the pixel values ​​of each pixel in the rendered image. Color information can include color types, such as red, blue, green, etc.

[0081] Step 204: Determine the object observed from the current viewpoint based on the color information.

[0082] Depending on the user's current viewpoint and the specific scene content, the color information can include at least one color type. For example, if there is no preset object within the current viewpoint, since the area without an object in the panoramic mask is the preset background color, such as black, the rendered image will be a black image, and the color information will only include black. If there is object A within the current viewpoint, and the color corresponding to object A in the panoramic mask is red, then the color information included in the rendered image can include both black and red color types. Therefore, it can be determined that the object observed under the current viewpoint includes object A. Based on this, fast object detection based on color detection is achieved.

[0083] The object detection method provided in this disclosure, by setting a panoramic mask image corresponding to a point location, can acquire the user's current viewpoint in real time when the user is watching VR. Based on the current camera attributes corresponding to the user's current viewpoint and the panoramic mask image corresponding to the user's current point location, a rendered image corresponding to the current viewpoint is generated. Then, the color information included in the rendered image is detected. Since different objects in the panoramic mask image are set with different colors, the object observed in the current viewpoint can be determined according to the color type. This enables rapid identification of objects that the user can see from the current viewpoint, greatly reduces the computational complexity of object recognition, effectively improves real-time performance, and solves the problems of high computational complexity and poor real-time performance of existing technologies for object recognition using panoramic images.

[0084] Figure 2 This is a flowchart illustrating an object detection method provided in another exemplary embodiment of this disclosure.

[0085] In some optional embodiments, step 202, generating a rendered image corresponding to the current viewpoint based on the current camera attributes and the panoramic mask image, includes:

[0086] Step 2021: Obtain the panoramic sphere model in the reference coordinate system corresponding to the panoramic mask and the mapping relationship between each coordinate point of the panoramic sphere model and the pixels of the panoramic mask.

[0087] In this model, the center point of the panoramic sphere model represents the camera position corresponding to the current viewpoint. The panoramic sphere model is a pre-built sphere model that corresponds to a real object in the 3D scene. Each coordinate point in the panoramic sphere model is a coordinate point in the reference coordinate system. The method for obtaining the panoramic sphere model can be found in the preceding content and will not be repeated here.

[0088] In some optional embodiments, each coordinate point of the panoramic sphere model can correspond one-to-one with the pixels of the panoramic mask image, so as to accurately obtain the pixel value of the rendered image pixel during the rendering image generation process.

[0089] Step 2022: Based on the current camera attributes, project the target coordinates of the panoramic sphere model within the current viewpoint's field of view onto the rendering canvas corresponding to the current viewpoint, and obtain the projected pixels corresponding to each target coordinate.

[0090] Specifically, based on the current camera attributes, the projection matrix from the three-dimensional spatial points of the reference coordinate system to the two-dimensional pixel coordinates of the rendering canvas can be determined. Thus, the coordinate points of the panoramic sphere model can be projected onto the rendering canvas according to the projection matrix to obtain the corresponding projected pixels on the rendering canvas.

[0091] Step 2023: Based on the panoramic mask and mapping relationship, determine the pixel value of the pixel in the panoramic mask corresponding to each target coordinate point.

[0092] The mapping relationship includes the correspondence between the coordinates of each point of the panoramic sphere model and the pixels of the panoramic mask. Based on this, the corresponding pixel value can be obtained from the panoramic mask according to the pixel corresponding to each target coordinate point.

[0093] Step 2024: Determine the pixel value corresponding to each projected pixel based on the pixel value corresponding to each target coordinate point.

[0094] Based on the correspondence between each target coordinate point and the projected pixel, the pixel values ​​of the panoramic mask corresponding to each target coordinate point obtained above can be used as the pixel values ​​of each projected pixel.

[0095] Step 2025: Determine the rendered image based on the pixel values ​​corresponding to each projected pixel.

[0096] In this method, the pixel values ​​corresponding to each projected pixel can be used as the pixel values ​​of each pixel in the rendered image to obtain the rendered image.

[0097] This embodiment uses a panoramic sphere model to establish the association between the pixels of the panoramic mask image and the three-dimensional spatial points of the three-dimensional scene, which facilitates the generation of rendered images for object detection and further improves real-time performance.

[0098] In some optional embodiments, step 203, detecting the color information included in the rendered image, includes:

[0099] Step 2031: Determine the color types included in the rendered image based on the pixel values ​​of each pixel in the rendered image.

[0100] Specifically, the color types included in the rendered image can be determined based on the color values ​​included in each pixel value and the correspondence between color values ​​and color types.

[0101] Figure 3 This is a flowchart illustrating an object detection method provided in yet another exemplary embodiment of this disclosure.

[0102] In some optional embodiments, step 203, detecting the color information included in the rendered image, includes:

[0103] Step 203a: Downsample the rendered image to obtain the target image at the target resolution.

[0104] Downsampling is used to reduce the resolution of the rendered image. For example, if the resolution of the rendered image is H*W, where H represents the height of the rendered image and W represents the width of the rendered image, and the target resolution is (H / 4)*(W / 4), the number of pixels can be reduced to reduce the computational load of color detection. Downsampling can be implemented in any feasible way, such as taking values ​​every other pixel or merging pixels, etc., without any specific limitations.

[0105] Step 203b: Determine the included color types based on the pixel values ​​of each pixel in the target image.

[0106] The specific operation of this step is similar to that of step 2031, and will not be repeated here.

[0107] This embodiment obtains the target image by downsampling the rendered image, which can effectively reduce the image resolution, thereby reducing the computational load of color detection and further improving the real-time performance of object detection.

[0108] In some optional embodiments, step 204, determining the object observed from the current viewpoint based on color information, includes:

[0109] Step 2041: Determine the object observed from the current viewpoint based on the color types included in the color information and the mapping relationship between color types and objects.

[0110] The mapping relationship between color type and object is established when the panoramic mask is obtained. This mapping relationship can include a one-to-one correspondence between color areas in the panoramic mask and objects in the panoramic image. Thus, based on the detected color type, it can accurately determine which object is visible from the current viewpoint, thereby achieving accurate object recognition.

[0111] This embodiment establishes a correspondence between the colors of the panoramic mask image and the objects in the scene through pre-learning, providing an effective reference for object recognition by color.

[0112] Any object detection method provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any object detection method provided in this disclosure can be executed by a processor, such as by a processor executing any object detection method mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0113] Exemplary device

[0114] Figure 4 This is a schematic diagram of an object detection apparatus provided in an exemplary embodiment of the present disclosure. The apparatus of this embodiment can be used to implement corresponding method embodiments of the present disclosure, such as… Figure 4 The device shown includes: an acquisition module 501, a first processing module 502, a second processing module 503, and a third processing module 504.

[0115] The acquisition module 501 is used to acquire the current camera attributes corresponding to the user's current viewpoint and the panoramic mask image corresponding to the current point; the regions corresponding to different objects in the panoramic mask image and the panoramic image of the current point are different colors.

[0116] The first processing module 502 is used to generate a rendered image corresponding to the current viewpoint based on the current camera attributes and the panoramic mask image.

[0117] The second processing module 503 is used to detect the color information included in the rendered image.

[0118] The third processing module 504 is used to determine the object observed from the current viewpoint based on the color information.

[0119] In some optional embodiments, the first processing module 502 is specifically used for:

[0120] Obtain the panoramic sphere model in the reference coordinate system corresponding to the panoramic mask, and the mapping relationship between the coordinates of each point of the panoramic sphere model and the pixels of the panoramic mask; the center point of the panoramic sphere model is the camera position corresponding to the current viewpoint; based on the current camera attributes, project each target coordinate point of the panoramic sphere model within the viewpoint range of the current viewpoint onto the rendering canvas corresponding to the current viewpoint, and obtain the projected pixels corresponding to each target coordinate point; based on the panoramic mask and the mapping relationship, determine the pixel value of the pixels in the panoramic mask corresponding to each target coordinate point; based on the pixel values ​​corresponding to each target coordinate point, determine the pixel value corresponding to each projected pixel; based on the pixel values ​​corresponding to each projected pixel, determine the rendered image.

[0121] In some optional embodiments, the second processing module 503 is specifically used for:

[0122] The color types included in the rendered image are determined based on the pixel values ​​of each pixel in the rendered image.

[0123] In some optional embodiments, the second processing module 503 is specifically used for:

[0124] The rendered image is downsampled to obtain the target image at the target resolution;

[0125] The color types included are determined based on the pixel values ​​of each pixel in the target image.

[0126] In some optional embodiments, the third processing module 504 is specifically used for:

[0127] Based on the color information, including the color types and the mapping relationship between color types and objects, determine the objects observed from the current viewpoint.

[0128] The beneficial technical effects corresponding to the exemplary embodiments of this device can be found in the corresponding beneficial technical effects of the exemplary method section above, and will not be repeated here.

[0129] In addition, this disclosure also provides an electronic device, including:

[0130] Memory, used to store computer programs;

[0131] A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the object detection method described in any of the above embodiments of the present disclosure.

[0132] Figure 5 This is a schematic diagram illustrating the structure of one application embodiment of the electronic device disclosed herein. Figure 5 As shown, the electronic device includes one or more processors and memory.

[0133] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0134] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the methods of the various embodiments of this disclosure described above and / or other desired functions.

[0135] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0136] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0137] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0138] Of course, for the sake of simplicity, Figure 5 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0139] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of this disclosure as described in the foregoing portions of this specification.

[0140] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0141] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this disclosure described in the foregoing portion of this specification.

[0142] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0143] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0144] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0145] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0146] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0147] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0148] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0149] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0150] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. An object detection method, characterized in that, include: Obtain the current camera attributes corresponding to the user's current viewpoint and the panoramic mask image corresponding to the current point; The regions in the panoramic mask that correspond to different objects in the panoramic image of the current point are each a different color. The panoramic mask is used for rendering images of object detection, and the panoramic image is used for rendering the tour screen; Based on the current camera attributes and the panoramic mask image, generate a rendered image corresponding to the current viewpoint; The rendered image is obtained by rendering based on the pixel values ​​of the panoramic mask image in the region corresponding to the current viewpoint. Detect the color information included in the rendered image; Based on the color information, determine the object observed from the current viewpoint; The step of generating the rendered image corresponding to the current viewpoint based on the current camera attributes and the panoramic mask image includes: Obtain the panoramic sphere model in the reference coordinate system corresponding to the panoramic mask image and the mapping relationship between each coordinate point of the panoramic sphere model and the pixels of the panoramic mask image; the center point of the panoramic sphere model is the camera position corresponding to the current viewpoint; Based on the current camera attributes, project each target coordinate point of the panoramic sphere model within the field of view of the current viewpoint onto the rendering canvas corresponding to the current viewpoint to obtain the projected pixels corresponding to each target coordinate point. Based on the panoramic mask and the mapping relationship, determine the pixel value of the pixel in the panoramic mask corresponding to each of the target coordinate points; Based on the pixel values ​​corresponding to each of the target coordinate points, determine the pixel values ​​corresponding to each of the projected pixels. The rendered image is determined based on the pixel values ​​corresponding to each of the projected pixels.

2. The method according to claim 1, characterized in that, The detection of color information included in the rendered image includes: The color types included in the rendered image are determined based on the pixel values ​​of each pixel in the rendered image.

3. The method according to claim 1, characterized in that, The detection of color information included in the rendered image includes: The rendered image is downsampled to obtain a target image at the target resolution; The color types included are determined based on the pixel values ​​of each pixel in the target image.

4. The method according to claim 1, characterized in that, Determining the object observed from the current viewpoint based on the color information includes: Based on the color information, including the color types and the mapping relationship between the color types and objects, the object observed from the current viewpoint is determined.

5. An object detection device, characterized in that, include: The acquisition module is used to acquire the current camera attributes corresponding to the user's current viewpoint and the panoramic mask image corresponding to the current point. The regions in the panoramic mask that correspond to different objects in the panoramic image of the current point are each a different color. The panoramic mask is used for rendering images of object detection, and the panoramic image is used for rendering the tour screen; The first processing module is used to generate a rendered image corresponding to the current viewpoint based on the current camera attributes and the panoramic mask image. The rendered image is obtained by rendering based on the pixel values ​​of the panoramic mask image in the region corresponding to the current viewpoint. The second processing module is used to detect the color information included in the rendered image; The third processing module is used to determine the object observed from the current viewpoint based on the color information; The first processing module is specifically used for: Obtain the panoramic sphere model in the reference coordinate system corresponding to the panoramic mask image and the mapping relationship between each coordinate point of the panoramic sphere model and the pixels of the panoramic mask image; the center point of the panoramic sphere model is the camera position corresponding to the current viewpoint; Based on the current camera attributes, project each target coordinate point of the panoramic sphere model within the field of view of the current viewpoint onto the rendering canvas corresponding to the current viewpoint to obtain the projected pixels corresponding to each target coordinate point. Based on the panoramic mask and the mapping relationship, determine the pixel value of the pixel in the panoramic mask corresponding to each of the target coordinate points; Based on the pixel values ​​corresponding to each of the target coordinate points, determine the pixel values ​​corresponding to each of the projected pixels. The rendered image is determined based on the pixel values ​​corresponding to each of the projected pixels.

6. The apparatus according to claim 5, characterized in that, The second processing module is specifically used for: The rendered image is downsampled to obtain a target image at the target resolution; The color types included are determined based on the pixel values ​​of each pixel in the target image.

7. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor is configured to execute a computer program product stored in the memory, wherein when the computer program product is executed, it implements the object detection method according to any one of claims 1-4.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the object detection method according to any one of claims 1-4.

Images